Refactor advisor script

okfn-brasil / serenata-de-amor

🕵 Artificial Intelligence for social control of public administration | **This repository does not receive frequent updates. Check out the README**

https://serenata.ai/en

MIT License

4.51k stars 662 forks source link

Refactor advisor script #237

Open jtemporal opened 7 years ago

jtemporal commented 7 years ago

As I mentioned in #149 there's a little bit of 💅 to be done on fetch_deputies_advisors script.

jtemporal commented 7 years ago

We need to study what changed that this is not working =/

jtemporal commented 7 years ago

hey @bruno-schmidt since you were the one majorly responsible for this script existence I thought you might want to take a look into it, what do you think? 😄

andrewsmedina commented 7 years ago

I'm not 100% sure, but I believe that the problem is that this page can't be scraped by a curl scrapper like requests.

A way to scrape this page is to use something like splinter (https://splinter.readthedocs.io/en/latest/) or selenium.

What do you think about to use splinter to do it? If you agree I can make a PR to fix this bug.

bruno-schmidt commented 7 years ago

Hey guys!

@jtemporal I just checked it out and the HTML of the page changed a lot. It was not that nice when I and @cuducos wrote the script lol. Some URL changes also. I'm working on it now. As soon as I have something I will post here n__nd

bruno-schmidt commented 7 years ago

Just pushed a PR (#252) @jtemporal ! I think everything is running fine now :D