everypolitician-scrapers / spain_congreso_es

Details of members of the Spanish Congress from the official website congreso.es
https://morph.io/everypolitician-scrapers/spain_congreso_es
1 stars 2 forks source link

WIP: Scrape all memberships using cookies #17

Open ondenman opened 7 years ago

ondenman commented 7 years ago

Note: This scraper depends on the correct cookies being sent with each request to a memberships list page. The local cache will need clearing out if you're running this branch for the first time.

Step one: The scraper scrapes each member page: data from the main profile tab and additional membership information from the 'all memberships' tab.

Step two: Additional memberships are merged with the person data from the member profile page to tab to create separate membership hashes.