search

everypolitician / wikidata-fetcher

Fetch information useful to EveryPolitician from Wikidata
MIT License
4 stars 2 forks source link

Scrape all languages of a category by default #27

Open octopusinvitro opened 8 years ago

octopusinvitro commented 8 years ago

For wikidata scrapers that scrape a category (for example Taiwan) it would be nice if it could automatically scrape all versions of that category in all languages available, so that we can get politicians from a category who may present in some languages but not in others.

For example, in the scraper linked above, Lee Ching-hua was removed from the Chinese page for the Category of Members of the 8th Legislative Yuan, but he was still there in the English version of the page. He indeed belonged in that term.

Since at the moment of writing this issue we were only scraping the Chinese version of that category, we lost him. If we were also scraping the English version, we wouldn't.

octopusinvitro commented 8 years ago

Since I want to learn to work with the Wikipedia API, and without having taken a look at the code yet, I'm scaringly and adventurously assigning myself to this one.