Crawl Wikipedia pages and save to local MongoDB

[dudarev]

Crawl Wikipedia pages.

Settings:

• Secret token to send data to locodb
• Initial URL(s)
• Time till next request

Data extracted:

• URL
• is_city
• population
• census date (or other latests measurement/estimation date)
• coordinates
• last_crawled

Some moments:

• Do not crawl URLs that were crawled before and for which is_city = False.

[dudarev]

http://en.wikipedia.org/wiki/List_of_towns_and_cities_with_100,000_or_more_inhabitants/cityname:_A

[xentatt]

Tasks Implement simple JavaScript crawler

• [x] Implement mechanism to visit page (use casperjs) given page (in first implementation is a constant)
• [x] Collect list of available cities
• [ ] Visit each node (city) and collect data (keep it in json)
• [x] Move application to a nodejs application
• [x] Use npm mongodb connector for saving data to a DB
• [ ] Save information in DB
• [x] Implement secret token mechanims(?) move to a separate ticket
• [x] Make bot be able to be initiated from other side, with given some data move to a separate ticket
• [ ] Use GeoJson specs to keep location.