Add senate crawler - Githubissues

ACTtaiwan / CongressionalGuide

Web based look up for U.S. Congress members and candidates to see if they support certain legislation or issues.

MIT License

8 stars 5 forks source link

Add senate crawler #33

Closed balamark closed 8 years ago

balamark commented 8 years ago

used multiple spiders to crawl different html layout.
added duplicate filter in item pipeline
enabled crawler to search youtubex
for NH, FL, use xpath to locate candidates (spider4)
added readme for my scrapy script
added logging
senate.json contains all senate candidates
CA.json contains senators from CA

soarhevn commented 8 years ago

@balamark I am running your branch but not seeing info get filled out in the db SQLite database. For example, I looked up Chris Vance (challenger for WA Senator) and found on https://ballotpedia.org/Chris_Vance there are links for the website, twitter, and FB, but in the db, (and on the user interface) I don't see those filled out.

soarhevn commented 8 years ago

This is just to crawl.