search

BenWirus / ZombieVoters

Find dead people who are registered to vote.
MIT License
30 stars 7 forks source link

Error occurred during loading data. Trying to use cache server https://fake-useragent.herokuapp.com/browsers/0.1.11 #7

Open uveski opened 3 years ago

uveski commented 3 years ago

Tried running this on a Mac for Dallas County, Tx - any ideas on the below issue?

./death_scraper.py --sy 1900 --ey 2000 -s tx -c dallas

Obtaining zip codes for dallas tx... Obtaining a random user agent... Error occurred during loading data. Trying to use cache server https://fake-useragent.herokuapp.com/browsers/0.1.11 Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1342, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked)

BenWirus commented 3 years ago

Looks like it's having trouble grabbing a random user agent, I'll put up a fix in a few.

uveski commented 3 years ago 
This seems to work - just started running now

ua = UserAgent(verify_ssl=False)

BenWirus commented 3 years ago

I just pushed a commit that should fix the issue, I set verify_ssl to false and added a large list of user agents will be used if the user agent library fails. Mind testing?

uveski commented 3 years ago

Sure, trying to understand what is needed to get this working for TX / Dallas County, if even possible

https://teamrv-mvp.sos.texas.gov/MVP/mvp.do?ref=voteusa

BenWirus commented 3 years ago

I submitted some bogus data to the form, not sure what to look for in the response to signify if the person has voted, but the minimum requirements to look someone up do work with the data pulled from the myheritage API.

curl -d "selType=lfcd&firstName=john&lastName=doe&dob=07/12/1967&adZip5=75216" -X POST https://teamrv-mvp.sos.texas.gov/MVP/voterDetails.do

It's best to get an idea of what the response looks like when someone has voted and look for something specific in the response that signifies that the person is registered and/or has submitted a ballot.

BenWirus commented 3 years ago

Just an FYI the website is behind Cloudflare and I'm getting a captcha challenge when I use curl. This might cause issues even if the user agent is forged. Might need to figure out the real IP of the server and use that to go around Cloudflare.

paoloaq commented 3 years ago

Just an FYI the website is behind Cloudflare and I'm getting a captcha challenge when I use curl. This might cause issues even if the user agent is forged. Might need to figure out the real IP of the server and use that to go around Cloudflare.

It would be interesting to understand how to get the real IP :(