search

Mottl / GetOldTweets3

A Python 3 library and a corresponding command line utility for accessing old tweets
MIT License
367 stars 126 forks source link

How to handle a urlopen error? #67

Open jkjan opened 4 years ago

jkjan commented 4 years ago

Sometimes I get this error and I don't know how to handle it. This is what I get when I try to crawl tweets like 500 keyword, a keyword at a time. It takes a rest for 16 minutes after it collects 10k more tweets. This error is just random. My teammates has not reported this one yet.

"An error occured during an HTTP request: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond Try to open in browser: ~~ twitter link ~~"

I know why the problem occurs and that is something I cannot fix. This is very rare and I don't have time or capability of fixing that occasional connection error. And if it happens, all processes stop, so I should restart the whole program.

So I decided to avoid that error and go on to the next keyword using try and except. Because I can save a keyword that made an error and search for that later. I tried Exception, but it didn't catch that error. Also HTTPError or URLError, didn't catch that either. What kind of exception can catch that error?

Summary : This is an example code. image

How can I print that "Error!" ?

marmistrz commented 4 years ago

You can't. The library just calls sys.exit. Until #70 is merged, you can use my fork: https://github.com/Mottl/GetOldTweets3/pull/70. Warning: the fork will not be updated.