search

jrobi001 / Final-Project-Cryptocurrency-ML

Repository for my final year CS project
0 stars 1 forks source link

SNScrape tweepy crash - rate limit exceeded #1

Closed jrobi001 closed 3 years ago

jrobi001 commented 3 years ago

It may be possible to handle this exception, pause and carry on after 5 mins or so (instead of having to manually comment out code to run just the parts that failed again later)

Does not happen every day or reliably, usually goes away if try again later in the day... already have wait_on_rate_limit=True... there may be some other way to mitigate the crash?

As an alternative could create a secondary file to run just what is needed after a crash. crash only happens in the Tweepy API calls, so just need to carry on from this point (with the variables required) - not ideal, but probably better than commenting out the main file.

jrobi001 commented 3 years ago

Happened for first time in a long while, may be able to catch error and set a time out or something image

jrobi001 commented 3 years ago

This might be a fix, have placed retry_count=10, retry_delay=5, retry_errors=set([503]) in, will see if the error occurs again.

https://stackoverflow.com/questions/48117126/when-using-tweepy-cursor-what-is-the-best-practice-for-catching-over-capacity-e

jrobi001 commented 3 years ago

now got error tweepy.error.TweepError: [{'message': 'Internal error', 'code': 131}] twice in a row, removing the above fixed that....

jrobi001 commented 3 years ago

I think I will wrap the call in a 'try catch' block and set a timeout of 30s or 1min if it fails. If can resolve this, then would feel confident setting up all the different components (tweet collection, tweet analysis, price collection etc.) to run from a single file. Currently have to comment out blocks of the code to resolve the issue whenever occurs.

jrobi001 commented 3 years ago

first occurrence since fix. It seems to have handled it, going to close this. image