bocchilorenzo / ntscraper

Scrape from Twitter using Nitter instances
MIT License
182 stars 29 forks source link

High number of retries detected. Testing all instances... New working instances: #21

Open MadAim123 opened 1 year ago

MadAim123 commented 1 year ago

I always get this message after a while: High number of retries detected. Testing all instances... New working instances:

It's an infinity loop. Only breaking the script works. There is no new working instances named (and used). Only thousand times the same message.

bocchilorenzo commented 1 year ago

Should be fixed, now if there are no working instances it will end the scraping without errors.

MadAim123 commented 1 year ago

Now there is another infinity loop:

Current stats for Bildung+vernetzt: 3 tweets, 0 threads...
Current stats for Bildung+vernetzt: 3 tweets, 0 threads...
Current stats for Bildung+vernetzt: 3 tweets, 0 threads...
Current stats for Bildung+vernetzt: 3 tweets, 0 threads...

Until it reaches the max number of retries or the IP is blocked: All instances are unreachable. Check your request and try again.

Can you try it for your own for the term "Bildung vernetzt"? Parameters:

The created link would be: https://nitter.ktachibana.party/search?f=tweets&q=Bildung+vernetzt&since=2022-12-25 The first tweet is always visible (on every page after click on "Load more").

zedeus commented 1 year ago

Perhaps they would stop going down if you stopped scraping them.

bocchilorenzo commented 1 year ago

This scraper does not abuse the instances. I limited it to 1 request at least every 2 seconds, you can see it in the code that it sleeps for 1-2 seconds on every search and then another 2 seconds before every retry. I hope such a low request frequency is not an issue.

thelollipopka commented 1 year ago

Same problem with infinite testing on version 0.2.9

scraper.get_tweets('#TESLA', mode='hashtag', number=256)

изображение