search

taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 581 forks source link

Does anyone know why "Retrying" occurs in the middle of outputs? #306

Closed Kai292-tech closed 4 years ago

Kai292-tech commented 4 years ago

I followed the aforementioned one-line solution and was still confused on why "Retrying" occurs in the middle of outputs. The finality as well as whole process did not tell me any error and returned to "has more_items". However, I only got nearly a half of actual number. I guess I lost the "Retrying" pages but have no idea on why. Does anyone know the reason and solutions? Thanks a lot.

lapp0 commented 4 years ago

Retrying is shown when either 1) there is an error (failed to download, failed to parse, etc) or 2) there is no response

Counterintuitively, it appears that has_more_items will show if there aren't more items

                    if not has_more_items:
                        logger.info("Twitter returned : 'has_more_items' ")
                        return [], None

I'm able to replicate your issue of getting only half the tweets from a users page. I'm not sure why. I can't get the RELOAD_URL_USER to work in browser for me. Perhaps twitter has disabled the legacy user api.

I have tried query.py on --user andrewlapp0 and got 8 tweets. However with #302 (query_js.py) I got 21, which is the actual number of tweets (for retweets it has two status IDs, the original and the duplicate)

You could give #302 a whirl and see if its resolves the problems we've been seeing in other threads including https://github.com/taspinar/twitterscraper/issues/305 (closing because duplicate please post further questions, issues, etc in that thread)