Got the same tweets - Githubissues

taspinar / twitterscraper

Scrape Twitter for Tweets

MIT License

2.4k stars 581 forks source link

Got the same tweets #185

Open JiayingCai9 opened 5 years ago

JiayingCai9 commented 5 years ago

The number of tweets I got was larger than the actual number. I searched it and found there were only 22 tweets, but the code returned 41. As I scraped the users' information, I compared them and found tweets were saved repeatedly.

This is my code. list_of_tweets = query_tweets("Tesla Sale", limit=None, begindate=dt.date(2019, 3, 13), enddate=dt.date(2019, 3, 14), poolsize=20, lang='')

The results are like:

AntonioNieto92 commented 5 years ago

same problem, got the same tweets repeated twice

chaojiang06 commented 5 years ago

same thing happened to me, is there any solution?

Thanks!

renatomello commented 5 years ago

The same issue is happening to me as well, especially while scraping one specific user

bchen32 commented 5 years ago

Yeah I'm getting duplicate tweets. Even worse, it doesn't always happen. The easiest way is to probably just save all of them and then process out duplicates later