search

taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 579 forks source link

always completely different result for same search #244

Open Ledgrev opened 4 years ago

Ledgrev commented 4 years ago

Hello!

I have noticed that I always get a completely different amount of scraped tweets for the same search. I am scraping all tweets with a certain hashtag - and I know there must several thousands at least in the case I'm looking at - from 01/01/2007 up until today. But every time I let Twitterscraper do its work, it gives me a completely different result.

The first time I ran my command twitterscraper #xxx --begindate 2007-01-01 --csv --output tweets_#xxx.csv it gave me about 500 tweets, the second time 2964. Then I left out the begindate and got 5067 results, I ran it again and got 3578 results. So then I changed the poolsize to 100 and got back 8985 results. This seemed like a realistic amount, but when I ran the exact same command later in order to check it, it only gave me 14 results!

It is really important for my research that I get every single tweet there is with this hashtag.

Could anyone help me to solve this problem? How can I make sure (and be sure/check) that twitterscraper really scraped ALL the tweets with the hashtag I'm looking for?

Thank you in advance!

parayamelo commented 4 years ago

Same issue here, working on a report and when I run the code again, I get totally different number of tweets. Any idea why>

Ledgrev commented 4 years ago

Hello @parayamelo I did not find a solution for the problem, but I did find a programme that worked better for me. I was able to scrape Twitter without any problem using mikhailklassen's Mining-the-Social-Web-3rd-Edition (https://github.com/mikhailklassen/Mining-the-Social-Web-3rd-Edition). Perhaps that might solve your problem as well?

parayamelo commented 4 years ago

Thanks, @ledgreve! Will check it out. Somebody also recommended me GetOldTweets3, https://github.com/Mottl/GetOldTweets3.

Thanks again!