search

sdelgadoc / download-tweets-ai-text-gen-plus

Python script to download public Tweets from a given Twitter account into a format suitable for AI text generation
MIT License
35 stars 8 forks source link

Fewer tweets than requested in the output #12

Open Jabernard075 opened 3 years ago

Jabernard075 commented 3 years ago

I've been following the steps in the readme to scrape tweets for a single user from the "Full Archive" dev environment.

When I ran a test of python3 download_tweets.py <username>100 The script ran and the bar filled from 0 to 100, but when I checked the output csv, there were only the 25 most recent tweets by the user.

Is this a bug or expected behavior because of the removal of replys and retweets?

sdelgadoc commented 3 years ago

It's most likely because by default the code does not include replies or tweet with links. To confirm that the code is working correctly, run the following command and check that you're getting the full 100 tweets.

python3 download_tweets.py <twitter_username> 100 True True

Let me know if that works for you.

Jabernard075 commented 3 years ago

I ran it with those two new arguments and now it returned 58 tweets, there's still something missing. Retweets?

sdelgadoc commented 3 years ago

Yes, even though the code counts a retweet as an iteration in the progress bar, the code does not return retweets. It only returns the tweets created by the account.

Can you check if the retweets account for the 42 missing tweets? I wasn't able to reproduce the issue on my side, so like to find the bug if there is one.