search

sdelgadoc / download-tweets-ai-text-gen-plus

Python script to download public Tweets from a given Twitter account into a format suitable for AI text generation
MIT License
35 stars 8 forks source link

Getting "[!] if get this error but you know for sure..." error when trying to run the download tweets command #7

Open NoahSediqe opened 3 years ago

NoahSediqe commented 3 years ago

Having an issue running the python3 download_tweets.py <twitter_username> command in my current working directory

I pip installed everything and have the updated download_tweets.py and keys.py in my

This is what my terminal looks like running the example:

Noahs-MacBook-Pro-2:download-otp-tweets noahsediqe$ python3 download_tweets.py santiagodc
Retrieving tweets for @santiagodc...
0it [00:00, ?it/s]Expecting value: line 1 column 1 (char 0) [x] run.Feed
[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!
Expecting value: line 1 column 1 (char 0) [x] run.Feed
[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!
Expecting value: line 1 column 1 (char 0) [x] run.Feed
[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!
Expecting value: line 1 column 1 (char 0) [x] run.Feed
[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!
0it [00:25, ?it/s]
Noahs-MacBook-Pro-2:download-otp-tweets noahsediqe$
NoahSediqe commented 3 years ago

I came to this repo after digging through a rabbit hole of issues I had using https://github.com/minimaxir/download-tweets-ai-text-gen which ended up with a similar error as the one I have above

sdelgadoc commented 3 years ago

Unfortunately, Twitter has continued to implement changes that are making it harder for tools to collect information.

We believe that Twitter will continue making these types of changes so, even if we find a workaround, Twitter might make changes again.

Therefore, we are thinking about taking this code in a different direct that will fix the issues in a way that aligns with Twitter's long term strategy.

To help us guide this development, could you please fill out this super quick 3 question survey so we can decide where to move this code next? https://forms.gle/DvafJMUSJKhFvPoz7

albydarned commented 3 years ago

So is this project as well as the one it is based off of dead for now? Are there working alternatives?

sdelgadoc commented 3 years ago

Have not seen any working alternatives. At this point, what we're evaluating is whether it would be worth porting this code to the Twitter App API, which appears to be the long-term path Twitter wants developers to use.

If you want to chime in, let us know by filling out this survey. https://forms.gle/DvafJMUSJKhFvPoz7

albydarned commented 3 years ago

Thanks, I filled out the survey. That would be great. Are you aware of any working alternatives that use the Twitter App API?

sdelgadoc commented 3 years ago

Nope, but give me a couple of days and I'll let you know in this thread 😉

albydarned commented 3 years ago

Thanks so much!

sdelgadoc commented 3 years ago

Ok, I have good news, and bad news.

Good news is, I have a functioning version of the code only using the Twitter API for you to test. It is located in the development branch of this repo, which you can clone with the following command:

git clone -b development https://github.com/sdelgadoc/download-tweets-ai-text-gen-plus.git

Once you have cloned the repo, type your Twitter API authentication tokens into the keys.py file, and follow the instructions in the README file.

Now for the bad news, Twitter has now limited their API's search functionality to the last 7 days. So, you will only be able to retrieve tweets from the last week, which is not enough to train the model.

But, there's some more good news. Twitter released an API that allows you to search their full archive, which will let us collect enough tweets to train a model.

However, the full archive search API has pretty low request quotas in the free tier, so we do not want to waste those requests testing this code if we can do so with the old API with higher quotas.

So, if you could give the current code a whirl and let me know if you find any bugs, it would help me make sure it works for others. It has been working fine for me.

In parallel, I'm going to start porting the code to the full archive API and hope I don't reach the monthly quotas.

sdelgadoc commented 3 years ago

The final version of the code is complete, and can be found in the experimental branch, which you can clone with the following command:

git clone -b experimental https://github.com/sdelgadoc/download-tweets-ai-text-gen-plus.git

I have been using it, and think I have ironed out the biggest bugs, so it's working for me.

This version requires you to create an app and Development Environment. I have updated the README file to explain how to setup the Development Environment.

Give it a whirl and let me know what you think.

albydarned commented 3 years ago

Thanks! I'll try it out.

sdelgadoc commented 3 years ago

Found a bug that only allowed tweet collection for the past 30 days. The issue is fixed In the latest version of the experimental branch.

sdelgadoc commented 3 years ago

A number of folks seem to be using the code successfully, so I'm going push the experimental branch to the main branch. The current main branch doesn't work anymore, so if nothing else, this will be better.