sdelgadoc
commented
3 years ago I tried everything I could think of to get the code to work without creating a Twitter app, but was unable to. Also, it looks like Twitter is trying to move developers to use their standard Twitter API, and making scraping harder. So, I do not know of a way to collect enough tweets to train a model other than download-tweets-ai-text-gen-plus repo.
However, @minimaxir is much smarter than I am, so he might have figured something out.
I will update issue https://github.com/minimaxir/download-tweets-ai-text-gen/issues/28 to clarify that the workaround no longer works.
According to @sdelgadoc on #33, "it looks like Twitter is making it harder for scraping tools and developing more robust tweet collection APIs." Likewise, it seems like I cannot get the download_tweets.py script to run correctly, as I keep encountering an index error (see #28), and then another error where when a number of tweets to be downloaded is specified (see #33). Does this mean that the scraping technique used on this repo is now non-functional and @sdelgadoc's Twitter-API dependent repo, linked here, must be used? If so, are there any plans to modify this script such that it can still be used, and such that I will not need to apply to create a Twitter app and thus gain access to Twitter's API?
To be more specific, I've been experiencing the same issue from #33 in attempting to overcome the issues with twint.run.Lookup failing to function correctly and causing an index list error as specified in #28. I have tried the fix specified by @sdelgadoc on #28 here by using his version of download_tweets.py in this repo to no avail. It seems like the only other potential fix he specified involves using the new Twitter API version linked earlier, download-tweets-ai-text-gen-plus.
I apologize in advance for any confusion or inconvenience this issue may provoke! Thank you all for your hard work on this open-source project!