Closed bsodxp closed 6 years ago
Hmm it seems that even adapting the mitigation applied for #66 , twint filter out some tweets. So I removed that and now it works. Unfortunately the problem is from the Twitter side, the mitigation (read check date) will be moved somewhere else.
This means that if one is looking for tweets since:2017-10-02 he/she will get some tweets since:2017-10-01 (not so much).
For a more detailed explanation of the problem see #66
@bsodxp now twint should work as expected
I am scraping an entire timeline of a user w/ 150K+ tweets. The output runes fine for 8-12k tweets (usually about 30 days), and then stops.
I'm starting to think this isn't an issue w/ the script, but probably a user's connection issues. I just scraped someone's feed w/ 150k tweets and had no issues
There were a problem with the script when the difference between until
and since
is less than default timedelta
. There is a problem, Twitter side, with the search: Twitter returns tweets before the since date.
I think that if the issue would be with the connection, I'll expect timeout
errors or stuff like this, but this was not reported. I was able to reproduce the conditions such for which the issue is expected to happen. Now it works as expected, at least this is what I'm having
Yes, you are right... maybe we should try handling connection issues (maybe ClientConnectionError
) and see what happens
First of all, thank you for such a great tool!
Description of Issue
I am scraping an entire timeline of a user w/ 150K+ tweets. The output runes fine for 8-12k tweets (usually about 30 days), and then stops. I have seen previous reports of scraping stopping, but I'm not sure if they are related. I have been adding the --until flag and working through approx 30 - 60 days at a time. Is this a bug, user error, or a Twitter limitation? Is there an easy way to batch commands together and run in chucks w/ --since and --until boundaries?
OS Details
Ubuntu / Buscador OSINT
Initial Check
Make sure you've checked the following.
Command
python3 twint.py -u username -o timeline.csv --csv
Once it fails, I then add a --until flag: python3 twint.py -u username --until 2017-06-01 -o timeline.csv --csv
Thank you!