search

vladkens / twscrape

2024! Twitter API scrapper with authorization support. Allows you to scrape search results, User's profiles (followers/following), Tweets (favoriters/retweeters) and more.
https://pypi.org/project/twscrape/
MIT License
784 stars 102 forks source link

Twscrapes requests returned different tweets #162

Open OkFranco opened 2 months ago

OkFranco commented 2 months ago

Hello, first thanks for the great work.

Recently, I tried to make a search request to the API (api.search(q)) with q= 'ambiente sustentabilidade lang:pt since:2007-01-01 until:2022-12-31 ' (request#1). Afterwards I did the same request but from 2022-01-01 until 2022-12-31 (request #2).

Considering this context I got 2 issues: 1- The API returned some tweets with lang different from pt. This was not a big deal as during data treatment I can overcome this situation. Just an FYI: 2- The tweets returned in the request#2 are not all in the request#1. In addition, I got less tweets from request#1 than in request#2, which is quite strange. Considering this situation, I am having concerns if the API is indeed returning all tweets that validate the query (q) provided.

Thanks for the help!

atesahmet0 commented 2 months ago

Sometimes I have got tweets from different accounts when using a dork:"@From:username". It was like %1 of the tweets or so. I believe it is something related to Twitter. To overcome second issue you may fetch tweets by 1-2 days interval and fetch more than once to make sure you've got all the tweets. And make sure to set limit to 1000 or more than enough. If your result doesn't have more tweets than limit you should be fine.