search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.65k stars 2.72k forks source link

nothing returns "twint -u username" #901

Open ytaijp opened 3 years ago

ytaijp commented 3 years ago

OS: osx wsl debian gcloud-ubuntu

command: twint -u jack

nothing returns. just freeze. why...?

command: twint -s jack twint --all jack returns data.

Voyniq commented 3 years ago

A lot of the old configuration methods seem to deprecated now. Also it seems to be at least partially dependent on what shadowbans (shadowban.eu) a user might have or what time range you're trying to scrape, which makes it even more inconsistent.

At the moment I'm basically doing everything with the built in twitter filters. So I believe this should work: twint -s from:jack

Voyniq commented 3 years ago

Check out more filters here: https://developer.twitter.com/en/docs/twitter-api/v1/rules-and-filtering/overview "filter:links", "filter:quote", "filter:native_video", "filter:vine", and "filter:periscope" all seem to work consistently regardless of bans or time-range searched.

However, I've noticed that using filters like "example.com", "filter:images", "filter:twimg", "filter:videos", and/or "filter:media" (which also seems to be equivalent to setting config.Images, config.Videos, or config.Media to True) will cause Twint to not fetch certain tweets for certain accounts (whether it happens or not seems related to search suggestion bans?). Usually this means any tweets past ~2020-01, but occasionally tweets before then will be skipped too.

Using "url:example.com" in the search query (instead of just "example.com") seems to have no consistency issues. Using "images" by itself in the search query (instead of using "filter:images" or setting config.Images=True) seems to have no consistency issues. Same with using "videos" (instead of "filter:videos" or setting config.Videos=True). I haven't found any consistent alternatives to "filter:media" aside from just searching "images" and then "videos". Also haven't found a consistent alternative to "filter:twimg".

I think this all might have something to do with them preparing for a transition over to twitter API v2? https://developer.twitter.com/en/docs/twitter-api/early-access Which I think was supposed to happen in July or August but got postponed or something. If anyone has any insight on whether that's the case or if they know any other filters/workarounds that'd be appreciated.

ytaijp commented 3 years ago

Thank you so mush Voyniq.

mis-tech commented 3 years ago

Hi ytaijp! twint -u username and all commands like above giving me syntax error... any help from ur side means a lot.

bushjavier commented 3 years ago

Hi ytaijp! twint -u username and all commands like above giving me syntax error... any help from ur side means a lot.

@mis-tech that was answered:

twint -s "from:username" replaces twint -u username, works as expected

read this: https://github.com/twintproject/twint/issues/863