Downloading from a certain Twitter account automatically stops at a certain point

[MarqFJA87]

I'm trying to download the media tweets of this account, but the app keeps stopping after about 420-something tweets (and this is after I specified in the config file to skip video tweets, since the user frequently tweets video reactions to replies and such). Confusingly, the app has no problem with downloading all applicable tweets from other accounts, such as this account's 600+ tweets (WARNING: plenty of NSFW material) or even this account's 1200+ tweets (AGAIN, FREQUENTLY HAS NSFW IMAGES).

For all I know, the problem isn't unique to this one account, but I've only started using this app with such "large" accounts last night.

[mikf]

The /media timeline only returns the latest ~1000 tweets with media files, but it more or less depends on Twitter's mood how many it actually returns. Skipping, i.e. ignoring, video tweets doesn't help in this regard, in only makes gallery-dl return even less files.

Your best bet is to combine results from /media, advanced search, and maybe even the regular timeline. So for account USER, you'd initially download from

• https://twitter.com/USER
• https://twitter.com/USER/media
• https://twitter.com/search?q=from:USER (*)

and use https://twitter.com/USER/media to keep up to date.

(*) you can add several other options to make this more precise and faster. For example filter:media, min_id:…, max_id:…, exclude:retweets, etc

[MarqFJA87]

The extra options for advanced search should be separated by a specific special symbol (? or &), right?

[mikf]

No, just with a space (or %20): "https://twitter.com/search?q=from:USER filter:media exclude:retweets"

If in doubt, go to https://twitter.com/search-advanced, fill out the form, and copy the URL from that.

[MarqFJA87]

I've been testing this method for the past few weeks, and it seems to be working well for the msot part. For several accounts, however, the "search?q=from:USER" part fails to return any results; I've tried manually searching for those accounts, and got no results as well, even though many such accounts have over 1000 tweets over the course of several months if not years. Anyone have any idea why this happens and whether this can be circumvented?

[mikf]

To get Tweets from accounts with "sensitive" content, you must be logged in and have "Hide sensitive content" (and "Remove blocked and muted accounts"?) disabled in your search settings. That's one of the reasons why gallery-dl doesn't use search results by default.

[MarqFJA87]

Having my username and password in the config file cannot substitute for being logged in? And why would "Remove blocked and muted accounts" affect my search settings when I haven't blocked or muted any of the relevant accounts?

[MarqFJA87]

Okay, I tried manually searching this NSFW-heavy Twitter account since the search result step in gallery-dl' once again failed, and even with "Hide sensitive content" and "Remove blocked and muted accounts" disabled, no search results are shown. I'm getting the feeling that there's some sort of "shadowban" going on.

[Hrxn]

The /media timeline only returns the latest ~1000 tweets with media files, but it more or less depends on Twitter's mood how many it actually returns. Skipping, i.e. ignoring, video tweets doesn't help in this regard, in only makes gallery-dl return even less files.

Your best bet is to combine results from /media, advanced search, and maybe even the regular timeline. So for account USER, you'd initially download from

• https://twitter.com/USER
• https://twitter.com/USER/media
• https://twitter.com/search?q=from:USER (*)

and use https://twitter.com/USER/media to keep up to date.

(*) you can add several other options to make this more precise and faster. For example filter:media, min_id:…, max_id:…, exclude:retweets, etc

I sure wish GitHub had a feature to basically bookmark single comments like this from anywhere of the site.. 😄

@MarqFJA87

There are definitely more than enough shenanigans going on with Twitter, but there's a site where you can check the specific account you are using with gallery-dl for any search- or ghostbans or whatever: https://shadowban.eu/

[mikf]

This is fixed with 3346f58a and all the changes/improvements following that.

[MarqFJA87]

This is fixed with 3346f58 and all the changes/improvements following that.

Wait, so does that mean that I don't need to use Hrxn's method above, and instead I just need to plug the account's default URL for gallery-dl to do it all on its own?

[mikf]

Kind of. Using the main profile URL now does /media (or /tweets) + /search and provides similar (and sometimes better) results as Twitter Media Downloader.

[MarqFJA87]

Excellent. That cuts down on the manual work that I have to do if I have multiple Twitter accounts I want to download from. Thank you very much.