mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.91k stars 976 forks source link

Which settings to avoid getting forcibly logged out/suspended from Twitter? #5775

Open cheatreal321 opened 4 months ago

cheatreal321 commented 4 months ago

So I updated to the latest version 1.27.0 this week and tried to download a bunch of Twitter accounts again with the same config from roughly 5 months ago. After hitting the rate limit a few times, Twitter forcibly logged me out everywhere and I got the known 401 Unauthorized (Could not authenticate you) error cause I was only using browser cookies to log in. This I was able to work around by logging back into my account and adding my username and password to the config for the relogin option to work. However after getting forcibly logged out again and using the gallery-dl relogin a few times, my account was temporarily locked and I had to do the Arkose challenge. I then cleared cache for twitter and added a delay with "sleep": 1, "sleep-request": 1, which was enough to prevent ever hitting rate limit, but I was still eventually logged out, and after triggering the relogin multiple times my account was suspended for trying to evade suspension.

I just got my account unlocked and I still want to download everything I can, but want to avoid having any further troubles. What settings can/should I use to prevent getting forcibly logged out and then suspended? I saw the comments in #5532 saying that upping sleep and sleep-request to 2 seconds is enough, but how reliable is that? Also, could a Premium subscription increase the rate limit at all?

For reference here's my current config that got my account suspended:

{
    "extractor":{
    "cookies": ["firefox"],
        "twitter":{
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "username": "username",
            "password": "password",
        "base-directory": "G:/twitter_gallery-dl/",
        "users": "https://twitter.com/{legacy[screen_name]}",
            "text-tweets":false,
            "quoted":true,
            "retweets":true,
            "replies":true,
            "logout":true,
        "relogin":true,
        "locked": "wait",
        "ratelimit": "wait:1800",
        "sleep": 1,
        "sleep-request": 1,
            "filename": "{author['name']}-{tweet_id}-{date:%Y%m%d_%H%M%S}-0{num}.{extension}",
            "directory":{
                "quote_id   != 0": ["twitter", "{quote_by}"  , "quote-retweets"],
                "retweet_id != 0": ["twitter", "{user[name]}", "retweets"  ],
                ""               : ["twitter", "{user[name]}"              ]
            },
            "postprocessors":[
                {
            "name": "mtime",
                    "key": "date"
                },
                {   
                    "name": "metadata",
                    "event": "post",
                    "filename": "{author['name']}-{tweet_id}-{date:%Y%m%d_%H%M%S}.json"
                }
            ]
        }
    },
    "downloader": {

    },
    "output": {

    },
    "postprocessor": {

    }
}
a84r7a3rga76fg commented 4 months ago

What did you tell Twitter to get unbanned? I'm also getting banned for evading when I get rate limited.

These are my settings

            "include": ["avatar", "background", "timeline", "media"],
            "image-filter": "author is user",
            "sleep-request": 30,
            "sleep": 5,
            "quoted": true,
            "locked": "wait",
            "twitpic": true,
            "videos": true,
            "cards": false,
            "expand": false,
            "pinned": true,
            "unique": true,
            "skip": true,
            "metadata": true,
cheatreal321 commented 4 months ago

I wrote something short like "Hi, I seem to have been mistakenly suspended cause of an issue with a browser addon that was constantly logging in and out of my account. I thought the emails notifying me of a new login were an error and didn't react in time. I've removed the addon now so it shouldn't happen again. Please unlock the account. Thank you."

a84r7a3rga76fg commented 4 months ago

I wrote something similar when I got banned months ago, never got a reply back. It seems like Twitter is doing collateral punishment by banning every single person for "ban evasion" after getting rate limited and sharing the same IP address as a banned user because I haven't gotten banned after getting rate limited while using a residential IP address.

harshbhardwaj77 commented 4 months ago

I wrote something similar when I got banned months ago, never got a reply back. It seems like Twitter is doing collateral punishment by banning every single person for "ban evasion" after getting rate limited and sharing the same IP address as a banned user because I haven't gotten banned after getting rate limited while using a residential IP address.

You haven't gotten banned after using "resident IP" or home IP ? Were you using VPN before ? I am facing the same problem. If i don't use VPN , will it help me not get logged out and eventually banned ??

jojo2833432 commented 4 months ago

Is there even a solution to this? I have gotten 3 accounts banned on twitter. even with the -sleep 3-10 and -sleep-request 3-10.

Hrxn commented 4 months ago

Yes, do not use a VPN, a VPS, or something like that.

Use your residential ISP (your "home" ISP). The only other thing you can do is to use your account as "realistic" as possible.

Anti-bot measures are very aggressive.

jojo2833432 commented 4 months ago

Yes, do not use VPN, a VPS, or something like that.

Use your residential ISP (your "home" ISP). The only other thing you can do is to use your account as "realistic" as possible.

  • Add an avatar and maybe a header picture, don't keep the predefined defaults. Add a short bio or description text.
  • Verify your email in the options
  • Maybe also verify via mobile number, use 2FA, use the mobile App
  • Use the site normally once in a while. Post something, like something, repost something, search something etc.

Anti-bot measurements are very aggresive.

Ah, I see. I probably was being blatant about the 3 account being throwaways. thanks!

cheese529 commented 4 months ago

@cheatreal321 my account just got permanently suspended as well for using gallery-dl. sent an appeal to them, any idea on how long it took you to get the account back and was all your account data such as direct messages and likes still there after they restored it?

06000208 commented 3 months ago

The account I use with gallery-dl usually just gets locked and I have to pass the arkrose challenge manually, but now it's been suspended. I'll update if I get it back.

jojo2833432 commented 3 months ago

The account I use with gallery-dl usually just gets locked and I have to pass the arkrose challenge manually, but now it's been suspended. I'll update if I get it back.

Yeah I've tried to use an alt-account like a "human" would use. Like: I'd post comments on random tweets/make random tweets about any subject I can think of and still got hit with a suspension.

cheatreal321 commented 3 months ago

The account I use with gallery-dl usually just gets locked and I have to pass the arkrose challenge manually, but now it's been suspended. I'll update if I get it back.

Yeah I've tried to use an alt-account like a "human" would use. Like: I'd post comments on random tweets/make random tweets about any subject I can think of and still got hit with a suspension.

My guess is that Twitter flags your IP if you've gotten suspended or hit with the Arkose challenge at least once, and you need to create and use any alt accounts from a different IP address to not trigger that system.

ForxBase commented 2 months ago

Anyone who doesn't get suspended?

Twi-Hard commented 2 months ago

Anyone who doesn't get suspended?

I download Twitter 24/7 with a low sleep setting. Before I made the time between each request random (using a range) I had to have a lot higher sleep between each request. This is using my home IP with my actual twitter account I use. After I lowered the sleep and made it a range I've recieved no rate limiting at all. Before the change I'd get told to wait until a certain time before continuing. This is a little confusing to me because I'm probably making a lot more requests per day than Elon allocates to free accounts. I'm not home so I can't tell you the settings yet. I actually copied them from somewhere else in gallery-dl's issues.

cheatreal321 commented 2 months ago

I download Twitter 24/7 with a low sleep setting. Before I made the time between each request random (using a range) I had to have a lot higher sleep between each request. This is using my home IP with my actual twitter account I use. After I lowered the sleep and made it a range I've recieved no rate limiting at all. Before the change I'd get told to wait until a certain time before continuing. This is a little confusing to me because I'm probably making a lot more requests per day than Elon allocates to free accounts. I'm not home so I can't tell you the settings yet. I actually copied them from somewhere else in gallery-dl's issues.

Would still love to see those settings cause I got suspended despite using ranges at high values up to 30 seconds.

Twi-Hard commented 2 months ago

I download Twitter 24/7 with a low sleep setting. Before I made the time between each request random (using a range) I had to have a lot higher sleep between each request. This is using my home IP with my actual twitter account I use. After I lowered the sleep and made it a range I've recieved no rate limiting at all. Before the change I'd get told to wait until a certain time before continuing. This is a little confusing to me because I'm probably making a lot more requests per day than Elon allocates to free accounts. I'm not home so I can't tell you the settings yet. I actually copied them from somewhere else in gallery-dl's issues.

Would still love to see those settings cause I got suspended despite using ranges at high values up to 30 seconds.

-o "sleep=[1.5,5]"
-o "sleep-request=[6.0,12.0]"
WarmWelcome commented 2 months ago

I download Twitter 24/7 with a low sleep setting. Before I made the time between each request random (using a range) I had to have a lot higher sleep between each request. This is using my home IP with my actual twitter account I use. After I lowered the sleep and made it a range I've recieved no rate limiting at all. Before the change I'd get told to wait until a certain time before continuing. This is a little confusing to me because I'm probably making a lot more requests per day than Elon allocates to free accounts. I'm not home so I can't tell you the settings yet. I actually copied them from somewhere else in gallery-dl's issues.

Would still love to see those settings cause I got suspended despite using ranges at high values up to 30 seconds.

-o "sleep=[1.5,5]"
-o "sleep-request=[6.0,12.0]"

Mine is 15-45 seconds sleep, it still triggers lockout :)