[Twitter] [X] "AuthorizationError: user's Tweets are protected" even when using browser cookies

[AmperAndSand]

Description

I'm encountering difficulties while attempting to download media from a protected X (formerly Twitter) account. Despite being an approved follower, gallery-dl is unable to access the content even if it's using my logged-in browser cookies. I've confirmed my follower status by successfully viewing the user's media through Firefox.

Steps to Reproduce

1. Ensure follower status on the protected X account using the Firefox browser
2. Execute the following command:

gallery-dl -v --dest "C:\rips"  --filter "datetime(2023, 5, 16) <= date < datetime(2024, 8, 8)" --cookies-from-browser "Firefox" "https://x.com/user/media"

Note: The actual X username has been replaced with "user" for privacy.

Observed Behavior

The command produces the following output:

[gallery-dl][debug] Version 1.27.2
[gallery-dl][debug] Python 3.10.6 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.1
[gallery-dl][debug] Configuration Files ['%APPDATA%\\gallery-dl\\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://x.com/user/media'
[twitter][debug] Using TwitterMediaExtractor for 'https://x.com/user/media'
[cookies][debug] Extracting cookies from C:\Users\Windowsprofilename\AppData\Roaming\Mozilla\Firefox\Profiles\h5l9u78i.default-release\cookies.sqlite
[cookies][info] Extracted 2013 cookies from Firefox
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/k5XapwcSikNsEsILW5FvgA/UserByScreenName?variables=%7B%22screen_name%22%3A%22user%22%2C%22withSafetyModeUserFields%22%3Atrue%7D&features=%7B%22hidden_profile_likes_enabled%22%3Atrue%2C%22hidden_profile_subscriptions_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22responsive_web_twitter_article_notes_tab_enabled%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22subscriptions_verification_info_is_identity_verified_enabled%22%3Atrue%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%7D HTTP/1.1" 200 778
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/tO4LMUYAZbR4T0SqQ85aAw/UserMedia?variables=%7B%22userId%22%3A%224837652664%22%2C%22count%22%3A100%2C%22includePromotedContent%22%3Afalse%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22rweb_video_timestamps_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_media_download_video_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 200 45
[twitter][debug] {'user': {}}
[twitter][error] AuthorizationError: user's Tweets are protected

Note: The actual X username has been replaced with "user" for privacy.

Expected Behavior

Given my approved follower status, gallery-dl should successfully download the media from the protected account.

[fireattack]

I cannot reproduce this issue with a protected account I follow. Everything just works.

Can you double check in your Firefox that you can view her tweets, /media page fine, and you're indeed using the profile folder \Firefox\Profiles\h5l9u78i.default-release? (You can check usingabout:support.)

[AmperAndSand]

@fireattack

Yes, I can view the user's tweets and /media page normally using Firefox.

about:support states that may Profile Folder is at C:\Users\Windowsprofilename\AppData\Roaming\Mozilla\Firefox\Profiles\h5l9u78i.default-release.

I can also download from private Instagram profiles with no issues using Firefox browser cookies.

[fireattack]

Thanks.

Then I have no clue why it doesn't work. Maybe try the following:

1. update to newest master (instruction in readme)
2. Export the cookies of x.com using Firefox (using something like this addon, or this) to a text file and then try with --cookies path/to/file, and see if that makes a difference.

At this point I don't know which part is causing the problem so we would like to control the variables first.

[aonecho]

I have the same problem for 3 days now. (v1.27.2) Today I updated to v1.27.3 and the symptoms are still the same. However, I exported the cookies and ran "gallery-dl URL --cookies cookie.txt" and the download started normally. I have tried multiple accounts and all have the same symptoms. (Translated by DeepL)

[fireattack]

@aonecho Are you also using Firefox? I can't reproduce it myself even on Firefox, but that would help us narrowing down. Like maybe it's related to some of its privacy features (as well as the use of Containers).

It also helps if you guys can create a new profile for whatever browser you're using and try with that (--cookies-from-browser firefox:profilename) or container (firefox::containername, also try firefox::none).

[aonecho]

@fireattack I use Firefox as well. You are certainly right, Containers seems to be the cause of the problem! Just a few days ago I created another account on twitter using Containers. And furthermore, now I added firefox::none and it started downloading successfully. Thank you very much!

I created a new profile and verified it. As for firefox:profilename, if I specify the profile where I am logged in to Twitter, the download started successfully. But then, if I added a new Container and logged into Twitter from a different account in it (it cannot view protected accounts), the output was Tweets are protected.

As for firefox::containername, I specified ::none and it downloaded successfully. In addition, if I add a new Container (TEMP) and execute firefox::NEWPROFILE::TEMP after logging in to Twitter in the Container, it was successfully downloaded.

I hope this will be helpful for you.

Translated with www.DeepL.com/Translator (free version)

[AmperAndSand]

And furthermore, now I added firefox::none and it started downloading successfully.

@aonecho Thanks! --cookies-from-browser "Firefox::none" worked. Although I'm not sure why --cookies-from-browser "Firefox" didn't work since I'm using the default container for the Twitter cookies I want to use.

[fireattack]

JFYI, -cookies-from-browser "Firefox" worked for me but I never used Container at all in Firefox.

Maybe there is a difference between this and "default" container, whatever that means,.

[AmperAndSand]

Maybe there is a difference between this and "default" container, whatever that means,.

What I call the default container is essentially the one you're using when you first open Firefox—it's where your browser cookies are stored if you're not using multiple containers. I guess it can also be called the default profile.

I've always used Firefox with containers, and Twitter cookies have worked fine for me in the past. This is actually the first time I've run into this issue with gallery-dl. Usually, when you don't specify a container, gallery-dl uses the cookies from the default profile/container. So this behavior is pretty unexpected, based on my experience.

JFYI, -cookies-from-browser "Firefox" worked for me but I never used Container at all in Firefox.

It's possible there's a bug specifically related to container usage.

[fireattack]

Passing --cookies-from-browser firefox and --cookies-from-browser firefox::none should not make ANY difference in term of behaviors.

Please try to run the command with --verbose for both and observe the log.

It should both say "[cookies][info] Extracted xxxx cookies from Firefox" with the exact same number xxxx.

If one works but the other does not, please paste the full log for both.

[mikf]

Passing --cookies-from-browser firefox and --cookies-from-browser firefox::none should not make ANY difference in term of behaviors.

firefox::none extracts only cookies which are not part of a container. firefox extracts all cookies, which is equivalent to firefox::none + cookies from all containers.

Maybe the problem is that firefox combines the cookies from several login sessions?

[fireattack]

Passing --cookies-from-browser firefox and --cookies-from-browser firefox::none should not make ANY difference in term of behaviors.

firefox::none extracts only cookies which are not part of a container. firefox extracts all cookies, which is equivalent to firefox::none + cookies from all containers.

Maybe the problem is that firefox combines the cookies from several login sessions?

Ah my bad, I read it too quickly. I didn't realise one is container_id == False and the other is container_id == None.

[fireattack]

Then it probably is the reason -- if the user have multiple accounts in different containers, they overwrite each other.

I think we should set ::none as default behavior then, and having something like ::all for reading all. Because using all the containers (and therefore causing potential conflict) by default surely is not the users would expect, at least not for users that use containers to login multiple social media accounts.

For the same reason we don't by default read all the profiles the users' browser(s) have, basically.

[mikf]

I think we should set ::none as default behavior then, and having something like ::all for reading all.

Sounds like a really good idea.

Because using all the containers (and therefore causing potential conflict) by default surely is not the users would expect, at least not for users that use containers to login multiple social media accounts.

In my defense, extracting all cookies across all containers is also yt-dlp's default behavior.

[mikf]

Done: https://github.com/mikf/gallery-dl/commit/2d5552158213f546177e1c2a77d0481ce01188b2

[AmperAndSand]

firefox::none extracts only cookies which are not part of a container. firefox extracts all cookies, which is equivalent to firefox::none + cookies from all containers.

Maybe the problem is that firefox combines the cookies from several login sessions?

That clears things up! So gallery-dl was mixing cookies from all the containers. I also noticed you've made 'none' the default option now. That's a helpful change!