mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

[Twitter] 404 Error when trying to download SOME media tweets #4053

Closed MarqFJA87 closed 1 year ago

MarqFJA87 commented 1 year ago

For a while now, I keep having 404 Errors when I try to download SOME media tweets, but not others.

This one can be downloaded without issue, for example.

Below, however, is what I get when I try to download this one (rather NSFW). Strangely enough, my postprocessor for downloading a tweet's text into a .txt file works even when the program fails to download the media file.

[gallery-dl][debug] Version 1.25.4
[gallery-dl][debug] Python 3.10.0 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.7
[gallery-dl][debug] Configuration Files ['%USERPROFILE%\\gallery-dl\\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/ArcadiaMKII/status/1192329611608113152'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/ArcadiaMKII/status/1192329611608113152'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.twitter.com:443
[urllib3.connectionpool][debug] https://api.twitter.com:443 "GET /graphql/AV_lPTkN6Fc6LgerQpK8Zg/TweetDetail?variables=%7B%22focalTweetId%22%3A%221192329611608113152%22%2C%22referrer%22%3A%22profile%22%2C%22with_rux_injections%22%3Afalse%2C%22includePromotedContent%22%3Atrue%2C%22withCommunity%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withBirdwatchNotes%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%7D&features=%7B%22blue_business_profile_image_shape_enabled%22%3Afalse%2C%22responsive_web_twitter_blue_verified_badge_is_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22vibe_api_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22longform_notetweets_richtext_consumption_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 200 1991
[twitter][debug] Active postprocessor modules: [MetadataPP]
[downloader.ytdl][debug] [twitter] Extracting URL: https://twitter.com/i/web/status/1192329611608113152
[downloader.ytdl][debug] [twitter] 1192329611608113152: Downloading guest token
[downloader.ytdl][debug] [twitter] 1192329611608113152: Downloading JSON metadata
[downloader.ytdl][warning] [twitter] Twitter API gave 404 response, retrying with deprecated auth token. Only one media item can be extracted
[downloader.ytdl][debug] [twitter] 1192329611608113152: Downloading guest token
[downloader.ytdl][debug] [twitter] 1192329611608113152: Downloading JSON metadata
[downloader.ytdl][error] ERROR: [twitter] 1192329611608113152: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[download][error] Failed to download ytdl:https://twitter.com/i/web/status/1192329611608113152

Yes, I am using the latest version of both gallery-dl and yt-dlp.

mikf commented 1 year ago

I'd guess this is the same problem as in #4032.

Enabling forward-cookies while being logged in with gallery-dl seems to work.

$ gallery-dl https://twitter.com/ArcadiaMKII/status/1192329611608113152 -u USERNAME -o videos=ytdl
[downloader.ytdl][error] ERROR: [twitter] 1192329611608113152: Age-restricted adult content. This content might not be appropriate for people under 18 years old. To view this media, you’ll need to log in to Twitter. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies
[download][error] Failed to download ytdl:https://twitter.com/i/web/status/1192329611608113152

$ gallery-dl https://twitter.com/ArcadiaMKII/status/1192329611608113152 -u USERNAME -o videos=ytdl -o forward-cookies=true
/tmp/_/twitter/ArcadiaMKII/1192329611608113152_1.mp4
MarqFJA87 commented 1 year ago

It's working. Thanks!