search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.56k stars 949 forks source link

[Twitter] Auth error. gives 404 #4453

Open gthu34 opened 1 year ago

gthu34 commented 1 year ago

I download images from some twitter account. I passed it cookies.txt location in config but it is throwing [twitter][error] 404 Not Found () . Then I tried giving it username & password but still same error. After that I gave auth_token in cookie but still no luck. The only way it works for me is to pass -u username -p password along with url which is annyoing for numerous uses. Same cookie.txt is working fine for yt-dlp but not for gallery-dl. This issue is on T ermux. But it is working fine for gallery-dl on windows Wsl with cookies. 

"twitter":
        {
             "cookies": "$home/storage/shared/Ϟ/twittercookies.txt",
            "cards": false,
            "conversations": false,                                                                                                                                 
            "pinned": false,
            "quoted": false,
            "replies": true,
            "retweets": true,
             "size": ["small"],
            "strategy": null,
            "text-tweets": false,
            "twitpic": false,
            "unique": true,
            "users": "timeline",
            "videos": false,
             "filename": "{author[name]}_{tweet_id}_{num}_{count}.{extension}"
        },

here is verbose output


[gallery-dl][debug] Python 3.11.4 - Linux-4.14.113-20747890-aarch64-with-libc
[gallery-dl][debug] requests 2.29.0 - urllib3 1.26.15
[gallery-dl][debug] Configuration Files ['${HOME}/.config/gallery-dl/config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/culturaltutor'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/culturaltutor'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/XA6F1nJELYgC2Ekmg/UserByScreenName?variables=%7B%22screen_name%22%3A%22culturaltutor%22%2C%22withSafetyModeUserFields%22%3Atrue%7D&features=%7B%22hidden_profile_likes_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D HTTP/1.1" 200 1132
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/lo965xe1Jc-W_A/UserMedia?variables=%7B%22userId%22%3A%221524407286910930944%22%2C%22count%22%3A100%2C%22includePromotedContent%22%3Afalse%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22rweb_lists_timeline_redesign_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 404 0
[twitter][error] 404 Not Found ()
Hrxn commented 1 year ago

[..] . Same cookie.txt is working fine for yt-dlp but not for gallery-dl. This issue is on T ermux. But it is working fine for gallery-dl on windows Wsl with cookies.

Well, then it is maybe not actually an issue with gallery-dl.. Without any chance to replicate the issue at hand, I'm afraid but there isn't much that can be done here.

mikf commented 1 year ago

here is verbose output

This is missing a line about it loading cookies, meaning it doesn't load anything from $home/storage/shared/Ϟ/twittercookies.txt and it probably doesn't even see any of your twitter options.

Here's how it should look like.

[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/USER/timeline'
[twitter][debug] Loading cookies from '/home/mike/cookies-twitter-com.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
gthu34 commented 1 year ago

I again tried giving it auth_token in cookies as gallery-dl isnt picking up location of cookies.txt . again same error. Then I moved the cookies.txt to location of config.json & image history file archive.sqlite3 but again same error. Other config options of twitter works fine like custom filename when using with any single tweet. client_id client_secret and refresh_token given to Devianart and Pixiv works fine for me but giving username and password to twitter in config doesnt. It also give same error. here config using auth_token


"twitter":
        {
            "username": null,
            "password": null,
            "cookies": [
                {
                        "auth_token":"REDACTED"
                }
            ],
            "cards": false,
            "conversations": false,
            "pinned": false,
            "quoted": false,
            "replies": true,
            "retweets": false,
                "size": ["small"],
            "strategy": null,
            "text-tweets": false,
            "twitpic": false,
            "unique": true,
            "users": "timeline",
            "videos": false,
                "filename": "{author[name]}_{tweet_id}_{num}_{count}.{extension}"

vetbose output for this

gallery-dl --verbose https://twitter.com/culturaltutor
[gallery-dl][debug] Version 1.25.8
[gallery-dl][debug] Python 3.11.4 - Linux-4.14.113-20747890-aarch64-with-libc
[gallery-dl][debug] requests 2.29.0 - urllib3 1.26.15
[gallery-dl][debug] Configuration Files ['${HOME}/.config/gallery-dl/config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/culturaltutor'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/culturaltutor'
[twitter][info] Requesting guest token
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.twitter.com:443
[urllib3.connectionpool][debug] https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/XA6F1nJELOC2Ekmg/UserByScreenName?variables=%7B%22screen_name%22%3A%22culturaltutor%22%2C%22withSafetyModeUserFields%22%3Atrue%7D&features=%7B%22hidden_profile_likes_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D HTTP/1.1" 200 1130
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/lo9SM1Jc-W_A/UserMedia?variables=%7B%22userId%22%3A%221524407286910930944%22%2C%22count%22%3A100%2C%22includePromotedContent%22%3Afalse%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22rweb_lists_timeline_redesign_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 404 0
[twitter][error] 404 Not Found ()

verbose output for a single tweet

$ gallery-dl --verbose https://twitter.com/tiraniddo/status/1694115196032217467
[gallery-dl][debug] Version 1.25.8
[gallery-dl][debug] Python 3.11.4 - Linux-4.14.113-20747890-aarch64-with-libc
[gallery-dl][debug] requests 2.29.0 - urllib3 1.26.15
[gallery-dl][debug] Configuration Files ['${HOME}/.config/gallery-dl/config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/tiraniddo/status/1694115196032217467'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/tiraniddo/status/1694115196032217467'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/2ICDtulZcYrtpTuQ/TweetResultByRestId?variables=%7B%22tweetId%22%3A%221694115196032217467%22%2C%22withCommunity%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withVoice%22%3Afalse%7D&features=%7B%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Afalse%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22responsive_web_media_download_video_enabled%22%3Afalse%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticleRichContentState%22%3Afalse%7D HTTP/1.1" 200 1708
[twitter][debug] Using download archive '/data/data/com.t ermax/files/home/.config/gallery-dl/pngdb.sqlite3'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): pbs.twimg.com:443
[urllib3.connectionpool][debug] https://pbs.twimg.com:443 "GET /media/F4K0gYMWgAAkqsU?format=jpg&name=orig HTTP/1.1" 200 240547
Hrxn commented 1 year ago

[..] Other config options of twitter works fine like custom filename when using with any single tweet. client_id client_secret and refresh_token given to Devianart and Pixiv works fine for me but giving username and password to twitter in config doesnt. It also give same error. here config using auth_token

"twitter":
        {
            "username": null,
            "password": null,
            "cookies": [
                {
                        "auth_token":"REDACTED"
                }
            ]

You have "auth_token" in an object in a JSON array here, but it should simply be an object, i.e.

"cookies": 
{
    "cookie-name": "cookie-value"
}

And it should definitely pick up a cookies.txt file.. If you are sure that the path is correct, maybe you've changed some file permissions or access modes? Maybe chmod ed to some wrong setting?

mikf commented 1 year ago 
            "cookies": [
                {
                        "auth_token":"REDACTED"
                }
            ],

This would cause a crash if it actually got used, but, again, your twitter settings do not get applied for whatever reason. Maybe you closed the extractor block prematurely; wouldn't be the first time that happened to someone.

If you are sure that the path is correct, maybe you've changed some file permissions or access modes? Maybe chmoded to some wrong setting?

There would be an error message if gallery-dl was given a cookies.txt path but couldn't read the file.

[twitter][warning] cookies: [Errno 2] No such file or directory: 'cookies.txt'
[twitter][warning] cookies: [Errno 13] Permission denied: 'cookies.txt'
gthu34 commented 1 year ago

I did that before but changed it to see if something happens. I again fixed it but no effect.

"cookies":
                {
                        "auth_token":"REDACTED"
                },

I checked path name , filenames numerous times. nothing wrong. Now archive file, config file and cookie file all have the same location. But only archive and config file are picked up. :E

You have "auth_token" in an object in a JSON array here, but it should simply be an object, i.e.

"cookies": 
{
    "cookie-name": "cookie-value"
}

And it should definitely pick up a cookies.txt file.. If you are sure that the path is correct, maybe you've changed some file permissions or access modes? Maybe chmod ed to some wrong setting?

gthu34 commented 1 year ago

I am not sure what is closing "extractor block prematurely" is but I never fiddled with source files of Gallery-dl . Though I used --clear-cache twitter to clean the cache after I passed -u username and -p password to gallery-dl with url in terminal. only this works. passing anything in config doesnt.

This would cause a crash if it actually got used, but, again, your twitter settings do not get applied for whatever reason. Maybe you closed the extractor block prematurely; wouldn't be the first time that happened to someone.

Hrxn commented 1 year ago

I am not sure what is closing "extractor block prematurely" is but I never fiddled with source files of Gallery-dl . [..]

Got nothing to do with gallery-dl's source code.

The outline of a config is basically this:

{
    "extractor":
    {

    },

    "downloader":
    {

    },

    "output":
    {

    },

    "postprocessor":
    {

    }

}

You must make sure that your extractor options, for Twitter in your case, that is the entire "twitter": { ... } object must be inside of "extractor": { .. }

gthu34 commented 1 year ago

I am truly sorry for wasting time of you guys. The issue doesnt exist. T ermux and Gallery-dl is fine. It was me dumbo who was editing wrong file at wrong location. The real config.json is located at $HOME/.config but I was editing another config.json at $HOME/.cache . 🤣😭🤡 I am ashamed.

apparently Gallery-dl creates this copy of config file in /.cache along with cache.sqlite3 . I noticed this when I saw cache.sqlite3 but not my ArchiveDb.sqlite3 in so-called config directory.

Last time I edited config.json on T ermux months ago. so pretty much forgot everything. I CDed to $HOME and started checking every directory and found config.json in ./cache first and started editing it without giving much thought or checking further.

Now everything is fine btw. it recives cookie , auth_token and username-password finely. Thanks a lot to you guys for helping me.

mikf commented 1 year ago

apparently Gallery-dl creates this copy of config file in /.cache along with cache.sqlite3

It only creates ~/.cache/gallery-dl/cache.sqlite3 automatically.

The only way to get it to create a config file is with --config-create and that'd put it at ~/.config/gallery-dl/config.json.

gthu34 commented 1 year ago

I assumed wrong 😅 ; I guess months ago I moved that config.json into .cache as I was trying every possible location to get the config work. btw thanks for the all help.