[downloader.ytdl][error] Cannot import module 'yt_dlp'

alscaldas commented 3 weeks ago

Hello,

Each time I try to download a video from a couple of sites (the ones I more recently tested in order to get enough data to open this issue were Reddit and Twitter) I got the following error:

When downloading from Reddit:

[user@archvm ~]$ gallery-dl --verbose --config $HOME/.config/gallery-dl/config.json https://www.reddit.com/r/hyprland/comments/1d7bbbo/i_made_a_module_that_shows_internet_speed_but_it/
[gallery-dl][debug] Version 1.27.0
[gallery-dl][debug] Python 3.12.3 - Linux-6.9.2-arch1-1-x86_64-with-glibc2.39
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.1
[gallery-dl][debug] Configuration Files ['${HOME}/.config/gallery-dl/config.json', '/home/user/.config/gallery-dl/config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.reddit.com/r/hyprland/comments/1d7bbbo/i_made_a_module_that_shows_internet_speed_but_it/'
[reddit][debug] Using RedditSubmissionExtractor for 'https://www.reddit.com/r/hyprland/comments/1d7bbbo/i_made_a_module_that_shows_internet_speed_but_it/'
[reddit][debug] Using custom API credentials (client-id HF8-h*********)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): oauth.reddit.com:443
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /comments/1d7bbbo/.json?limit=0&raw_json=1 HTTP/1.1" 200 5491
[reddit][debug] Using download archive '/home/user/.config/gallery-dl/arquivo.db'
[downloader.ytdl][error] Cannot import module 'yt_dlp'
[downloader.ytdl][debug] 
Traceback (most recent call last):
  File "/home/user/.local/share/pipx/venvs/gallery-dl/lib/python3.12/site-packages/gallery_dl/downloader/ytdl.py", line 44, in download
    module = ytdl.import_module(self.config("module"))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/gallery-dl/lib/python3.12/site-packages/gallery_dl/ytdl.py", line 23, in import_module
    return __import__(module_name.replace("-", "_"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'yt_dlp'
[download][error] Failed to download ytdl:https://www.reddit.com/r/hyprland/comments/1d7bbbo/i_made_a_module_that_shows_internet_speed_but_it/

When downloading from Twitter:

[user@archvm ~]$ gallery-dl --verbose --config $HOME/.config/gallery-dl/config.json https://x.com/FAFO_TV/status/179802189981487937                   
[gallery-dl][debug] Version 1.27.0                                                                                                                              
[gallery-dl][debug] Python 3.12.3 - Linux-6.9.2-arch1-1-x86_64-with-glibc2.39
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.1
[gallery-dl][debug] Configuration Files ['${HOME}/.config/gallery-dl/config.json', '/home/user/.config/gallery-dl/config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://x.com/FAFO_TV/status/179802189981487937'
[twitter][debug] Using TwitterTweetExtractor for 'https://x.com/FAFO_TV/status/179802189981487937'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/MWY3AO9_I3rcP_L2A4FR4A/TweetResultByRestId?variables=%7B%22tweetId%22%3A%22179802189981487
937%22%2C%22withCommunity%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withVoice%22%3Afalse%7D&features=%7B%22responsive_web_graphql_exclude_directive
_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timel
ine_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22c9s_tweet_anatomy_moderator_badge_enable
d%22%3Atrue%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet
_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_
twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C
%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22rweb_video_timestamps_enabl
ed%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_media_downloa
d_video_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 200 52
[twitter][error] An unexpected error occurred: KeyError - 'result'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue o
n https://github.com/mikf/gallery-dl/issues .
[twitter][debug] 
Traceback (most recent call last):
  File "/home/user/.local/share/pipx/venvs/gallery-dl/lib/python3.12/site-packages/gallery_dl/job.py", line 150, in run
    for msg in extractor:
  File "/home/user/.local/share/pipx/venvs/gallery-dl/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 87, in items
    for tweet in self.tweets():
  File "/home/user/.local/share/pipx/venvs/gallery-dl/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 779, in _tweets_single
    tweet = self.api.tweet_result_by_rest_id(tweet_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/pipx/venvs/gallery-dl/lib/python3.12/site-packages/gallery_dl/extractor/twitter.py", line 1046, in tweet_result_by_res
t_id

Just to be on the safe side I uninstalled and reinstalled everything:

[user@archvm ~]$ pipx install gallery-dl
  installed package gallery-dl 1.27.0, installed using Python 3.12.3
  These apps are now globally available
    - gallery-dl
  These manual pages are now globally available
    - man1/gallery-dl.1
    - man5/gallery-dl.conf.5
done! ✨ 🌟 ✨
[user@archvm ~]$ pipx install ytdl
  installed package ytdl 1.0.0rc5, installed using Python 3.12.3
  These apps are now globally available
    - ytdl
done! ✨ 🌟 ✨
[user@archvm ~]$ pipx install yt-dlp
  installed package yt-dlp 2024.5.27, installed using Python 3.12.3
  These apps are now globally available
    - yt-dlp
  These manual pages are now globally available
    - man1/yt-dlp.1
done! ✨ 🌟 ✨

And here is the corresponding part of my config file:

{
        "extractor":
        {
                "base-directory": "/home/user/X",
                "archive": "/home/user/.config/gallery-dl/arquivo.db",
                "ytdl":
                {
                        "enabled": true,
                        "module": "yt_dlp"
                 },

                 ...

                "reddit":
                {
                        "user-agent": "Python:gallery-app:v1.0 (by /u/myuser)",
                        "client-id": "xxxxxxxxxxxxx",
                        "skip": "abort:3",
                        "comments": 0,
                        "morecomments": true,
                        "recursion": 1,
                        "videos": "ytdl",
                        "format": "mp4",
                        "directory": [""],
                        "filename": {
                                "'title' not in locals()" : "{filename}.{extension}",
                                ""            : "{filename} - {title}.{extension}"
                        },      
                        "refresh-token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
                },
                "twitter":
                {
                        "skip": "abort:10",
                        "quoted": true,
                        "replies": false,
                        "unique": true,
                        "content": true,
                        "ratelimit": "wait",
                        "retweets": false,
                        "size": ["orig", "4096x4096", "large", "medium", "small"],
                        "videos": "ytdl",
                        "directory": ["Redes sociais", "Twitter", "{user[name]}"],
                        "filename": {
                                "count > 1" : "{date:%Y%m%d %H%M%S}{num:?_//}.{extension}",
                                ""          : "{date:%Y%m%d %H%M%S}.{extension}"
                        },
                        "postprocessors": [{
                                "name": "exec",
                                "command": ["~/.local/bin/commentsx", "{content}", "{_directory}{_filename}"],
                                "async": true,
                                "event": "after"
                        }]
                }
        },
        "downloader":
        {
                "ytdl":
                {
                        "module": "yt_dlp",
                        "format": "bestvideo+bestaudio/best",
                        "forward-cookies": true,
                        "logging": true,
                        "config-file": "/home/user/.config/yt-dlp/yt-dlp.conf",
                        "verify": true,
                        "outtmpl": "%(upload_date)s - %(title)s.%(ext)s",
                        "raw-options":
                        {
                                "verbose": true,
                                "usenetrc": true,
                                "ignoreerrors": true,
                                "nooverwrites": true,
                                "nocheckcertificate": true,
                                "download_archive": "/home/user/.config/yt-dlp/downloaded_files"
                        }
                }
        },

        ...

Is it something I'm doing wrong? Some misconfiguration, perhaps?

mikf commented 3 weeks ago

The Reddit problem is that pipx installs each of its packages into a separate virtual environment, meaning that even though you've installed both gallery-dl and yt-dlp, they cannot see and import one another, hence the "ModuleNotFoundError: No module named 'yt_dlp'" error.

edit: https://github.com/pypa/pipx/issues/88

Haven't tested this myself, but this might work

pipx inject gallery-dl yt-dlp

The Twitter error is because this Tweet isn't viewable as guest user and Twitter recently(?) changed its response for such Tweets to an absolute minimum, which doesn't get properly reported as "Login Required" by gallery-dl yet.

{"data":{"tweetResult":{}}}

edit: Never mind, it is also broken when logged in, but it seems to be on Twitter's end.

{"errors":[{"message":"_Missing: No status found with that ID.","locations":[{"line":5,"column":3}],"path":["threaded_conversation_with_injections_v2"],"extensions":{"name":"GenericError","source":"Server","code":144,"kind":"NonFatal","tracing":{"trace_id":"ebe89fa9fa603d0f"}},"code":144,"kind":"NonFatal","name":"GenericError","source":"Server","tracing":{"trace_id":"ebe89fa9fa603d0f"}}],"data":{}}

alscaldas commented 3 weeks ago

Hey @mikf , thank you for taking the time to help me.

As for the Reddit issue (the one related to pipx), based on your input I did some research and I believe I found a way (I'm writing it in the case someone have the same issue in the future).

pipx install gallery-dl
pipx inject gallery-dl yt-dlp module-1 ... module-n

This way the further modules get installed at the same environment as the first one.

As for the second error, if it's on the Twitter's end there's nothing to be done.

I do appreciate your help and I believe this issue may be closed.

mikf / gallery-dl

[downloader.ytdl][error] Cannot import module 'yt_dlp' #5689