mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.91k stars 976 forks source link

AssertionError when using yt-dlp to download from YouTube through gallery-dl (update to newest version of gdl in the actions tab) #6127

Closed ghost closed 2 months ago

ghost commented 2 months ago

I've looked through my galler-dl config file, but I can't seem to figure out where I went wrong. I even did away with the yt-dlp config file and added all that data into gdl, but I still get the same error.

PS C:\Apps> gdl --verbose https://www.youtube.com/@ViralGenesis                                                                                                                                         

C:\Apps>gallery-dl --verbose https://www.youtube.com/@ViralGenesis                                                                                                                                      
[gallery-dl][debug] Version 1.27.4-dev:2024.09.01 - Executable (dev/windows)                                                                                                                            
[gallery-dl][debug] Python 3.12.5 - Windows-11-10.0.22631-SP0                                                                                                                                           
[gallery-dl][debug] requests 2.31.0 - urllib3 2.2.2                                                                                                                                                     
[gallery-dl][debug] Configuration Files ['%APPDATA%\\gallery-dl\\config.json']                                                                                                                          
[gallery-dl][debug] Starting DownloadJob for 'https://www.youtube.com/@ViralGenesis'                                                                                                                    
[ytdl][debug] Using YoutubeDLExtractor for 'https://www.youtube.com/@ViralGenesis'                                                                                                                      
[cookies][debug] Extracting cookies from C:\Users\ariha\AppData\Roaming\Mozilla\Firefox\Profiles\64ym2bhh.default-release\cookies.sqlite                                                                
[cookies][debug] Only loading cookies not belonging to any container                                                                                                                                    
[cookies][info] Extracted 355 cookies from Firefox                                                                                                                                                      
[ytdl][debug] Using <module 'yt_dlp' from 'C:\\Users\\ariha\\AppData\\Local\\Temp\\_MEI97562\\yt_dlp\\__init__.pyc'>                                                                                    
[ytdl][debug] [youtube:tab] Extracting URL: https://www.youtube.com/@ViralGenesis                                                                                                                       
[ytdl][debug] [youtube:tab] @ViralGenesis: Downloading webpage                                                                                                                                          
[ytdl][debug] [youtube:tab] Downloading all uploads of the channel. To download only the videos in a specific tab, pass the tab's URL                                                                   
[ytdl][debug] [youtube:tab] @ViralGenesis/streams: Downloading webpage                                                                                                                                  
[ytdl][debug] [youtube:tab] @ViralGenesis/shorts: Downloading webpage                                                                                                                                   
[ytdl][debug] [youtube:tab] Downloading as multiple playlists, separated by tabs. To download as a single playlist instead, pass https://www.youtube.com/playlist?list=UUwuMwHtnseX2g4qFg5COlOA         
[ytdl][debug] [download] Downloading playlist: Viral Genesis                                                                                                                                            
[ytdl][debug] [youtube:tab] Playlist Viral Genesis: Downloading 3 items of 3                                                                                                                            
[ytdl][debug] [download] Downloading item 1 of 3                                                                                                                                                        
[ytdl][debug] [download] Downloading playlist: Viral Genesis - Videos                                                                                                                                   
[ytdl][debug] [youtube:tab] UCwuMwHtnseX2g4qFg5COlOA page 1: Downloading API JSON                                                                                                                       
[ytdl][debug] [youtube:tab] Playlist Viral Genesis - Videos: Downloading 55 items of 55                                                                                                                 
[ytdl][debug] [download] Downloading item 1 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 2 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 3 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 4 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 5 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 6 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 7 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 8 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 9 of 55                                                                                                                                                       
[ytdl][debug] [download] Downloading item 10 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 11 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 12 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 13 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 14 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 15 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 16 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 17 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 18 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 19 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 20 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 21 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 22 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 23 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 24 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 25 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 26 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 27 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 28 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 29 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 30 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 31 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 32 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 33 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 34 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 35 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 36 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 37 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 38 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 39 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 40 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 41 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 42 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 43 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 44 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 45 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 46 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 47 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 48 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 49 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 50 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 51 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 52 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 53 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 54 of 55                                                                                                                                                      
[ytdl][debug] [download] Downloading item 55 of 55                                                                                                                                                      
[ytdl][debug] [download] Finished downloading playlist: Viral Genesis - Videos                                                                                                                          
[ytdl][debug] [download] Downloading item 2 of 3                                                                                                                                                        
[ytdl][debug] [download] Downloading playlist: Viral Genesis - Live                                                                                                                                     
[ytdl][debug] [youtube:tab] Playlist Viral Genesis - Live: Downloading 1 items of 1                                                                                                                     
[ytdl][debug] [download] Downloading item 1 of 1                                                                                                                                                        
[ytdl][debug] [download] Finished downloading playlist: Viral Genesis - Live                                                                                                                            
[ytdl][debug] [download] Downloading item 3 of 3                                                                                                                                                        
[ytdl][debug] [download] Downloading playlist: Viral Genesis - Shorts                                                                                                                                   
[ytdl][debug] [youtube:tab] Playlist Viral Genesis - Shorts: Downloading 2 items of 2                                                                                                                   
[ytdl][debug] [download] Downloading item 1 of 2                                                                                                                                                        
[ytdl][debug] [download] Downloading item 2 of 2                                                                                                                                                        
[ytdl][debug] [download] Finished downloading playlist: Viral Genesis - Shorts                                                                                                                          
[ytdl][debug] [download] Finished downloading playlist: Viral Genesis                                                                                                                                   
[ytdl][debug] Using download archive 'C:/Apps/Tools - Utilities/DL-Files/archive/ytdl_archive.sqlite3'                                                                                                  
[ytdl][debug] Active postprocessor modules: [MetadataPP]                                                                                                                                                
[ytdl][error] An unexpected error occurred: AssertionError - . Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[ytdl][debug]                                                                                                                                                                                           
Traceback (most recent call last):                                                                                                                                                                      
  File "gallery_dl\job.py", line 152, in run                                                                                                                                                            
  File "gallery_dl\job.py", line 195, in dispatch                                                                                                                                                       
  File "gallery_dl\job.py", line 329, in handle_url                                                                                                                                                     
  File "gallery_dl\job.py", line 495, in download                                                                                                                                                       
  File "gallery_dl\downloader\ytdl.py", line 75, in download                                                                                                                                            
  File "gallery_dl\downloader\ytdl.py", line 134, in _download_playlist                                                                                                                                 
  File "yt_dlp\YoutubeDL.py", line 3195, in process_info                                                                                                                                                
AssertionError                                                                                                                                                                                          
PS C:\Apps>                                                                                                                                                                                            
ghost commented 2 months ago

It didn't like that many lines in the text box and my browser was lagging, so here is the rest from my config file:

{
    "extractor":
    {
        "base-directory": "K:/Data/Art/gallery-dl",
        "parent-directory": false,
        "path-restrict": {
            ":" : "-",
            "|" : "-",
            "/" : "_",
            "\\" : "_",
            "*" : "",
            "?" : "",
            "\"" : "",
            "<" : "[",
            ">" : "﹥]"
        },
        "postprocessors": [
        {
            "name": "ugoira",
            "whitelist": ["pixiv", "danbooru"],
            "extension": "gif",
            "ffmpeg-args": ["-filter_complex", "[0:v] split [a][b];[a] palettegen [p];[b][p] paletteuse"],
            "repeat-last-frame": false
        }
        ],
        "archive": "C:/Apps/Tools - Utilities/DL-Files/archive/{category}_archive.sqlite3",
        "cookies": ["firefox"],
        "cookies-update": true,
        "proxy": null,
        "skip": true,

        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0",
        "retries": 100,
        "timeout": 30.0,
        "verify": true,
        "fallback": true,

        "sleep": 0,
        "sleep-request": 0,
        "sleep-extractor": 0,

        "path-replace": "_",
        "path-remove": "\\u0000-\\u001f\\u007f",
        "path-strip": "auto",
        "extension-map": {
            "jpeg": "jpg",
            "jpe" : "jpg",
            "jfif": "jpg",
            "jif" : "jpg",
            "jfi" : "jpg"
        },

Started on individual extractors... Skipping down to ytdl extractor:

        "ytdl":
        {
            "enabled": true,
            "format": "bestvideo*+bestaudio/best",
            "module": "yt_dlp",
            "config-file" : null,
            "generic": true,
            "logging": true,
            "raw-options":
            {
                "parse-metadata": "title:%(meta_title)s",
                "S": "lang:en,width,hdr:DV,fps,acodec,asr,abr,channels,+size,br",
                "sub-format": "ass/srt/vtt/best",
                "write-subs": true,
                "sub-langs": "en-US,ja-JP,en,ja,-live_chat",
                "yes-playlist": true,
                "no-overwrites": true,
                "no-post-overwrites": true,
                "retries": "infinite",
                "fragment-retries": "infinite",
                "merge-output-format": "mkv",
                "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0",
                "ffmpeg-location": "C:/Apps/Editors - Media/ffmpeg/bin/ffmpeg.exe",
                "audio-multistreams":true
            },
            "postprocessors": [
            {
                "directory": "K:/Data/Art/gallery-dl (finished)/_META/ytdl",
                "name": "metadata",
                "mode": "custom",
                "extension-format": "txt",
                "content-format": ["{channel|uploader!C:?Channel or Uploader: //}{playlist:?\nPlaylist: '/' /}{playlist_index:? (/)/}","{fulltitle:?Title: '/'/}","{tags!S:?Tags: '/'/}","Height: {height}px","Width: {width}px","{webpage_url:?Link: //}","\n\n\n\n\n\n\n{description}"],
                "event": "after"
            }
            ],
            "directory": ["{category!C}", "{channel|uploader!C}", "{playlist:?//}"],
            "filename": "{channel|uploader!C} - {release_date[0:4]|upload_date[0:4]}-{release_date[4:6]|upload_date[4:6]}-{release_date[6:8]|upload_date[6:8]}, {playlist:?'/' /}{playlist_index:? (/), /}'{title!C}' - {subcategory!C}.{extension}"
        },

And the downloader and output segments:

    "downloader":
    {
        "filesize-min": null,
        "filesize-max": null,
        "mtime": true,
        "part": true,
        "part-directory": null,
        "progress": 3.0,
        "rate": null,
        "retries": 4,
        "timeout": 30.0,
        "verify": true,

        "http":
        {
            "adjust-extensions": true,
            "headers": null
        },

        "ytdl":
        {
            "format": "bestvideo*+bestaudio/best",
            "forward-cookies": true,
            "config-file" : null,
            "logging": true,
            "module": "yt_dlp",
            "outtmpl": null
        }
    },

    "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "ansi": false,
        "colors": {
            "success": "1;32",
            "skip"   : "2"
        },
        "skip": false,
        "log": null,
        "format": {
            "debug"  : "debug: {message}",
            "info"   : "[{name}] {message}",
            "warning": "Warning: {message}",
            "error"  : "ERROR: {message}"
        }
    },

    "netrc": false
}
ghost commented 2 months ago

Just realized the formatting from copying to this issue is abysmal. Let me know if it's hard to read, and I will try to make it neater.

mikf commented 2 months ago
  File "yt_dlp\YoutubeDL.py", line 3195, in process_info                                                                                                                                                
AssertionError                                                                           

gallery-dl triggers this error by calling one of yt-dlp's functions with the wrong argument type. This probably happens when handling a playlist of playlists.

https://github.com/yt-dlp/yt-dlp/blob/e8e6a982a1b659eed434d225d7922f632bac6568/yt_dlp/YoutubeDL.py#L3219

ghost commented 2 months ago

Thank you.

So I assume then that by trying to download a whole channel, EX: gallery-dl --verbose https://www.youtube.com/@ViralGenesis, it is then compiling a playlist of playlists as it notes here:

[ytdl][debug] [youtube:tab] Downloading as multiple playlists, separated by tabs. To download as a single playlist instead, pass https://www.youtube.com/playlist?list=UUwuMwHtnseX2g4qFg5COlOA

Is there any way to download whole channels, then, by specifying the channel link? Or will I need to go and download each tab separate (IE. the videos tab, live, shorts, etc.)?

mikf commented 2 months ago

Using https://www.youtube.com/@ViralGenesis as URL already downloads from all tabs, where each tab is given as a separate playlist by yt-dlp as I understand it, but gallery-dl combines them all into one big "playlist" anyway. Or you use https://www.youtube.com/playlist?list=UUwuMwHtnseX2g4qFg5COlOA as URL as stated by yt-dlp to have all channel videos in one real playlist.

ghost commented 2 months ago

Okay, thank you. Does gallery-dl -K <Youtube Channel> provide that link so I don't need to attempt download to find it?

mikf commented 2 months ago

It provides channel_id (UCwuMwHtnseX2g4qFg5COlOA) and channel_url (https://www.youtube.com/channel/UCwuMwHtnseX2g4qFg5COlOA), which only need some small changes to become https://www.youtube.com/playlist?list=UUwuMwHtnseX2g4qFg5COlOA, but I'm pretty sure you can just use https://www.youtube.com/@ViralGenesis and get the same results.

For this channel, https://www.youtube.com/@ViralGenesis is only missing two offline live streams that never started from a few years ago. It would probably be best if you asked the actual yt-dlp devs what's the difference between the two or searched their issue tracker.

ghost commented 2 months ago

Okay. Thank you. I think I understand now.

Have a good day.