search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
10.68k stars 881 forks source link

ytdl extractor error when unavailable video is encountered #5675

Closed Twi-Hard closed 2 weeks ago

Twi-Hard commented 3 weeks ago

I'm getting this error when gallery-dl runs into a video that doesn't exist:

[ytdl][debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=oxpWLPyBD0o
[ytdl][debug] [youtube] Sleeping 5.0 seconds ...
[ytdl][debug] [youtube] oxpWLPyBD0o: Downloading webpage
[ytdl][debug] [youtube] Sleeping 5.0 seconds ...
[ytdl][debug] [youtube] oxpWLPyBD0o: Downloading ios player API JSON
[ytdl][warning] [youtube] Video unavailable. This video has been removed by the uploader
[ytdl][debug] [debug] Extractor gave empty title. Creating a generic title
[ytdl][warning] No video formats found!
[ytdl][debug] [debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[ytdl][debug] [debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[ytdl][debug] [youtube] Sleeping 5.0 seconds ...
[ytdl][debug] [youtube] Downloading comment section API JSON
[ytdl][debug] [youtube] Extracted 0 comments
[ytdl][debug] [debug] Default format spec: bestvideo*+bestaudio/best
[ytdl][warning] Requested format is not available
[ytdl][debug] Sleeping 5.00 seconds (download)
[ytdl][error] An unexpected error occurred: KeyError - 'ext'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[ytdl][debug]
Traceback (most recent call last):
  File "/home/spike/.local/lib/python3.10/site-packages/gallery_dl/job.py", line 151, in run
    self.dispatch(msg)
  File "/home/spike/.local/lib/python3.10/site-packages/gallery_dl/job.py", line 194, in dispatch
    self.handle_url(url, kwdict)
  File "/home/spike/.local/lib/python3.10/site-packages/gallery_dl/job.py", line 331, in handle_url
    if not self.download(url):
  File "/home/spike/.local/lib/python3.10/site-packages/gallery_dl/job.py", line 497, in download
    return downloader.download(url, self.pathfmt)
  File "/home/spike/.local/lib/python3.10/site-packages/gallery_dl/downloader/ytdl.py", line 83, in download
    return self._download_video(ytdl_instance, pathfmt, info_dict)
  File "/home/spike/.local/lib/python3.10/site-packages/gallery_dl/downloader/ytdl.py", line 102, in _download_video
    pathfmt.set_extension(info_dict["ext"])
KeyError: 'ext'

The following are the relevant parts of the config. Metadata postprocessor (archive-format and $domain are set in a wrapper script):

{
    {
        "archive": "$HOME/tools/gallery-dl/archive/{_env[domain]}-metadata.db",
        "archive-prefix": "{_env[domain]}, ",
        "archive-format": null,
        "archive-pragma": [
            "journal_mode=WAL",
            "synchronous=NORMAL"
        ],
        "name": "metadata",
        "mode": "json",
        "extension": "json",
        "extension-format": "{ext!l}.json",
        "filename": "{upload_date} - {title[b:180]} - {id}.{ext!l}.json",
        "event": "file",
        "mtime": true,
        "whitelist": [
            "ytdl:Youtube",
            "ytdl:YoutubePlaylist",
            "ytdl:YoutubeTab"
        ]
    }
}

ytdl extractor (the subcategories should all be the same):

{
    "ytdl": {
        "enabled": false,
        "logging": true,
        "generic": false,
        "module": "yt_dlp",
        "Youtube": {
            "rate": null,
            "archive": "$HOME/tools/gallery-dl/archive/youtube.com-media.db",
            "archive-prefix": "youtube.com, ",
            "archive-format": "{channel_id}, {id}, {live_status}",
            "archive-pragma": [
                "journal_mode=WAL",
                "synchronous=NORMAL"
            ],
            "base-directory": "$OTHER/youtube.com/",
            "directory": [
                "videos",
                "{uploader[b:180]} [{channel_id}]"
            ],
            "filename": {
                "is_live == True": "{upload_date} - {title} - {id} [live].{ext!l}",
                "": "{upload_date} - {title} - {id}.{ext!l}"
            },
            "cmdline-args": [
                "--write-description",
                "--write-annotations",
                "--write-thumbnail",
                "--write-comments",
                "--embed-thumbnail",
                "#--add-metadata",
                "--no-continue",
                "--no-overwrites",
                "--all-subs",
                "--sub-langs",
                "all",
                "--embed-subs",
                "#--write-auto-subs",
                "--verbose",
                "--progress",
                "--ignore-errors",
                "--ignore-no-formats-error",
                "--sleep-requests",
                "5",
                "--sleep-interval",
                "5",
                "#--max-sleep-interval",
                "#30",
                "--merge-output-format",
                "mkv",
                "--force-ipv4"
            ]
        },
        "YoutubePlaylist": {
            "rate": null,
            "archive": "$HOME/tools/gallery-dl/archive/youtube.com-media.db",
            "archive-prefix": "youtube.com, ",
            "archive-format": "{channel_id}, {id}, {live_status}",
            "archive-pragma": [
                "journal_mode=WAL",
                "synchronous=NORMAL"
            ],
            "base-directory": "$OTHER/youtube.com/",
            "directory": [
                "videos",
                "{uploader[b:180]} [{channel_id}]"
            ],
            "filename": {
                "is_live == True": "{upload_date} - {title} - {id} [live].{ext!l}",
                "": "{upload_date} - {title} - {id}.{ext!l}"
            },
            "cmdline-args": [
                "--write-description",
                "--write-annotations",
                "--write-thumbnail",
                "--write-comments",
                "--embed-thumbnail",
                "#--add-metadata",
                "--no-continue",
                "--no-overwrites",
                "--all-subs",
                "--sub-langs",
                "all",
                "--embed-subs",
                "#--write-auto-subs",
                "--verbose",
                "--progress",
                "--ignore-errors",
                "--ignore-no-formats-error",
                "--sleep-requests",
                "5",
                "--sleep-interval",
                "5",
                "#--max-sleep-interval",
                "#30",
                "--merge-output-format",
                "mkv",
                "--force-ipv4"
            ]
        },
        "YoutubeTab": {
            "rate": null,
            "archive": "$HOME/tools/gallery-dl/archive/youtube.com-media.db",
            "archive-prefix": "youtube.com, ",
            "archive-format": "{channel_id}, {id}, {live_status}",
            "archive-pragma": [
                "journal_mode=WAL",
                "synchronous=NORMAL"
            ],
            "base-directory": "$OTHER/youtube.com/",
            "directory": [
                "videos",
                "{uploader[b:180]} [{channel_id}]"
            ],
            "filename": {
                "is_live == True": "{upload_date} - {title} - {id} [live].{ext!l}",
                "": "{upload_date} - {title} - {id}.{ext!l}"
            },
            "cmdline-args": [
                "--write-description",
                "--write-annotations",
                "--write-thumbnail",
                "--write-comments",
                "--embed-thumbnail",
                "#--add-metadata",
                "--no-continue",
                "--no-overwrites",
                "--all-subs",
                "--sub-langs",
                "all",
                "--embed-subs",
                "#--write-auto-subs",
                "--verbose",
                "--progress",
                "--ignore-errors",
                "--ignore-no-formats-error",
                "--sleep-requests",
                "5",
                "--sleep-interval",
                "5",
                "#--max-sleep-interval",
                "#30",
                "--merge-output-format",
                "mkv",
                "--force-ipv4"
            ]
        },
        "YoutubeWebArchive": {
            "rate": null,
            "archive": "$HOME/tools/gallery-dl/archive/youtube.com.db",
            "archive-prefix": "youtube.com, ",
            "archive-format": "{channel_id}, {id}, {live_status}",
            "archive-pragma": [
                "journal_mode=WAL",
                "synchronous=NORMAL"
            ],
            "base-directory": "$OTHER/youtube.com/archive/archive.org-channel",
            "directory": [
                "{uploader} [{channel_id}]"
            ],
            "filename": {
                "is_live == True": [
                    "{channel} - {upload_date} - {title} [{id}] [live].{extension!l}"
                ],
                "": [
                    "{channel} - {upload_date} - {title} [{id}].{extension!l}"
                ]
            },
            "cmdline-args": [
                "--write-description",
                "--write-annotations",
                "--write-thumbnail",
                "#--write-comments",
                "--embed-thumbnail",
                "#--add-metadata",
                "--no-continue",
                "--no-overwrites",
                "--all-subs",
                "--embed-subs",
                "--verbose",
                "--progress",
                "#--no-progress",
                "--ignore-errors",
                "#--check-formats",
                "--sleep-requests",
                "1",
                "--sleep-interval",
                "5",
                "#--max-sleep-interval",
                "#30",
                "--match-filter",
                "!is_live & !live",
                "--merge-output-format",
                "mkv"
            ]
        }
    }
}

I have absolutely no idea about what to do with this one. I've tried some stuff like making {ext!l} be {ext!l|empty} but that didn't help. I'm posting it as in issue in case it's a bug.

❯ gallery-dl --version
1.27.0
mikf commented 3 weeks ago

in case it's a bug.

It is. The current code expects all returned ytdl info_dicts to have an ext field, which is apparently not the case.

With default settings, it only outputs a short "video unavailable" message:

$ gallery-dl ytdl:https://www.youtube.com/watch?v=oxpWLPyBD0o
[ytdl][error] ERROR: [youtube] oxpWLPyBD0o: Video unavailable. This video has been removed by the uploader
[ytdl][error] Failed to extract video data
allendema commented 3 weeks ago

I think this is related to https://github.com/allendema/gallery-dl/commit/036f69c7215d5f5acf8960efcaf9571e745881e0.

mikf commented 3 weeks ago

Kind of, but this is a Youtube video without ext and without formats:

[ytdl][warning] No video formats found!

mikf commented 3 weeks ago

The exception is fixed (https://github.com/mikf/gallery-dl/commit/3ec7ee6baa8103163e437d077f3ebb734e598cda), but it now hangs seemingly forever trying to download thumbnail number 37 down to 0, all of which don't exist.

This is all mainly due to --ignore-errors and --ignore-no-formats-error