Add ability to use gallery-dl features on any site youtube-dl supports?

[Twi-Hard]

Could you make it possible to use gallery-dl's features on any youtube-dl download? There's been many times that the way gallery-dl handles naming from metadata would have been extremely useful. You could maybe make youtube-dl a category in the config with each site being a subcategory so you can configure the directories and filenames for each site separately. I don't know if this would be possible or not but it's something I've been wanting for a long time now.

[mikf]

At least the basics for this are done in https://github.com/mikf/gallery-dl/commit/36ac2197dbae5a4028fa680092373af212cad1d9.

You can tell gallery-dl to download anything with youtube-dl by prefixing it with ytdl: (gallery-dl ytdl:BaW_jenozKc), or you enable the extractor.ytdl.enabled option (suggestion for a better name?) and it'll match any URL unless it got matched by another extractor first.

The subcategory gets set to the ie_key() value of the youtube-dl extractor that gets used, e.g. Youtube for single youtube videos, YoutubeTab for youtube playlists. (Should those names be lowercased? Should it use something different?)

Todo:

• [x] cookies support
• [x] username & password support
• [x] option to disable generic extractor
• [x] download progress (#1519)
• [ ] youtube-dl command-line options?

and there are definitely bugs ...

edit: #878 is more or less the same as this issue

[Twi-Hard]

The subcategory gets set to the ie_key() value of the youtube-dl extractor that gets used, e.g. Youtube for single youtube videos, YoutubeTab for youtube playlists. (Should those names be lowercased? Should it use something different?)

The key would be used as the subcategory name so it would make sense for it to be lowercase. Although when people add the keys they might just copy exactly what's in the list of keys which wouldn't be lowercase.

I can't get "raw-options" to work. Before changing the config at all I tried downloading a youtube video with "ytdl:" and it used all the options I had in "downloader.ytdl.raw-options". The options in "extractor.ytdl.raw-options" don't do anything at all even after removing "downloader.ytdl" to test it. I also tried moving the raw-options to "extractor.ytdl.Youtube" but that didn't work either. All of the directory and filename options I put in "extractor.ytdl.Youtube" worked fine.

[Twi-Hard]

Adding "module" to a subcategory doesn't change the module used. Sometimes features work on one version of youtube-dl but not the other.

[Hrxn]

Are you talking about the extractor.ytdl.module option?

Like this?

"extractor":
{
    "ytdl":
    {
        "module": "youtube_dl"
    }
}

[Twi-Hard]

This makes the module "youtube_dl":

{
  "extractor": {
    "ytdl": {
      "module": "youtube_dl"
    }
  }
}

This doesn't make the module for "Youtube" "yt_dlp":

{
  "extractor": {
    "ytdl": {
      "module": "youtube_dl",
      "Youtube": {
        "module": "yt_dlp"
      }
    }
  }
}

[Hrxn]

Huh, that kind of nested object doesn't make sense. Using dot property notation, this would be something like this: extractor.ytdl.Youtube.module which then gets set to "yt_dlp" And this option does not exist.

I think yt_dlp is correct though, with the underscore, because that's the internal module name used by Python as opposed to the project name or binary name..

So what if you just try this?

{
  "extractor": {
    "ytdl": {
      "module": "yt_dlp"
    }
  }
}

[Twi-Hard]

I already know that works. I was trying to make it use a different module for Youtube specifically (as an example). youtube-dl isn't always the best for every site and yt-dlp isn't always the best for every site either.

Edit: I thought it should have worked because other options like "filename", "base-directory" and "directory" can be set to different things in each subcategory to override the option specified in the base category.

[Hrxn]

Edit: I thought it should have worked because other options like "filename", "base-directory" and "directory" can be set to different things in each subcategory to override the option specified in the base category.

Yeah, but the ytdl extractor is not a usual extractor and does not have any subcategories (for now).

Okay, so you want to switch dynamically between youtube-dl and yt-dlp, depending on the input? I see..

Don't know how to properly tackle this, honestly. I mean, how to know - in advance - which is more suitable of these two downloader modules for a given site?

But you can do something like this as a workaround:

Use gallery-dl with input files https://github.com/mikf/gallery-dl/blob/36ac2197dbae5a4028fa680092373af212cad1d9/gallery_dl/option.py#L99-L104

Like this: gallery-dl -i <yourinputfile.txt>

Because gallery-dl can parse those input files, and you can define options there: For the next URL (- base-directory = "/tmp/"), or globally (-G base-directory = "/tmp/"), so you can group a bunch of URLs you want to feed to gallery-dl together, something like this:

-G extractor.ytdl.module = "youtube_dl"
YOUR_URL_1
YOUR_URL_2
YOUR_URL_3
YOUR_URL_4
YOUR_URL_5
-G extractor.ytdl.module = "yt_dlp"
YOUR_URL_6
YOUR_URL_7
YOUR_URL_8
-G extractor.ytdl.module = "youtube_dl"
YOUR_URL_9
YOUR_URL_10
YOUR_URL_11
YOUR_URL_12

See what I mean? I think this should work.. 😄

[Twi-Hard]

I could just use -o. I just brought it up as an idea because this extractor is still in development.

I found a big bug but I want to test it a lot more before going into detail.

[mikf]

Using a different youtube_dl module per subcategory should work with https://github.com/mikf/gallery-dl/commit/9a849cdf61a6b4e1c4262062a588c0f00ae03629.

It now

• imports the main module from extractor.ytdl.module first,
• finds a suitable extractor and sets its ie_key() as subcategory,
• checks if extractor.ytdl.<subcategory>.module is different from the main one and imports that if needed

So it needs to imports up to 2 youtube_dl modules, the main one to find a subcategory value and the one to actually use. That means the main module must support the input URL, otherwise it'll just fall back to the generic extractor, ignore the URL, or even raise an error.

Another issue is the fact that the downloader uses its own youtube_dl module/object independently from the one used by the extractor. I'll most likely just transfer the extractor ytdl to the downloader and use that, ignoring any downloader.ytdl.* options.

[mikf]

I think this should work.. :smile:

The idea is good, but it wouldn't have worked with how I implemented it. The main youtube_dl module gets read and imported only once and a cached version is used after that, i.e. extractor.ytdl.module is ignored after the first URL.

[Hrxn]

Wait a sec... So this is a distinctiveness of the youtube_dl module? Because it gets cached, unlike some other settings? Because I've been using input files since I've started using the program, and have grouped together URLs like this. To change keyword or metadata settings, basically, like filename and directory settings.

[Twi-Hard]

I found a big bug but I want to test it a lot more before going into detail.

This got fixed with something you changed.

[Twi-Hard]

Using yt-dlp as the module:

• makes it only download the json (the one the metadata postprocessor creates)
• the log makes it look like it successfully downloaded and merged the audio/video but the file isn't in the output folder
• makes it think the video's already downloaded after trying again even if the output folder no longer exists
• no raw-options work (I tried placing them in "extractor.ytdl.raw-options", "extractor.ytdl.Youtube.raw-options" and "downloader.ytdl.raw-options" (not at the same time))

Using youtube-dl as the module:

• out of these options:

      "writeinfojson": true,
      "writedescription": true,
      "writethumbnail": true,
      "merge_output_format": "mkv"

  only "writeinfojson" worked when placed in "extractor.ytdl.raw-options" ("writeinfojson" is redundant anyways)

• all options worked when placing raw-options in "downloader.ytdl.raw-options"

---

No "raw-options" work in subcategories.

config for "extractor":

"ytdl": {
      "enabled": false,
      "format": "bestvideo+bestaudio/best",
      "logging": true,
      "generic": true,
      "module": "youtube_dl",
      "directory": "",
      "raw-options": {
        "writecomments": true,
        "writeinfojson": true,
        "writedescription": true,
        "writethumbnail": true,
        "merge_output_format": "mkv"
      },
      "Youtube": {
        "base-directory": "/example/path/youtube.com/",
        "directory": [
          "{channel}_{channel_id}"
        ]
      }
    }

config for "downloader":

"downloader": {
    "filesize-min": null,
    "filesize-max": null,
    "part": true,
    "part-directory": null,
    "mtime": true,
    "rate": null,
    "retries": 4,
    "timeout": 30.0,
    "verify": true,
    "http": {
      "adjust-extensions": true,
      "headers": null
    },
    "ytdl": {
      "format": null,
      "forward-cookies": false,
      "logging": true,
      "module": "youtube_dl",
      "outtmpl": null,
      "raw-options": {
        "writecomments": true,
        "writeinfojson": true,
        "writedescription": true,
        "writethumbnail": true,
        "merge_output_format": "mkv"
      }
    }
  }

Each time I tested something I made sure that the value for "module" in "downloader.ytdl" and "extractor.ytdl" were the same.

---

only "writeinfojson" worked when placed in "extractor.ytdl.raw-options" ("writeinfojson" is redundant anyways)

After writing this comment out I can't get this to work anymore either

---

It's really important that "raw-options" works in subcategories because that is often different for each site.

[mikf]

• yt_dlp issue is fixed (https://github.com/mikf/gallery-dl/commit/fc19010808ce8aa9a5330d50f3feaaa042bce7ad). It doesn't use outtmpl like youtube_dl does anymore, so it just downloaded with the default naming settings (current working directory, default name)
• "raw-options" and options in general not working is fixed (https://github.com/mikf/gallery-dl/commit/06e69ea79a2fa6eb8b7a1423ab96dfc6d73ea580). That was just me being dumb.
• The download process now only uses options set for the ytdl extractor and completely ignores anything under downloader.ytdl.* (https://github.com/mikf/gallery-dl/commit/f5b097165ef65c2d5a9f1d988010bc5f5d2313c4)
• Changing youtube_dl modules by setting new values for extractor.ytdl.module in input files now works as well (https://github.com/mikf/gallery-dl/commit/e622e004f0f7a7dc0b8533389f4c63c22fbd61a6)

[github-account1111]

youtube-dl isn't always the best for every site and yt-dlp isn't always the best for every site either.

Which sites is yt-dlp not the best for? Isn't it a direct fork that only improves upon the main repo?

[Twi-Hard]

Which sites is yt-dlp not the best for? Isn't it a direct fork that only improves upon the main repo?

Both youtube-dl and yt-dlp get a 429 error in one of the requests it takes to download a video/metadata. youtube-dl successfully downloads the video while yt-dlp doesn't. I got these errors after not downloading any YouTube video at all with that IP. I can't change the IP for my server.

Edit: I accidently clicked the comment and close button while typing this out lol

Edit 2: There'll likely be other situations like this in the future

[github-account1111]

Both youtube-dl and yt-dlp get a 429 error in one of the requests it takes to download a video/metadata. youtube-dl successfully downloads the video while yt-dlp doesn't

I don't understand what you're saying. So it errored out but still downloaded? Makes no sense. Can you provide an example site/url?

There'll likely be other situations like this in the future

I don't see how.

[Twi-Hard]

I can't get it to do it at the moment. It has happened off and on. I can't grep the lines I need out of the log either. Also, the log is 4 GB so I can't open it in a text editor without it freezing and crashing. Showing the log doesn't matter at this point anyways. The features are already implemented. I've been busy for a few days now so I haven't been able to test the ytdl: feature much yet.

[github-account1111]

I can't get it to do it at the moment. It has happened off and on.

This tells me that this wasn't a ytdlp-VS-youtubedl problem to begin with, rather a network problem (which sleep/sleep-request are there for).

[Twi-Hard]

Command line options would definitely be good. These options can't do everything that I'd want to use (many things are missing actually). Maybe instead of putting keys (like "writethumbnail") from that page I just linked into the config file you could use the argument names (like "write-thumbnail" or "--write-thumbnail") instead (you might know a better way to do this)?

[Twi-Hard]

@mikf Are you okay with donations for priority? I could donate $100. Command line options would be really important for me.

[mikf]

I must have somehow missed your previous comment, my apologies.

Are you okay with donations for priority? I could donate $100

That's not really the purpose of the "Sponsor" button, and $100 would be far too much for me to accept in good conscience. Thank you, but I have to decline.

These options can't do everything that I'd want to use (many things are missing actually)

What exactly? I'd need to know what command-line options specifically to be able to take a look at how youtube-dl handles such an option internally, so I can maybe add a workaround. --add-header is one I know of, but that's it for me.

Maybe instead of putting keys (like "writethumbnail") from that page I just linked into the config file you could use the argument names (like "write-thumbnail" or "--write-thumbnail")

That can be done, but it would just be via internal translation from --write-thumbnail to writethumbnail. Using youtube-dl the way gallery-dl does makes it more ore less impossible (or at least really hard) to use native youtube-dl command-line arguments.

By "youtube-dl command-line options" at the top I meant adding some extra command-line options to gallery-dl to (quickly) be able to override some often-used youtube-dl options, for example --ytdl-format. These would then be translated to one of these options, as I thought it would be possible to control almost everything with them.

[mikf]

Just looked at the youtube-dl source code again to figure out how and where it processes command-line arguments and found quite a few that are cannot easily be done with raw_options: mostly Post-processing Options.

They can be used, though, but it's a bit more complicated. To use --add-metadata --embed-subs, you'd have to add the following to raw_options:

"postprocessors": [
    {"key": "FFmpegMetadata"},
    {"key": "FFmpegEmbedSubtitle"}
],

It is also possible to reuse youtube-dl's command-line option parser and load options from a youtube-dl config file it seems, although that would mean a lot of duplicated lines of code. And then there's the issue of making this also compatible with yt-dlp.

I'll see what I can do.

[Twi-Hard]

I'm not sure how useful youtube-dl will be once something breaks. There's been no changes for 5 months now. yt-dlp has improved a lot of stuff too. As far as youtube goes, I was wanting to try to get the options as similar to this as I can: link Here's what the options look like: link

[mikf]

Added the ability to specify youtube-dl/yt-dlp command-line options (cmdline-args) and/or a ytdl config file (config-file): https://github.com/mikf/gallery-dl/commit/efa178cc91407673f427f33811035a4ef8395c6d. I've tested it with the options from here and at least that works.

[Twi-Hard]

Thank you so much :) I can't get it to work though. In the extractor section of the config I've tried these (with and without specifying the module):

"ytdl": {
    "cmdline-args": {
      "write-info-json": true,
      "write-description": true,
      "write-thumbnail": true,
      "merge-output-format": "mkv"
    }
  }

and this:

{
  "ytdl": {
    "Youtube": {
      "cmdline-args": {
        "write-info-json": true,
        "write-description": true,
        "write-thumbnail": true,
        "merge-output-format": "mkv"
      }
    }
  }
}

If I try one of those I get this error:

[ytdl][debug] Using <module 'yt_dlp' from '/usr/local/lib/python3.8/dist-packages/yt_dlp/__init__.py'>
[ytdl][error] An unexpected error occurred: TypeError - unhashable type: 'slice'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[ytdl][debug]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gallery_dl/job.py", line 80, in run
    for msg in extractor:
  File "/usr/local/lib/python3.8/dist-packages/gallery_dl/extractor/ytdl.py", line 72, in items
    ytdl_instance = ytdl.construct_YoutubeDL(
  File "/usr/local/lib/python3.8/dist-packages/gallery_dl/ytdl.py", line 34, in construct_YoutubeDL
    opts = parse_command_line(module, argv) if argv else user_opts
  File "/usr/local/lib/python3.8/dist-packages/gallery_dl/ytdl.py", line 65, in parse_command_line
    parser, opts, args = module.parseOpts(argv)
  File "/usr/local/lib/python3.8/dist-packages/yt_dlp/options.py", line 1564, in parseOpts
    opts, args = parser.parse_args(overrideArguments)
  File "/usr/lib/python3.8/optparse.py", line 1369, in parse_args
    rargs = self._get_args(args)
  File "/usr/lib/python3.8/optparse.py", line 1353, in _get_args
    return args[:]              # don't modify caller's list
TypeError: unhashable type: 'slice'

If I remove the "cmdline-args" section of the config it runs fine but without the options. If I try putting the options in the "download" section instead of "extractor" section the options don't work for this command but it doesn't error. I've tried all of this both locally and in docker with the same exact results. What am I doing wrong?

[mikf]

Sorry for not giving a proper example or adding any docs on how to use these options.

cmdline-args is supposed to be a simple string with pure command-line arguments in more or less the same way you'd specify them in a terminal, or it can be a list of strings with all arguments already split

"cmdline-args": "--write-info-json --write-description --write-thumbnail --merge-output-format mkv",

"cmdline-args": [
    "--write-info-json",
    "--write-description",
    "--write-thumbnail",
    "--merge-output-format", "mkv"
]

config-file is supposed to be a regular path to a youtube-dl config file.

"config-file": "~/.config/youtube-dl/config"

If I try putting the options in the "download" section instead of "extractor" section the options don't work for this command but it doesn't error.

The options in the downloader section get ignored for the ytdl extractor, because otherwise the whole "different module and options per subcategory" thing couldn't work.

[Twi-Hard]

Something I should probably point out is that if downloader.progress isn't set to null, it will error on many videos every time I try to download them. This happens with both my main computer and a fresh ubuntu container in Docker with python3.8 installed. I don't know if this only happens with the ytdl extractor or not. This is the error:

 88% 195.55MB   7.54MB/s [ytdl][debug] [download]  88.7% of 211.46MiB at 11.87MiB/s ETA 00:02
 88% 196.60MB  12.44MB/s [ytdl][debug] [download]  89.6% of 211.46MiB at 16.94MiB/s ETA 00:01
 89% 198.69MB  17.75MB/s [ytdl][debug] [download]  91.5% of 211.46MiB at 26.33MiB/s ETA 00:00
 91% 202.89MB  27.60MB/s [ytdl][debug] [download]  92.4% of 211.46MiB at 29.54MiB/s ETA 00:00
 92% 204.85MB  30.97MB/s [ytdl][debug] [download]  92.4% of 211.46MiB at Unknown speed ETA Unknown ETA
[downloader.ytdl][debug] Traceback
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gallery_dl/downloader/ytdl.py", line 106, in _download_video
    ytdl_instance.process_info(info_dict)
  File "/usr/local/lib/python3.8/dist-packages/yt_dlp/YoutubeDL.py", line 2836, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/usr/local/lib/python3.8/dist-packages/yt_dlp/YoutubeDL.py", line 2578, in dl
    return fd.download(name, new_info, subtitle)
  File "/usr/local/lib/python3.8/dist-packages/yt_dlp/downloader/common.py", line 399, in download
    ret = self.real_download(filename, info_dict)
  File "/usr/local/lib/python3.8/dist-packages/yt_dlp/downloader/http.py", line 373, in real_download
    return download()
  File "/usr/local/lib/python3.8/dist-packages/yt_dlp/downloader/http.py", line 308, in download
    self._hook_progress({
  File "/usr/local/lib/python3.8/dist-packages/yt_dlp/downloader/common.py", line 415, in _hook_progress
    ph(status)
  File "/usr/local/lib/python3.8/dist-packages/gallery_dl/downloader/ytdl.py", line 127, in _progress_hook
    int(info["speed"]),
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Without setting downloader.progress to null, it always errors when it says "Unknown speed ETA Unknown ETA" It doesn't crash with it set to null:

[ytdl][debug] [download]  19.6% of 286.46MiB at 30.07MiB/s ETA 00:07
[ytdl][debug] [download]  20.3% of 286.46MiB at 33.71MiB/s ETA 00:06
[ytdl][debug] [download]  20.3% of 286.46MiB at Unknown speed ETA Unknown ETA
[ytdl][debug] [download]  20.3% of 286.46MiB at 495.82KiB/s ETA 07:51
[ytdl][debug] [download]  20.3% of 286.46MiB at 543.20KiB/s ETA 07:10
[ytdl][debug] [download]  20.3% of 286.46MiB at 759.59KiB/s ETA 05:07
[ytdl][debug] [download]  20.3% of 286.46MiB at  1.14MiB/s ETA 03:20
[ytdl][debug] [download]  20.3% of 286.46MiB at  1.74MiB/s ETA 02:11
[ytdl][debug] [download]  20.3% of 286.46MiB at  2.49MiB/s ETA 01:31
[ytdl][debug] [download]  20.4% of 286.46MiB at  3.56MiB/s ETA 01:04
[ytdl][debug] [download]  20.5% of 286.46MiB at  5.03MiB/s ETA 00:45
[ytdl][debug] [download]  20.7% of 286.46MiB at  7.75MiB/s ETA 00:29
[ytdl][debug] [download]  21.0% of 286.46MiB at 12.84MiB/s ETA 00:17
[ytdl][debug] [download]  21.7% of 286.46MiB at 20.16MiB/s ETA 00:11
[ytdl][debug] [download]  23.1% of 286.46MiB at 28.90MiB/s ETA 00:07
[ytdl][debug] [download]  23.7% of 286.46MiB at 31.59MiB/s ETA 00:06
[ytdl][debug] [download]  23.7% of 286.46MiB at Unknown speed ETA Unknown ETA
[ytdl][debug] [download]  23.7% of 286.46MiB at 854.64KiB/s ETA 04:22
[ytdl][debug] [download]  23.7% of 286.46MiB at 874.83KiB/s ETA 04:15
[ytdl][debug] [download]  23.7% of 286.46MiB at  1.18MiB/s ETA 03:05

I'm just mentioning it so it's not a problem for other people. It would be nice to be able to have the progress bar on other extractors and not have to disable it completely.

[mikf]

@Twi-Hard fixed in https://github.com/mikf/gallery-dl/commit/19403a7fffd65a361912ac7917a5597568537857 It is also possible to disable the progress bar only for the ytdl downloader by setting downloader.ytdl.progress to null instead of downloader.progress. That way it is still enabled for regular HTTP downloads.

[Twi-Hard]

It's crashing on private videos and copyright claimed videos (and possibly other types of unavailable videos) example:

[ytdl][debug] [debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=7EX5FBt9w0Q
[ytdl][debug] [youtube] Sleeping 1.0 seconds ...
[ytdl][debug] [youtube] 7EX5FBt9w0Q: Downloading webpage
[ytdl][debug] [youtube] Sleeping 1.0 seconds ...
[ytdl][debug] [youtube] 7EX5FBt9w0Q: Downloading android player API JSON
[ytdl][error] ERROR: [youtube] 7EX5FBt9w0Q: Private video. Sign in if you've been granted access to this video
[ytdl][error] Traceback (most recent call last):
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 593, in extract
    ie_result = self._real_extract(url)
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/extractor/youtube.py", line 2646, in _real_extract
    self.raise_no_formats(reason, expected=True)
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 1089, in raise_no_formats
    raise ExtractorError(msg, expected=expected, video_id=video_id)
yt_dlp.utils.ExtractorError: Private video. Sign in if you've been granted access to this video
Traceback (most recent call last):
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 593, in extract
    ie_result = self._real_extract(url)
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/extractor/youtube.py", line 2646, in _real_extract
    self.raise_no_formats(reason, expected=True)
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 1089, in raise_no_formats
    raise ExtractorError(msg, expected=expected, video_id=video_id)
yt_dlp.utils.ExtractorError: Private video. Sign in if you've been granted access to this video

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 1280, in wrapper
    return func(self, *args, **kwargs)
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 1305, in __extract_info
    ie_result = ie.extract(url)
  File "/home/twilight/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 609, in extract
    raise ExtractorError(
yt_dlp.utils.ExtractorError: [youtube] 7EX5FBt9w0Q: Private video. Sign in if you've been granted access to this video

[ytdl][error] An unexpected error occurred: TypeError - argument of type 'NoneType' is not iterable. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[ytdl][debug]
Traceback (most recent call last):
  File "/home/twilight/.local/lib/python3.8/site-packages/gallery_dl/job.py", line 80, in run
    for msg in extractor:
  File "/home/twilight/.local/lib/python3.8/site-packages/gallery_dl/extractor/ytdl.py", line 95, in items
    for info_dict in results:
  File "/home/twilight/.local/lib/python3.8/site-packages/gallery_dl/extractor/ytdl.py", line 113, in _process_entries
    if "entries" in info_dict:
TypeError: argument of type 'NoneType' is not iterable

The errors are skipped automatically when not using gallery-dl.

[mikf]

@Twi-Hard fixed in https://github.com/mikf/gallery-dl/commit/2076d40681c7e8959b4a280350a20ffd9a509bb2. Errors now get ignored and only produce a logging message, even without -i/--ignore-errors for ytdl.

[Twi-Hard]

It seems postprocessors don't work with this extractor. I'm going to start using an exec postprocessor on all of the files I download with gallery-dl. Would this be possible? Thanks :)

[Hrxn]

Huh? Not working, which extractor specifically?

[Twi-Hard]

The extractor that this issue is about (ytdl). Any site I've tested with that extractor doesn't do the postprocessors set up in the config. The postprocessesors work on every other extractor just fine.

[Hrxn]

Yeah, that is not a "normal" extractor, it just handles importing yt-dlp so that it can be called from within gallery-dl..

I don't even know what you are trying to do, to be honest..

The output options for yt-dlp are here: https://github.com/yt-dlp/yt-dlp#output-template

[Twi-Hard]

Are you think of https://github.com/mikf/gallery-dl/blob/master/gallery_dl/downloader/ytdl.py rather than https://github.com/mikf/gallery-dl/blob/master/gallery_dl/extractor/ytdl.py ?

The output options for yt-dlp are here: https://github.com/yt-dlp/yt-dlp#output-template

The output options are all handled by gallery-dl. Example (wip) config for single YouTube videos:

"Youtube": {
        "module": "yt_dlp",
        "#format": "bestvideo+bestaudio/best",
        "#rate": "100k",
        "archive": "$ROOT/downloads/gallery-dl/archives/youtube.sqlite3",
        "archive-prefix": "youtube.com, ",
        "archive-format": "{channel_id}, {id}, {live_status}",
        "cmdline-args": [
          "#--write-info-json",
          "--write-description",
          "--write-annotations",
          "--write-thumbnail",
          "--embed-thumbnail",
          "--add-metadata",
          "--no-continue",
          "--no-overwrites",
          "--all-subs",
          "--embed-subs",
          "--verbose",
          "--progress",
          "#--no-progress",
          "--ignore-errors",
          "--check-formats",
          "--sleep-requests",
          "1",
          "--sleep-interval",
          "5",
          "#--max-sleep-interval",
          "#30",
          "--merge-output-format",
          "mkv",
          "--external-downloader",
          "aria2c",
          "--external-downloader-args",
          "-x 4 -k 1M",
          "#--list-formats"
        ],
        "base-directory": REDACTED,
        "directory": [
          "{channel} [{channel_id}]"
        ],
        "filename": {
          "is_live == True": "{channel} - {upload_date} - {title} [{id}] [live].{extension!l}",
          "": "{channel} - {upload_date} - {title} [{id}].{extension!l}"
        }
      }

Being able to run "exec" postprocessors on the files downloaded would be extremely useful.

[Hrxn]

No, I didn't miss that, I was thinking of https://github.com/mikf/gallery-dl/blob/master/gallery_dl/extractor/ytdl.py

But if you've read that even cursorily, you might have noticed that it's basically a wrapper around https://github.com/mikf/gallery-dl/blob/master/gallery_dl/ytdl.py (oversimplification, I know)

Being able to run "exec" postprocessors on the files downloaded would be extremely useful.

You could have said that in the first comment..

The output options are all handled by gallery-dl. Example (wip) config for single YouTube videos:

You know, if you want to download something from YouTube, simply using yt-dlp would probably be a lot easier..

[Twi-Hard]

But if you've read that even cursorily, you might have noticed that it's basically a wrapper around https://github.com/mikf/gallery-dl/blob/master/gallery_dl/ytdl.py (oversimplification, I know)

I don't know how to read code

You could have said that in the first comment..

I very clearly did. That was the whole point of the comment.

This conversation is pointless. Most of the comments I see you post on issues just distract from the issue. If you reply to this, I'm not responding since that would lead nowhere.

[mikf]

@Twi-Hard what exactly are you trying to do?

I just ran a simple test with

    "postprocessors": [
        {
            "name": "exec",
            "command": ["echo", "{channel}"]
        }
    ]

and that, as expected, echoed the channel name after the download. --write-metadata and --write-info-json also work.

[Twi-Hard]

I'm so dumb.. The postprocessors I tested are whitelisted to specific sites.. that's why it wasn't outputting anything. I'm really sorry about all of this.