search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.34k stars 923 forks source link

[Reddit] ytdl format not working for reddit hosted videos #4367

Closed ghbook closed 1 year ago

ghbook commented 1 year ago

My config file:

extractor: 
{
"reddit":
      {
    "client-id": "6gR***********",
    "user-agent": "Pyth*************",
    "videos": "ytdl",
    "refresh-token": "67*******",
    "directory": ["{category}"],
    "filename": "{category}_{id}{num:?_//}.{extension}",
    "postprocessors":[
        {
            "name": "metadata", 
            "event": "post",
            "filename": "{category}_{id}.json",
            "skip": true
        }
    ]
      },
"ytdl":
      {
          "enabled": true,
          "module": "yt_dlp",

    "directory": ["{subcategory}"],
    "filename": "{subcategory}_{id}{num:?_//}.{extension}",
    "format": "bestvideo[height<=480]+bestaudio/[height<=480]",
          "cmdline-args": "--no-overwrites --write-thumbnail --write-info-json"
      }
}

ytdl format option is not working for reddit hosted videos, it always downloads reddit videos at the highest quality. But it working for other websites for example youtube, pornhub.

D:\>gallery-dl -v https://www.reddit.com/r/Unexpected/comments/x8xt04/filming_a_white_tiger_what_could_possibly_go_wrong/
[gallery-dl][debug] Version 1.25.7
[gallery-dl][debug] Python 3.11.4 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.3
[gallery-dl][debug] Configuration Files ['%APPDATA%\\gallery-dl\\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.reddit.com/r/Unexpected/comments/x8xt04/filming_a_white_tiger_what_could_possibly_go_wrong/'
[reddit][debug] Using RedditSubmissionExtractor for 'https://www.reddit.com/r/Unexpected/comments/x8xt04/filming_a_white_tiger_what_could_possibly_go_wrong/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): oauth.reddit.com:443
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /comments/x8xt04/.json?limit=0&raw_json=1 HTTP/1.1" 200 40310
[reddit][debug] Active postprocessor modules: [MetadataPP]
[downloader.ytdl][debug] [Reddit] Extracting URL: https://www.reddit.com/r/Unexpected/comments/x8xt04/filming_a_white_tiger_what_could_possibly_go_wrong/
[downloader.ytdl][debug] [Reddit] x8xt04: Downloading JSON metadata
[downloader.ytdl][debug] [Reddit] x8xt04: Downloading m3u8 information
[downloader.ytdl][debug] [Reddit] x8xt04: Downloading MPD manifest
  F:\\dled-gallery-dl\reddit\reddit_x8xt04.mp4[downloader.ytdl][debug] [download] Destination: \\?\F:\dled-gallery-dl\reddit\reddit_x8xt04.ffallback.mp4
[downloader.ytdl][debug] [download]   0.0% of    5.67MiB at   40.01KiB/s ETA 02:25
[downloader.ytdl][debug] [download]   0.1% of    5.67MiB at  111.14KiB/s ETA 00:52
[downloader.ytdl][debug] [download]   0.1% of    5.67MiB at  241.46KiB/s ETA 00:24
[downloader.ytdl][debug] [download]   0.3% of    5.67MiB at   54.42KiB/s ETA 01:46
[downloader.ytdl][debug] [download]   0.5% of    5.67MiB at   78.58KiB/s ETA 01:13
[downloader.ytdl][debug] [download]   1.1% of    5.67MiB at   83.74KiB/s ETA 01:08
[downloader.ytdl][debug] [download]   2.2% of    5.67MiB at  127.20KiB/s ETA 00:44
[downloader.ytdl][debug] [download]   4.4% of    5.67MiB at  201.36KiB/s ETA 00:27
[downloader.ytdl][debug] [download]   8.8% of    5.67MiB at  334.60KiB/s ETA 00:15
[downloader.ytdl][debug] [download]  17.6% of    5.67MiB at  562.13KiB/s ETA 00:08
 17%   1.04MB 575.61kB/s [downloader.ytdl][debug] [download]  35.2% of    5.67MiB at  928.08KiB/s ETA 00:04
 35%   2.09MB 950.35kB/s [downloader.ytdl][debug] [download]  70.5% of    5.67MiB at    1.52MiB/s ETA 00:01
 70%   4.19MB   1.59MB/s [downloader.ytdl][debug] [download] 100.0% of    5.67MiB at    1.91MiB/s ETA 00:00
100%   5.95MB   2.00MB/s [downloader.ytdl][debug] [download] 100% of    5.67MiB in 00:00:04 at 1.33MiB/s
[downloader.ytdl][debug] [download] Destination: \\?\F:\dled-gallery-dl\reddit\reddit_x8xt04.fdash-audio_0_132418.m4a
[downloader.ytdl][debug] [download]   0.5% of  193.45KiB at  499.14KiB/s ETA 00:00
[downloader.ytdl][debug] [download]   1.6% of  193.45KiB at  499.68KiB/s ETA 00:00
[downloader.ytdl][debug] [download]   3.6% of  193.45KiB at  778.31KiB/s ETA 00:00
[downloader.ytdl][debug] [download]   7.8% of  193.45KiB at   46.86KiB/s ETA 00:03
[downloader.ytdl][debug] [download]  16.0% of  193.45KiB at   78.52KiB/s ETA 00:02
[downloader.ytdl][debug] [download]  32.6% of  193.45KiB at   89.50KiB/s ETA 00:01
[downloader.ytdl][debug] [download]  65.6% of  193.45KiB at  127.14KiB/s ETA 00:00
[downloader.ytdl][debug] [download] 100.0% of  193.45KiB at  164.54KiB/s ETA 00:00
[downloader.ytdl][debug] [download] 100% of  193.45KiB in 00:00:02 at 68.42KiB/s
[downloader.ytdl][debug] [Merger] Merging formats into "\\?\F:\dled-gallery-dl\reddit\reddit_x8xt04.mp4"
[downloader.ytdl][debug] Deleting original file \\?\F:\dled-gallery-dl\reddit\reddit_x8xt04.ffallback.mp4 (pass -k to keep)
[downloader.ytdl][debug] Deleting original file \\?\F:\dled-gallery-dl\reddit\reddit_x8xt04.fdash-audio_0_132418.m4a (pass -k to keep)
* F:\\dled-gallery-dl\reddit\reddit_x8xt04.mp4
D:\>gallery-dl -v https://www.youtube.com/watch?v=hwNWx1GTSKo
[gallery-dl][debug] Version 1.25.7
[gallery-dl][debug] Python 3.11.4 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.3
[gallery-dl][debug] Configuration Files ['%APPDATA%\\gallery-dl\\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.youtube.com/watch?v=hwNWx1GTSKo'
[ytdl][debug] Using YoutubeDLExtractor for 'https://www.youtube.com/watch?v=hwNWx1GTSKo'
[ytdl][debug] Using <module 'yt_dlp' from 'C:\\Python3\\Lib\\site-packages\\yt_dlp\\__init__.py'>
[ytdl][debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=hwNWx1GTSKo
[ytdl][debug] [youtube] hwNWx1GTSKo: Downloading webpage
[ytdl][debug] [youtube] hwNWx1GTSKo: Downloading ios player API JSON
[ytdl][debug] [youtube] hwNWx1GTSKo: Downloading android player API JSON
[ytdl][debug] [youtube] hwNWx1GTSKo: Downloading m3u8 information
  F:\\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.mkv[ytdl][debug] [info] Downloading video thumbnail 41 ...
[ytdl][debug] [info] Writing video thumbnail 41 to: \\?\F:\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.webp
[ytdl][debug] [info] Writing video metadata as JSON to: \\?\F:\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.info.json
[ytdl][debug] [download] Destination: \\?\F:\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.f333.webm
100%  31.88MB 527.73kB/s [ytdl][debug] [download] Download completed
[ytdl][debug] [download] Destination: \\?\F:\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.f258.m4a
100%  15.83MB   1.62MB/s [ytdl][debug] [download] Download completed
[ytdl][debug] [Merger] Merging formats into "\\?\F:\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.mkv"
[ytdl][debug] Deleting original file \\?\F:\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.f258.m4a (pass -k to keep)
[ytdl][debug] Deleting original file \\?\F:\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.f333.webm (pass -k to keep)
* F:\\dled-gallery-dl\Youtube\Youtube_hwNWx1GTSKo.mkv

sfw video:

D:\>gallery-dl -v https://www.pornhub.com/view_video.php?viewkey=ph5e3c83096dcba
[gallery-dl][debug] Version 1.25.7
[gallery-dl][debug] Python 3.11.4 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.3
[gallery-dl][debug] Configuration Files ['%APPDATA%\\gallery-dl\\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.pornhub.com/view_video.php?viewkey=ph5e3c83096dcba'
[ytdl][debug] Using YoutubeDLExtractor for 'https://www.pornhub.com/view_video.php?viewkey=ph5e3c83096dcba'
[ytdl][debug] Using <module 'yt_dlp' from 'C:\\Python3\\Lib\\site-packages\\yt_dlp\\__init__.py'>
[ytdl][debug] [PornHub] Extracting URL: https://www.pornhub.com/view_video.php?viewkey=ph5e3c83096dcba
[ytdl][debug] [PornHub] ph5e3c83096dcba: Downloading pc webpage
[ytdl][debug] [PornHub] ph5e3c83096dcba: Downloading m3u8 information
[ytdl][debug] [PornHub] ph5e3c83096dcba: Downloading m3u8 information
[ytdl][debug] [PornHub] ph5e3c83096dcba: Downloading m3u8 information
[ytdl][debug] [PornHub] ph5e3c83096dcba: Downloading m3u8 information
[ytdl][debug] [PornHub] ph5e3c83096dcba: Downloading JSON metadata
  F:\\dled-gallery-dl\PornHub\PornHub_ph5e3c83096dcba.mp4[ytdl][debug] [info] Downloading video thumbnail 0 ...
[ytdl][debug] [info] Writing video thumbnail 0 to: \\?\F:\dled-gallery-dl\PornHub\PornHub_ph5e3c83096dcba.jpg
[ytdl][debug] [info] Writing video metadata as JSON to: \\?\F:\dled-gallery-dl\PornHub\PornHub_ph5e3c83096dcba.info.json
[ytdl][debug] [download] Destination: \\?\F:\dled-gallery-dl\PornHub\PornHub_ph5e3c83096dcba.mp4
100%   4.40MB 718.37kB/s [ytdl][debug] [download] Download completed
* F:\\dled-gallery-dl\PornHub\PornHub_ph5e3c83096dcba.mp4
mikf commented 1 year ago

There's a difference between the ytdl extractor and the ytdl downloader.

For youtube or pornhub URLs, which are not supported by gallery-dl itself, and when it prints [ytdl][debug] Using YoutubeDLExtractor, it uses the config settings from extractor.ytdl.

For supported URLs like reddit, it only uses ytdl to download some files and this will get its options from downloader.ytdl.

You basically have to copy your ytdl settings and also put them under downloader:

{
    "extractor": 
    {
        "ytdl": {}
    },
    "downloader":
    {
        "ytdl": {}
    }
}
ghbook commented 1 year ago

There's a difference between the ytdl extractor and the ytdl downloader.

For youtube or pornhub URLs, which are not supported by gallery-dl itself, and when it prints [ytdl][debug] Using YoutubeDLExtractor, it uses the config settings from extractor.ytdl.

For supported URLs like reddit, it only uses ytdl to download some files and this will get its options from downloader.ytdl.

You basically have to copy your ytdl settings and also put them under downloader:

{
    "extractor": 
    {
        "ytdl": {}
    },
    "downloader":
    {
        "ytdl": {}
    }
}

Thanks, working as expected now.