search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.76k stars 963 forks source link

[ytdl] Directory passed to yt-dlp makes aria2c downloader error on attempted directory creation #3021

Closed Bubsy-26d closed 2 years ago

Bubsy-26d commented 2 years ago

I'm attempting to use the aria2c downloader for yt-dlp to increase download speeds, but am running into download failures as it seems gallery-dl is sending in a weirdly-formatted directory structure that aria2c doesn't know what to do with.

When calling yt-dlp through gallery-dl, it attempts to download to the directory "\\?\Z:\YT Channels\HD Mixtapes\", while using yt-dlp directly uses "Z:\YT Channels\HD Mixtapes\" as its directory.

From the verbose gallery-dl output:

gallery-dl -v --no-skip -o extractor.ytdl.cmdline-args="--downloader-args aria2c:'-l log2.txt'" ytdl:https://www.youtube.com/watch?v=cOYmxRclKwQ
[gallery-dl][debug] Version 1.23.2
[gallery-dl][debug] Python 3.10.5 - Windows-10-10.0.19044-SP0
[gallery-dl][debug] requests 2.28.0 - urllib3 1.26.9
[gallery-dl][debug] Starting DownloadJob for 'ytdl:https://www.youtube.com/watch?v=cOYmxRclKwQ'
[ytdl][debug] Using YoutubeDLExtractor for 'ytdl:https://www.youtube.com/watch?v=cOYmxRclKwQ'
[ytdl][debug] Using <module 'yt_dlp' from 'C:\\Users\\{username}\\AppData\\Roaming\\Python\\Python310\\site-packages\\yt_dlp\\__init__.py'>
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading webpage
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading android player API JSON
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading MPD manifest
[ytdl][debug] [youtube] cOYmxRclKwQ: Downloading MPD manifest
[ytdl][debug] Using download archive './gallery-dl.db'
Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].mkv[ytdl][debug] [info] Downloading video thumbnail 41 ...
[ytdl][debug] [info] Writing video thumbnail 41 to: \\?\Z:\YT Channels\HD Mixtapes\Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].webp
[ytdl][debug] [download] Destination: \\?\Z:\YT Channels\HD Mixtapes\Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4

10/06 02:08:56 [ERROR] CUID#7 - Download aborted. URI=https://rr5---sn-vgqsrnsd.googlevideo.com/videoplayback?expire=1665061734&ei=Bn8-Y5zaELSb2_gPtriY-Ao&ip=147.219.172.43&id=o-AM1bYJV8pHKFMc3snbRKCi-JOujdQMGllCDrLpWGowy9&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C26&mn=sn-vgqsrnsd%2Csn-q4fl6nsy&ms=au%2Conr&mv=m&mvi=5&pl=16&initcwndbps=2021250&spc=yR2vp2_OWW7sZkg0rXQAyB8a_QuSqsg&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665039676&fvip=3&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRAIgT66SqUekF7O5HJ65B0FpPQHiHlRBHRDxQadTDM-kbxACIF8PSKxqTUVkeed59Ju8l5WKdGdORXVdGZSG1caWc-rd&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALBRFSSL2tSf9jdRiY-JokZSdn1aHvYYgmLqz7LEOc37AiEA4JvPabVc2xbl5Wm_XZPnYKtvKF63_CFVtmtnJ1EwVgU%3D
Exception: [AbstractCommand.cc:403] errorCode=18 URI=https://rr5---sn-vgqsrnsd.googlevideo.com/videoplayback?expire=1665061734&ei=Bn8-Y5zaELSb2_gPtriY-Ao&ip=147.219.172.43&id=o-AM1bYJV8pHKFMc3snbRKCi-JOujdQMGllCDrLpWGowy9&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C26&mn=sn-vgqsrnsd%2Csn-q4fl6nsy&ms=au%2Conr&mv=m&mvi=5&pl=16&initcwndbps=2021250&spc=yR2vp2_OWW7sZkg0rXQAyB8a_QuSqsg&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665039676&fvip=3&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRAIgT66SqUekF7O5HJ65B0FpPQHiHlRBHRDxQadTDM-kbxACIF8PSKxqTUVkeed59Ju8l5WKdGdORXVdGZSG1caWc-rd&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALBRFSSL2tSf9jdRiY-JokZSdn1aHvYYgmLqz7LEOc37AiEA4JvPabVc2xbl5Wm_XZPnYKtvKF63_CFVtmtnJ1EwVgU%3D
  -> [RequestGroup.cc:761] errorCode=18 Download aborted.
  -> [util.cc:1948] errNum=17 errorCode=18 Failed to make the directory //?/Z:/YT Channels/HD Mixtapes//., cause: File exists

From the aria2c logs:

2022-10-06 01:55:29.475633 [INFO] [DefaultBtProgressInfoFile.cc:415] The segment file //?/Z:/YT Channels/HD Mixtapes//./Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4.part.aria2 does not exist.
2022-10-06 01:55:29.475633 [DEBUG] [File.cc:203] UNC Prefix //?/Z:
2022-10-06 01:55:29.475633 [DEBUG] [File.cc:234] Making directory //?/Z:/YT Channels
2022-10-06 01:55:29.475633 [DEBUG] [File.cc:240] Failed to create //?/Z:/YT Channels
2022-10-06 01:55:29.475633 [ERROR] [AbstractCommand.cc:401] CUID#7 - Download aborted. URI=https://rr5---sn-vgqsknek.googlevideo.com/videoplayback?expire=1665060929&ei=4Xs-Y7_0A8uLlu8P-bGkSA&ip=147.219.172.43&id=o-AAWy9eNE9BNjErMuFYcFbSgsgyW3oeh_HzNO9RXtqPSy&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C29&mn=sn-vgqsknek%2Csn-vgqsrnsd&ms=au%2Crdu&mv=m&mvi=5&pl=16&initcwndbps=2083750&spc=yR2vp72ztODvimxPGgQXh8fUN-VJOSo&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665038945&fvip=5&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAI9pf7uAYkz78zJqHcYVm_YkrvUKOzK8zaKV4meKaXwyAiEA3G2dsKKtca1bgJjAfggYgt8q6j5kzjRwvLLv71co3nY%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAP-nDb5KI-FlgWEQy9yQ0XxiJeGzZ3FMA0na5FY1L5BuAiEA_njlC3ATY3J1Tti4v6GpiZetMQlrCGwrvI3r9V8BnxI%3D
Exception: [AbstractCommand.cc:403] errorCode=18 URI=https://rr5---sn-vgqsknek.googlevideo.com/videoplayback?expire=1665060929&ei=4Xs-Y7_0A8uLlu8P-bGkSA&ip=147.219.172.43&id=o-AAWy9eNE9BNjErMuFYcFbSgsgyW3oeh_HzNO9RXtqPSy&itag=137&source=youtube&requiressl=yes&mh=-H&mm=31%2C29&mn=sn-vgqsknek%2Csn-vgqsrnsd&ms=au%2Crdu&mv=m&mvi=5&pl=16&initcwndbps=2083750&spc=yR2vp72ztODvimxPGgQXh8fUN-VJOSo&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=365566839&otfp=1&dur=5613.707&lmt=1661811981738595&mt=1665038945&fvip=5&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=6216224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAI9pf7uAYkz78zJqHcYVm_YkrvUKOzK8zaKV4meKaXwyAiEA3G2dsKKtca1bgJjAfggYgt8q6j5kzjRwvLLv71co3nY%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAP-nDb5KI-FlgWEQy9yQ0XxiJeGzZ3FMA0na5FY1L5BuAiEA_njlC3ATY3J1Tti4v6GpiZetMQlrCGwrvI3r9V8BnxI%3D
  -> [RequestGroup.cc:761] errorCode=18 Download aborted.
  -> [util.cc:1948] errNum=17 errorCode=18 Failed to make the directory //?/Z:/YT Channels/HD Mixtapes//., cause: File exists

Compared to the aria2c logs of using yt-dlp directly:

2022-10-06 01:58:06.155875 [INFO] [DefaultBtProgressInfoFile.cc:415] The segment file Z:/YT Channels/HD Mixtapes/./Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4.part.aria2 does not exist.
2022-10-06 01:58:06.155875 [DEBUG] [DefaultPieceStorage.cc:168] usedPieces_.size()=1
2022-10-06 01:58:06.155875 [DEBUG] [SegmentMan.cc:140] Attach segment#0 to CUID#7.
2022-10-06 01:58:06.155875 [DEBUG] [SegmentMan.cc:146] Flushing cached data, size=0
2022-10-06 01:58:06.155875 [DEBUG] [WrDiskCache.cc:95] Update cache entry size=0, delta=0, clock=1
2022-10-06 01:58:06.155875 [DEBUG] [SegmentMan.cc:163] index=0, length=1048576, segmentLength=1048576, writtenLength=0
2022-10-06 01:58:06.156375 [DEBUG] [WrDiskCache.cc:95] Update cache entry size=0, delta=0, clock=2
2022-10-06 01:58:06.156877 [INFO] [DefaultBtProgressInfoFile.cc:205] Saving the segment file Z:/YT Channels/HD Mixtapes/./Heckman - Into Your Subconscious (2000) [HD] [cOYmxRclKwQ].f137.mp4.part.aria2

I'm seeing this behavior no matter what drive or directory I download to. The yt-dlp native downloader seems to handle the passed-in directory... kind of. I've been having issues where the first time I attempt downloading a file it's 50-50 whether yt-dlp will error out on attempting to download the first fragments. Deleting the files it created and re-downloading usually works the 2nd time. I don't have any settings for directory, base-directory, or any extractor-specific output directory settings in my gallery-dl config, nor in my yt-dlp config.

mikf commented 2 years ago

it seems gallery-dl is sending in a weirdly-formatted directory structure \?\Z:\YT Channels\HD Mixtapes

Prepending a path with \\?\ is perfectly valid, but a lot of tools throw an error when they get a path like this.

Initially this path modification was done to get around the 260 character limit for filesystem paths, but it shouldn't be all that necessary anymore with Win10 now having native support for long paths.

I'll just add an option to disable this behavior.

Bubsy-26d commented 2 years ago

Interesting, I've never seen that before. But an option to disable sounds like it would handle this, thanks!

mikf commented 2 years ago

These are called "extended-length path[s]"

… the maximum length for a path is MAX_PATH, which is defined as 260 characters To specify an extended-length path, use the "\?\" prefix. For example, "\?\D:\very long path".

(https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation) (https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#win32-file-namespaces)

Anyway, with commit https://github.com/mikf/gallery-dl/commit/7f30a0d7a78e0e51fde2405952b9bc9da2330897 you can disable them by setting path-extended to false