mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.81k stars 968 forks source link

[ex/e-hentai][windows] Download fails if folder name too long #715

Closed colpocleisis closed 4 years ago

colpocleisis commented 4 years ago

Similar to #545 v1.13.5 This error occur only when gallery-dl is run inside a VM, does NOT occur if run on host machine. Only occurs with folders with long names (occurred twice in 300 galleries downloads).

My setup is having multiple guest VM point to the same host shared folder. Each VM is connected to VPN with different IP to avoid SadPanda bans.

[exhentai][error] Unable to download data: OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\Z:\\gallery-dl\\Doujin ENG\\Exhentai\\(C97) [Nhoooooooooooooo (Oosawara Sadao)] Dorei Shounin ni Yowami o Nigirarete Muyou de Nama Onaho Atsukai o Ukeru Y Buta & R Ko no Cosplay Hamedori Hon (Taimanin Yukikaze) [English] [slathe] - 1622524\\' [exhentai][debug] Traceback (most recent call last): File "gallery_dl\job.py", line 50, in run File "gallery_dl\job.py", line 88, in dispatch File "gallery_dl\job.py", line 258, in handle_directory File "gallery_dl\job.py", line 343, in initialize File "gallery_dl\util.py", line 765, in set_directory File "os.py", line 223, in makedirs

mikf commented 4 years ago

The directory name isn't even that long. 236 characters in total, 201 for the gallery title, so it shouldn't hit any limits ...

Well, as in #545, you can limit the gallery title length with something like {title[:150]}, but I have absolutely no idea what's really going on here. Maybe, when accessing a network share (your host shared folder,) it does some path wrangling under the hood, strips the \\?\ from the beginning, and hits the regular 260 character limit.

Have you tried disabling Windows' path length limit? You are running Windows for both host and guest systems, right?


Also, you can wrap tracebacks etc. in triple backticks to format them nicely

``` your traceback here ```

becomes

your traceback here
colpocleisis commented 4 years ago

Yes, it is enabled and yes, both host and guest are on windows I will be testing with {title[:150]} to see if it will work.

Running the same command on host does not cause the issue the command I'm running is

gallery-dl -i "C:\Users\%USERNAME%\Downloads\Firefox\SadPanda\favorites(doujinENG).txt" -d "C:\Users\%USERNAME%\Downloads\gallery-dl\Doujin ENG" --write-log "C:\Users\%USERNAME%\Downloads\gallery-dl\Doujin ENG\log.txt"

If it changes anything the path for the shared folder in my host computer is C:\Users\%USERNAME%\Downloads\gallery-dl\Doujin ENG\Exhentai so that might be an issue

I have attached the full log in case it can be of any help. As you can see it appears to only occur when the names are rather long logVM3.txt

mikf commented 4 years ago

The actual path on your host machine is, depending on %USERNAME%, longer than 260 characters:

C:\Users\%USERNAME%\Downloads\gallery-dl\Doujin ENG\Exhentai\(C97) [Nhoooooooooooooo (Oosawara Sadao)] Dorei Shounin ni Yowami o Nigirarete Muyou de Nama Onaho Atsukai o Ukeru Y Buta & R Ko no Cosplay Hamedori Hon (Taimanin Yukikaze) [English] [slathe] - 1622524\
('C:\' + 249 + USERNAME + '\0')

So, if your username is longer than 7 characters, you'd go over the 260 char limit. That's usually not an issue, because gallery-dl prepends paths with \\?\ to disable this limit, but I guess the guest-to-host interface of your VM software ignores and/or strips it.

And enabling NTFS long paths in your Windows settings doesn't work for all programs:

Enabling NTFS long paths will allow manifested win32 applications and Windows Store applications to access paths beyond the normal 260 char limit per node.

(from https://mspoweruser.com/ntfs-260-character-windows-10/)

colpocleisis commented 4 years ago

Reporting back: my %USERNAME% is 3 characters so it shouldn't go over 260 char limit.

Using {title[0:150]} seems to fix the issue. As you have said it's probably due to some VMware shared folder PATH shenanigans.