Exception is thrown if NTFS partition is mounted with 'windows_names' option and a directory or file name in the output template ends with a dot

ghost commented 8 years ago

[x] I've verified and I assure that I'm running youtube-dl 2016.05.16

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[X] Searched the bugtracker for similar issues including closed ones
What is the purpose of your issue?
[X] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

Log output

$ youtube-dl -v -o "%(uploader)s/%(playlist)s/%(playlist_index)s. %(title)s-%(id)s.%(ext)s" "https://www.youtube.com/playlist?list=PLAZNU5fM7FIBqwbWDY3tBK5EEz16tax0e"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '-o', '%(uploader)s/%(playlist)s/%(playlist_index)s. %(title)s-%(id)s.%(ext)s', 'https://www.youtube.com/playlist?list=PLAZNU5fM7FIBqwbWDY3tBK5EEz16tax0e']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.05.16
[debug] Python version 3.5.1 - Linux-4.5.4-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.0.2, ffprobe 3.0.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube:playlist] PLAZNU5fM7FIBqwbWDY3tBK5EEz16tax0e: Downloading webpage
[download] Downloading playlist: Singles/Misc.
[youtube:playlist] playlist Singles/Misc.: Downloading 7 videos
[download] Downloading video 1 of 7
[youtube] PiMJ6Sy2ph4: Downloading webpage
[youtube] PiMJ6Sy2ph4: Downloading video info webpage
[youtube] PiMJ6Sy2ph4: Extracting video information
[youtube] PiMJ6Sy2ph4: Downloading MPD manifest
ERROR: unable to create directory [Errno 22] Invalid argument: 'Proximity/Singles_Misc.'
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1497, in process_info
    os.makedirs(dn)
  File "/usr/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
OSError: [Errno 22] Invalid argument: 'Proximity/Singles_Misc.'

Description of the issue

NTFS partitions mounted with the windows_names option disallows files and directories from containing <>:"/\|?* and from ending with a .. I believe the former is taken care of by youtube-dl, but the latter is not. Python will throw a ERROR: unable to create directory [Errno 22] Invalid argument exception when you try to create a directory ending with a ..

In my example, I try to download a playlist named Singles/Misc. with the output template %(uploader)s/%(playlist)s/%(playlist_index)s. %(title)s-%(id)s.%(ext)s. If the playlist's name, and I presume the uploader's name, ends with . the exception will be thrown.

Example /etc/fstab for mounting an NTFS partition with the windows_names option:

/dev/sda1 /mnt/ntfs-partition ntfs-3g defaults,windows_names 0 0

Suggested solution

I see three main alternatives, although if any of you find a more creative solution then please let me know:

Detect if the partition the file/directory is being written to is NTFS and mounted with the windows_names option and follow Windows filename compatibility if that is the case.
Make the --restrict-filenames option also replace dots from the end of directory names and filenames.
Always replace dots from the end of all directories and filenames.

NOTE: Any solution would need to replace an arbitrary amount of dots from the end of the file and directory names, as their names could contain multiple dots: playlist name...

yan12125 commented 8 years ago

As a side note, from http://linux.die.net/man/8/mount.ntfs-3g:

windows_names
    This option prevents files, directories and extended attributes to be created with a 
    name not allowed by windows, either because it contains some not allowed character
    (which are the nine characters " * / : < > ? \ | and those whose code is less than 0x20)
    or because the last character is a space or a dot. Existing such files can still be read
    (and renamed).

yan12125 commented 8 years ago

Oh sanitize_open already implements logics for generating valid Windows paths. However, detecting windows_names is not practical as there are other NTFS implementations. An option like --use-windows-path may a good idea. This option can be useful for those who share files between NTFS and non-NTFS filesystems, too.

ytdl-org / youtube-dl