Subprocess module unicode support

mbnoimi commented 9 years ago

It doesn't work under Windows in case I pointed to Unicode names (whether folder name or video name). The log doesn't show any useful info I just get this:

[04/13/15 11:53:36] WARNING: There's no description to write.

You can test it using this Arabic video under DailyMotion (this bug occurs for YouTube too) In case I pointed to Arabic folder name it failes to download. If I pointed to Latin folder name it download the file as following:

 Royal Secrets _      -1_12-   .mp4

While it should be as following:

 سلسلة عمليات إحتيال دولية -1/12- وصية السيدة العجوز.mp4

2015-04-13_12-03-56

mbnoimi commented 9 years ago

It seems this bug related to youtube-dl bug under Windows as mentioned in this issue

MrS0m30n3 commented 9 years ago

Hi, @mbnoimi

First of all, if you try to download the same video using youtube-dl.exe from the Windows command line you still get the same result? If yes can you please add the -v (verbose) option and paste the output here?

mbnoimi commented 9 years ago

@MrS0m30n3 the log please:

C:\Python 3.5\Scripts>youtube-dl.exe -v --newline -o G:\tmp\%(title)s.%(ext)s --
proxy http://127.0.0.1:8118/ -i --write-description http://www.dailymotion.com/v
ideo/x2h0g6c_%D9%88%D8%AB%D8%A7%D8%A6%D9%82%D9%8A-royal-secrets-%D8%B3%D9%84%D8%
B3%D9%84%D8%A9-%D8%B9%D9%85%D9%84%D9%8A%D8%A7%D8%AA-%D8%A5%D8%AD%D8%AA%D9%8A%D8%
A7%D9%84-%D8%AF%D9%88%D9%84%D9%8A%D8%A9-1-12-%D9%88%D8%B5%D9%8A%D8%A9-%D8%A7%D9%
84%D8%B3%D9%8A%D8%AF%D8%A9-%D8%A7%D9%84%D8%B9%D8%AC%D9%88%D8%B2_shortfilms
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '--newline', '-o', 'G:\\tmp\\%(title)s.%(ext)s
', '--proxy', 'http://127.0.0.1:8118/', '-i', '--write-description', 'http://www
.dailymotion.com/video/x2h0g6c_%D9%88%D8%AB%D8%A7%D8%A6%D9%82%D9%8A-royal-secret
s-%D8%B3%D9%84%D8%B3%D9%84%D8%A9-%D8%B9%D9%85%D9%84%D9%8A%D8%A7%D8%AA-%D8%A5%D8%
AD%D8%AA%D9%8A%D8%A7%D9%84-%D8%AF%D9%88%D9%84%D9%8A%D8%A9-1-12-%D9%88%D8%B5%D9%8
A%D8%A9-%D8%A7%D9%84%D8%B3%D9%8A%D8%AF%D8%A9-%D8%A7%D9%84%D8%B9%D8%AC%D9%88%D8%B
2_shortfilms']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2015.04.09
[debug] Python version 3.5.0a3 - Windows-post2012Server-6.3.9600
[debug] exe versions: none
[debug] Proxy map: {'https': 'http://127.0.0.1:8118/', 'http': 'http://127.0.0.1
:8118/'}
[dailymotion] x2h0g6c: Downloading webpage
[dailymotion] x2h0g6c: Extracting information
[dailymotion] x2h0g6c: Downloading embed page
WARNING: There's no description to write.
[debug] Invoking downloader on 'http://www.dailymotion.com/cdn/H264-848x480/vide
o/x2h0g6c.mp4?auth=1429130807-2562-bwfc690w-362d90e339a2f7154915a1d6a566f314'
[download] Resuming download at byte 7284256
[download] Destination: G:\tmp\?????? Royal Secrets _ ?????  ?????? ?????? ?????
 -1_12- ???? ?????? ??????.mp4
[download]   2.5% of 276.51MiB at 63.99KiB/s ETA 01:11:53
[download]   2.5% of 276.51MiB at 191.96KiB/s ETA 23:57
[download]   2.5% of 276.51MiB at 447.90KiB/s ETA 10:16
[download]   2.5% of 276.51MiB at 959.79KiB/s ETA 04:47
[download]   2.5% of 276.51MiB at 165.31KiB/s ETA 27:49
[download]   2.5% of 276.51MiB at 149.32KiB/s ETA 30:48
[download]   2.6% of 276.51MiB at 162.55KiB/s ETA 28:17
[download]   2.6% of 276.51MiB at 263.22KiB/s ETA 17:27
[download]   2.7% of 276.51MiB at 247.76KiB/s ETA 18:32
[download]   2.8% of 276.51MiB at 320.03KiB/s ETA 14:20
[download]   2.9% of 276.51MiB at 406.51KiB/s ETA 11:16
[download]   3.2% of 276.51MiB at 440.33KiB/s ETA 10:22

ERROR: Interrupted by user

C:\Python 3.5\Scripts>

mbnoimi commented 9 years ago

BTW, I used same exactly used parameters by youtube-dlg

MrS0m30n3 commented 7 years ago

Summary (work in progress)

Problem occurs in Python 2.* because the C implementation of the subprocess module can't handle unicode (http://bugs.python.org/issue1759845).

youtube-dlg uses subprocess to spawn a youtube-dl process
youtube-dl uses subprocess to spawn a ffmpeg process (https://github.com/rg3/youtube-dl/issues/4787)

(1.) youtube-dlg -> youtube-dl

Youtube-dlg does not yet support Python 3.*
On OS's that use UTF-8 (e.g. Linux) we can encode the unicode parameters at the youtube-dlg side and let the youtube-dl decode them at the other end without losing any characters
The above solution does not work on Windows since they use encodings like cp1252, cp437, etc, that can't handle unicode characters (Windows suck!!!)
On Windows we need a way to encode the unicode parameters at the youtube-dlg side using UTF-8 (we have that) and decode them at the youtube-dl side again using UTF-8 (we dont have that, see https://github.com/rg3/youtube-dl/issues/6119)

(2.) youtube-dl -> ffmpeg

Can be solved if we use a Python 3.* youtube-dl binary to do our job (https://github.com/rg3/youtube-dl/issues/5094)
Also check https://github.com/rg3/youtube-dl/issues/5527

mbnoimi commented 5 years ago

After years of waiting it seems this bug still occurs

elico commented 4 years ago

@MrS0m30n3 How can someone resolve this bug?

mbnoimi commented 4 years ago

@MrS0m30n3 How can someone resolve this bug?

This project no longer maintained so I currently use https://github.com/axcore/tartube it's full of advanced options in addition to the classic ui

MrS0m30n3 / youtube-dl-gui