RTMP downloader does not support unicode filenames

bbepis commented 4 years ago

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2020.01.15
[ ] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--write-info-json', '--write-sub', '--all-subs', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://live.nicovideo.jp/watch/lv309042194', '--verbose', '-i']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2020.01.15
[debug] Git HEAD: 702405f05
[debug] Python version 3.6.6 (CPython) - Windows-10-10.0.17134-SP0
[debug] exe versions: ffmpeg git-2019-12-23-5b42d33, ffprobe git-2019-12-23-5b42d33, rtmpdump 2.3
[debug] Proxy map: {}
[NiconicoLive] Logging in
[NiconicoLive] lv309042194: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video description metadata as JSON to: ロックマン30周年記念生放送-lv309042194.info.json
[debug] Invoking downloader on 'rtmp://nlaoe113.live.nicovideo.jp:1935/fileorigin/04/mp4:/content/20171213/lv309042194_184426134000_1_4d071f.f4v'
[download] Destination: ロックマン30周年記念生放送-lv309042194.flv
[debug] rtmpdump command line: rtmpdump --verbose -r "rtmp://nlaoe113.live.nicovideo.jp:1935/fileorigin/04/mp4:/content/20171213/lv309042194_184426134000_1_4d071f.f4v" -o ロックマン30周年記念生放送-lv309042194.flv.part --conn "S:90858899:lv309042194:0:1579272301:305f6ffbe8ac5033" --resume --skip 1
[rtmpdump] RTMPDump v2.3
[rtmpdump] (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
[rtmpdump] DEBUG: Parsing...
[rtmpdump] DEBUG: Parsed protocol: 0
[rtmpdump] DEBUG: Parsed host    : nlaoe113.live.nicovideo.jp
[rtmpdump] DEBUG: Parsed app     : fileorigin/04
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Protocol : RTMP
[rtmpdump] DEBUG: Hostname : nlaoe113.live.nicovideo.jp
[rtmpdump] DEBUG: Port     : 1935
[rtmpdump] DEBUG: Playpath : mp4:/content/20171213/lv309042194_184426134000_1_4d071f.f4v
[rtmpdump] DEBUG: tcUrl    : rtmp://nlaoe113.live.nicovideo.jp:1935/fileorigin/04
[rtmpdump] DEBUG: app      : fileorigin/04
[rtmpdump] DEBUG: live     : no
[rtmpdump] DEBUG: timeout  : 30 sec
[rtmpdump] Failed to open file! ?????30???????-lv309042194.flv.part
ERROR: unable to download video data: [WinError 2] The system cannot find the file specified: 'ロックマン30周年記念生放送-lv309042194.flv.part'
Traceback (most recent call last):
  File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\YoutubeDL.py", line 1948, in process_info
    success = dl(filename, info_dict)
  File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\YoutubeDL.py", line 1887, in dl
    return fd.download(name, info)
  File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\downloader\common.py", line 395, in download
    return self.real_download(filename, info_dict)
  File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\downloader\rtmp.py", line 183, in real_download
    prevsize = os.path.getsize(encodeFilename(tmpfilename))
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'ロックマン30周年記念生放送-lv309042194.flv.part'

Description

I've been working on implementing #17415 which uses RTMP, but I've discovered that it doesn't support unicode filenames. This can be tested with any extractor that uses the RTMP downloader, and specifying the -f format flag with a filename that uses unicode. Inversely, my particular issue with the above log is fixed if I specify an ASCII filename.

After some investigation, this seems to be an issue with rtmpdump itself as its code uses char instead of wchar for output filenames, meaning that it's incapable of actually using unicode.

There are multiple fixes that could be used for this:

Using a different temporary filename
Manually piping stdout from the process to a file handle created by python
Using an altered version of rtmpdump/librtmp that has support for unicode filenames
Using a different application entirely, such as ffmpeg. I am aware that the zeranoe builds do not ship with librtmp by default, so this may be limited

remitamine commented 4 years ago

works fine with both rtmpdump and FFmpeg:

python __main__.py -v -f rtmp_sd -o '「學而時習之.%(ext)s' https://www.webofstories.com/play/julia.hartwig/16
[debug] System config: []
[debug] User config: ['-f', 'best[height<=768]/bestvideo[height<=768]+bestaudio/best', '--hls-prefer-ffmpeg', '--external-downloader', 'curl', '--sub-format', 'ass/tt/ttml/dfxp/scc/vtt/srt/sami/best']
[debug] Custom config: []
[debug] Command-line args: ['-v', '-f', 'rtmp_sd', '-o', '「學而時習之.%(ext)s', 'https://www.webofstories.com/play/julia.hartwig/16']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.01.15
[debug] Git HEAD: c968f738d
[debug] Python version 3.8.1 (CPython) - Linux-5.4.12-arch1-1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[WebOfStories] 16: Downloading webpage
[debug] Invoking downloader on 'rtmp://eu-cdn1.webofstories.com/cfx/st/'
[download] Destination: 「學而時習之.flv
[debug] rtmpdump command line: rtmpdump --verbose -r rtmp://eu-cdn1.webofstories.com/cfx/st/ -o '「學而時習之.flv.part' --pageUrl https://www.webofstories.com/play/julia.hartwig/16 --playpath stories/50026/16 --resume --skip 1
[rtmpdump] RTMPDump v2.4
[rtmpdump] (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
[rtmpdump] DEBUG: Parsing...
[rtmpdump] DEBUG: Parsed protocol: 0
[rtmpdump] DEBUG: Parsed host    : eu-cdn1.webofstories.com
[rtmpdump] DEBUG: Parsed app     : cfx/st
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Protocol : RTMP
[rtmpdump] DEBUG: Hostname : eu-cdn1.webofstories.com
[rtmpdump] DEBUG: Port     : 1935
[rtmpdump] DEBUG: Playpath : stories/50026/16
[rtmpdump] DEBUG: tcUrl    : rtmp://eu-cdn1.webofstories.com:1935/cfx/st
[rtmpdump] DEBUG: pageUrl  : https://www.webofstories.com/play/julia.hartwig/16
[rtmpdump] DEBUG: app      : cfx/st
[rtmpdump] DEBUG: live     : no
[rtmpdump] DEBUG: timeout  : 30 sec
[rtmpdump] DEBUG: Setting buffer time to: 36000000ms
[rtmpdump] Connecting ...
[rtmpdump] DEBUG: RTMP_Connect1, ... connected, handshaking
[rtmpdump] DEBUG: HandShake: Type Answer   : 03
[rtmpdump] DEBUG: HandShake: Server Uptime : 335325041
[rtmpdump] DEBUG: HandShake: FMS Version   : 3.5.7.1
[rtmpdump] WARNING: HandShake: client signature does not match!
[rtmpdump] DEBUG: HandShake: Handshaking finished....
[rtmpdump] DEBUG: RTMP_Connect1, handshaked
[rtmpdump] DEBUG: Invoking connect
[rtmpdump] INFO: Connected...
[rtmpdump] DEBUG: HandleServerBW: server BW = 1310720
[rtmpdump] DEBUG: HandleClientBW: client BW = 2500000 2
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 242 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:             fmsVer, STRING:  FMS/3,5,7,7009>
[rtmpdump] DEBUG: Property: <Name:       capabilities, NUMBER:  127.00>
[rtmpdump] DEBUG: Property: <Name:               mode, NUMBER:  1.00>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:              level, STRING:  status>
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetConnection.Connect.Success>
[rtmpdump] DEBUG: Property: <Name:        description, STRING:  Connection succeeded.>
[rtmpdump] DEBUG: Property: <Name:     objectEncoding, NUMBER:  0.00>
[rtmpdump] DEBUG: Property: <Name:               data, ECMA_ARRAY>
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:            version, STRING:  3,5,7,7009>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_result>
[rtmpdump] DEBUG: HandleInvoke, received result for method call <connect>
[rtmpdump] DEBUG: sending ctrl. type: 0x0003
[rtmpdump] DEBUG: Invoking createStream
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 21 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <onBWDone>
[rtmpdump] DEBUG: Invoking _checkbw
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 29 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_result>
[rtmpdump] DEBUG: HandleInvoke, received result for method call <createStream>
[rtmpdump] DEBUG: SendPlay, seekTime=0, stopTime=0, sending play: stories/50026/16
[rtmpdump] DEBUG: Invoking play
[rtmpdump] DEBUG: sending ctrl. type: 0x0003
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 16419 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 32803 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[rtmpdump] DEBUG: HandleChangeChunkSize, received: chunk size change to 4096
[rtmpdump] DEBUG: HandleCtrl, received ctrl. type: 4, len: 6
[rtmpdump] DEBUG: HandleCtrl, Stream IsRecorded 1
[rtmpdump] DEBUG: HandleCtrl, received ctrl. type: 0, len: 6
[rtmpdump] DEBUG: HandleCtrl, Stream Begin 1
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 174 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:              level, STRING:  status>
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetStream.Play.Reset>
[rtmpdump] DEBUG: Property: <Name:        description, STRING:  Playing and resetting stories/50026/16.>
[rtmpdump] DEBUG: Property: <Name:            details, STRING:  stories/50026/16>
[rtmpdump] DEBUG: Property: <Name:           clientid, STRING:  RNrg1PnO>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <onStatus>
[rtmpdump] DEBUG: HandleInvoke, onStatus: NetStream.Play.Reset
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 168 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:              level, STRING:  status>
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetStream.Play.Start>
[rtmpdump] DEBUG: Property: <Name:        description, STRING:  Started playing stories/50026/16.>
[rtmpdump] DEBUG: Property: <Name:            details, STRING:  stories/50026/16>
[rtmpdump] DEBUG: Property: <Name:           clientid, STRING:  RNrg1PnO>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <onStatus>
[rtmpdump] DEBUG: HandleInvoke, onStatus: NetStream.Play.Start
[rtmpdump] Starting download at: 0.000 kB
[rtmpdump] DEBUG: RTMP_ClientPacket, received: notify 24 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: RTMP_ClientPacket, received: notify 44 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetStream.Data.Start>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: RTMP_ClientPacket, received: notify 268 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:           duration, NUMBER:  88.82>
[rtmpdump] DEBUG: Property: <Name:              width, NUMBER:  640.00>
[rtmpdump] DEBUG: Property: <Name:             height, NUMBER:  360.00>
[rtmpdump] DEBUG: Property: <Name:      videodatarate, NUMBER:  625.00>
[rtmpdump] DEBUG: Property: <Name:          framerate, NUMBER:  25.00>
[rtmpdump] DEBUG: Property: <Name:       videocodecid, NUMBER:  7.00>
[rtmpdump] DEBUG: Property: <Name:      audiodatarate, NUMBER:  93.75>
[rtmpdump] DEBUG: Property: <Name:    audiosamplerate, NUMBER:  44100.00>
[rtmpdump] DEBUG: Property: <Name:    audiosamplesize, NUMBER:  16.00>
[rtmpdump] DEBUG: Property: <Name:             stereo, BOOLEAN: TRUE>
[rtmpdump] DEBUG: Property: <Name:       audiocodecid, NUMBER:  10.00>
[rtmpdump] DEBUG: Property: <Name:           filesize, NUMBER:  8327041.00>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] INFO: Metadata:
[rtmpdump] INFO:   duration              88.82
[rtmpdump] INFO:   width                 640.00
[rtmpdump] INFO:   height                360.00
[rtmpdump] INFO:   videodatarate         625.00
[rtmpdump] INFO:   framerate             25.00
[rtmpdump] INFO:   videocodecid          7.00
[rtmpdump] INFO:   audiodatarate         93.75
[rtmpdump] INFO:   audiosamplerate       44100.00
[rtmpdump] INFO:   audiosamplesize       16.00
[rtmpdump] INFO:   stereo                TRUE
[rtmpdump] INFO:   audiocodecid          10.00
[rtmpdump] INFO:   filesize              8327041.00
[download] 22.90KiB at Unknown speed (00:01)
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 49187 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[download]   5.5% of ~7.93MiB at 125.50KiB/s ETA 01:00
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 65573 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 81957 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 98341 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 114725 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[download]   6.8% of ~8.26MiB at 123.47KiB/s ETA 01:04
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 131109 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[download]  11.3% of ~7.73MiB at 206.22KiB/s ETA 00:33
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 40 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwdone>
[download]  99.8% of ~7.95MiB at 328.91KiB/s ETA 00:00
[rtmpdump] DEBUG: RTMP_ClientPacket, received: notify 102 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetStream.Play.Complete>
[rtmpdump] DEBUG: Property: <Name:              level, STRING:  status>
[rtmpdump] DEBUG: Property: <Name:           duration, NUMBER:  19.00>
[rtmpdump] DEBUG: Property: <Name:              bytes, NUMBER:  8327378.00>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleCtrl, received ctrl. type: 1, len: 6
[rtmpdump] DEBUG: HandleCtrl, Stream EOF 1
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 178 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:              level, STRING:  status>
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetStream.Play.Stop>
[rtmpdump] DEBUG: Property: <Name:        description, STRING:  Stopped playing stories/50026/16.>
[rtmpdump] DEBUG: Property: <Name:            details, STRING:  stories/50026/16>
[rtmpdump] DEBUG: Property: <Name:           clientid, STRING:  RNrg1PnO>
[rtmpdump] DEBUG: Property: <Name:             reason, STRING:  >
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <onStatus>
[rtmpdump] DEBUG: HandleInvoke, onStatus: NetStream.Play.Stop
[rtmpdump] WARNING: Larger timestamp than 24-bit: 0xffffff66
[rtmpdump] DEBUG: Invoking deleteStream
[rtmpdump] DEBUG: Got Play.Complete or Play.Stop from server. Assuming stream is complete
[download]  99.9% of ~7.95MiB at 329.16KiB/s ETA 00:00
[rtmpdump] DEBUG: RTMP_Read returned: 0
[rtmpdump] Download complete
[rtmpdump] DEBUG: Closing connection.
[rtmpdump] Downloaded 8327041 bytes
[download] 100% of 7.94MiB in 00:24

python __main__.py -v --external-downloader ffmpeg -f rtmp_sd -o '「學而時習之.%(ext)s' https://www.webofstories.com/play/julia.hartwig/16
[debug] System config: []
[debug] User config: ['-f', 'best[height<=768]/bestvideo[height<=768]+bestaudio/best', '--hls-prefer-ffmpeg', '--external-downloader', 'curl', '--sub-format', 'ass/tt/ttml/dfxp/scc/vtt/srt/sami/best']
[debug] Custom config: []
[debug] Command-line args: ['-v', '--external-downloader', 'ffmpeg', '-f', 'rtmp_sd', '-o', '「學而時習之.%(ext)s', 'https://www.webofstories.com/play/julia.hartwig/16']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.01.15
[debug] Git HEAD: c968f738d
[debug] Python version 3.8.1 (CPython) - Linux-5.4.12-arch1-1-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[WebOfStories] 16: Downloading webpage
[debug] Invoking downloader on 'rtmp://eu-cdn1.webofstories.com/cfx/st/'
[download] Destination: 「學而時習之.flv
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -rtmp_pageurl https://www.webofstories.com/play/julia.hartwig/16 -rtmp_playpath stories/50026/16 -i rtmp://eu-cdn1.webofstories.com/cfx/st/ -c copy -f flv 'file:「學而時習之.flv.part'
ffmpeg version n4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[tcp @ 0x56136d2cf580] Starting connection attempt to 52.222.160.21 port 1935
[tcp @ 0x56136d2cf580] Successfully connected to 52.222.160.21 port 1935
[h264 @ 0x56136d2d7d80] Reinit context to 640x368, pix_fmt: yuv420p
Input #0, flv, from 'rtmp://eu-cdn1.webofstories.com/cfx/st/':
  Duration: 00:01:28.82, start: 0.000000, bitrate: N/A
    Stream #0:0: Data: none
    Stream #0:1: Video: h264 (High), 1 reference frame, yuv420p(progressive, left), 640x360 (640x368) [SAR 1:1 DAR 16:9], 640 kb/s, 25 fps, 25 tbr, 1k tbn, 50 tbc
    Stream #0:2: Audio: aac (LC), 44100 Hz, stereo, fltp, 96 kb/s
Output #0, flv, to 'file:「學而時習之.flv.part':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (High), 1 reference frame ([7][0][0][0] / 0x0007), yuv420p(progressive, left), 640x360 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 640 kb/s, 25 fps, 25 tbr, 1k tbn, 1k tbc
    Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:2 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  101 fps=0.0 q=-1.0 size=     256kB time=00:00:04.07 bitrate= 514.8kbits/frame=  192 fps= 53 q=-1.0 size=     512kB time=00:00:07.71 bitrate= 543.4kbits/frame=  251 fps= 60 q=-1.0 size=     768kB time=00:00:10.06 bitrate= 625.1kbits/frame=  282 fps= 33 q=-1.0 size=    1024kB time=00:00:11.29 bitrate= 742.7kbits/frame=  302 fps= 33 q=-1.0 size=    1024kB time=00:00:12.08 bitrate= 694.4kbits/frame=  342 fps= 35 q=-1.0 size=    1024kB time=00:00:13.68 bitrate= 613.2kbits/frame=  363 fps= 33 q=-1.0 size=    1280kB time=00:00:14.54 bitrate= 720.9kbits/frame=  422 fps= 36 q=-1.0 size=    1536kB time=00:00:16.89 bitrate= 744.9kbits/frame=  457 fps= 29 q=-1.0 size=    1536kB time=00:00:18.28 bitrate= 688.3kbits/frame=  473 fps= 28 q=-1.0 size=    1536kB time=00:00:18.95 bitrate= 663.7kbits/frame=  493 fps= 28 q=-1.0 size=    1792kB time=00:00:19.72 bitrate= 744.3kbits/frame=  505 fps= 27 q=-1.0 size=    1792kB time=00:00:20.20 bitrate= 726.7kbits/frame=  529 fps= 27 q=-1.0 size=    1792kB time=00:00:21.16 bitrate= 693.7kbits/frame=  605 fps= 30 q=-1.0 size=    2048kB time=00:00:24.20 bitrate= 693.2kbits/frame=  651 fps= 29 q=-1.0 size=    2304kB time=00:00:26.06 bitrate= 724.2kbits/frame=  706 fps= 31 q=-1.0 size=    2304kB time=00:00:28.24 bitrate= 668.2kbits/frame=  749 fps= 27 q=-1.0 size=    2560kB time=00:00:29.96 bitrate= 700.0kbits/frame=  781 fps= 28 q=-1.0 size=    2560kB time=00:00:31.24 bitrate= 671.3kbits/frame=  811 fps= 28 q=-1.0 size=    2816kB time=00:00:32.47 bitrate= 710.4kbits/frame=  861 fps= 29 q=-1.0 size=    3072kB time=00:00:34.44 bitrate= 730.6kbits/frame=  879 fps= 29 q=-1.0 size=    3072kB time=00:00:35.18 bitrate= 715.2kbits/frame=  915 fps= 28 q=-1.0 size=    3072kB time=00:00:36.60 bitrate= 687.6kbits/frame= 1000 fps= 28 q=-1.0 size=    3584kB time=00:00:40.01 bitrate= 733.7kbits/frame= 1089 fps= 28 q=-1.0 size=    3840kB time=00:00:43.57 bitrate= 722.0kbits/frame= 1181 fps= 28 q=-1.0 size=    4096kB time=00:00:47.24 bitrate= 710.3kbits/frame= 1262 fps= 27 q=-1.0 size=    4352kB time=00:00:50.51 bitrate= 705.8kbits/frame= 1360 fps= 27 q=-1.0 size=    4864kB time=00:00:54.41 bitrate= 732.3kbits/frame= 1448 fps= 27 q=-1.0 size=    5120kB time=00:00:57.92 bitrate= 724.1kbits/frame= 1535 fps= 27 q=-1.0 size=    5376kB time=00:01:01.40 bitrate= 717.3kbits/frame= 1622 fps= 27 q=-1.0 size=    5632kB time=00:01:04.91 bitrate= 710.8kbits/frame= 1706 fps= 26 q=-1.0 size=    6144kB time=00:01:08.25 bitrate= 737.4kbits/frame= 1782 fps= 27 q=-1.0 size=    6400kB time=00:01:11.31 bitrate= 735.1kbits/frame= 1798 fps= 26 q=-1.0 size=    6400kB time=00:01:11.92 bitrate= 729.0kbits/frame= 1885 fps= 26 q=-1.0 size=    6656kB time=00:01:15.42 bitrate= 722.9kbits/frame= 1973 fps= 26 q=-1.0 size=    6912kB time=00:01:18.95 bitrate= 717.1kbits/frame= 2063 fps= 26 q=-1.0 size=    7424kB time=00:01:22.53 bitrate= 736.9kbits/frame= 2152 fps= 26 q=-1.0 size=    7680kB time=00:01:26.08 bitrate= 730.9kbits/No more output streams to write to, finishing.
frame= 2219 fps= 26 q=-1.0 Lsize=    8132kB time=00:01:28.78 bitrate= 750.4kbits/s speed=1.04x    
video:6983kB audio:1042kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.335714%
Input file #0 (rtmp://eu-cdn1.webofstories.com/cfx/st/):
  Input stream #0:0 (data): 2 packets read (68 bytes); 
  Input stream #0:1 (video): 2219 packets read (7150105 bytes); 
  Input stream #0:2 (audio): 3821 packets read (1067221 bytes); 
  Total: 6042 packets (8217394 bytes) demuxed
Output file #0 (file:「學而時習之.flv.part):
  Output stream #0:0 (video): 2219 packets muxed (7150105 bytes); 
  Output stream #0:1 (audio): 3821 packets muxed (1067221 bytes); 
  Total: 6040 packets (8217326 bytes) muxed
[AVIOContext @ 0x56136d2e0c80] Statistics: 2 seeks, 34 writeouts
[AVIOContext @ 0x56136d2d5480] Statistics: 8327256 bytes read, 0 seeks
[ffmpeg] Downloaded 8327086 bytes
[download] 100% of 7.94MiB in 01:28

bbepis commented 4 years ago

@remitamine Can you try this on Windows? This doesn't seem to be an issue on linux

[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.01.15
[debug] Git HEAD: c968f738d
[debug] Python version 3.8.1 (CPython) - Linux-5.4.12-arch1-1-x86_64-with-glibc2.2.5

This is why it works on linux. Linux supports UTF8 as an encoding while windows does not, only codepages

remitamine commented 4 years ago

the output I have posted shows that the initial claim that rtmpdump does not work with UTF names is incorrect, and even if it doesn't, it's an external bug that needs to be addressed in the external project(rtmpdump), not in youtube-dl, and as you can see, you have also the option to use FFmpeg an external downloader.

bbepis commented 4 years ago

The point of this ticket is to make a workaround for this issue. rtmpdump has not been maintained for 10 years and likely will never receive a fix for this, especially considering that rtmpdump is a linux-orientated tool in the first place. Nor does using ffmpeg as an extractor, as windows builds of ffmpeg do not come with librtmp integrated, meaning that it's impossible to use it for downloading RTMP streams.

There is no solution for this problem that doesn't involve changes within youtube-dl, unless you want to enforce a custom fork of rtmpdump or ffmpeg on windows users, which is laughable. Just because this works out of the box on Linux doesn't mean that my claim is wrong.

I have already made a fix in my fork which requires a change to a single line of code that checks for windows, and then strips unicode characters. If it takes a PR for this to be fixed then so be it.

dstftw commented 4 years ago

There is already a generic purpose workaround --restrict-filenames. No need in another layer of duct tape.

bbepis commented 4 years ago

It's not a clean workaround. --restrict-filenames affects the output filename as well, rather than only the temporary .part filename which is the only filename seen by rtmpdump.

Having to run a ren command/renaming the file after downloading it is also very inconvenient considering that it's youtube-dl itself that generates the output filename.

I'm not sure why this is considered duct tape but allowing streams to cut out at 99.8% due to some sites not reporting stream length correctly isn't, when technically that should be an external bug too.

remitamine commented 4 years ago

rtmpdump has not been maintained for 10 years

last commit from 2019.

I'm not sure why this is considered duct tape but allowing streams to cut out at 99.8% due to some sites not reporting stream length correctly isn't, when technically that should be an external bug too.

website owners wouldn't fix the streams they serve just to make them downloadable with youtube-dl.

and BTW flash player end of life will be at the end of this year and think 99% of the websites serving videos offer at least alternative streams that do not rely on browser plugins(Flash Player and Silver light). you're asking for a workaround for a very rare case(happens 1 website when you're downloading on windows and when the video is live and when the file name has UTF characters and rtmpdump is used), while there are few solutions that has been presented here.

bbepis commented 4 years ago

last commit from 2019.

There has not been a release since 2011, and the maintainers don't participate in the mailing list anymore. I highly doubt that they would implement a fix for it, let alone me figuring out how to contact them in the first place. https://rtmpdump.mplayerhq.hu/ChangeLog

you're asking for a workaround for a very rare case(happens 1 website when you're downloading on windows and when the video is live and when the file name has UTF characters and rtmpdump is used)

This happens on all sites that use RTMP. I can find you another site if you don't believe me.

And keep in mind that this is for the biggest video sharing/streaming site in Japan. Ignoring the fact that the extractor for regular niconico videos is broken in the first place, the reason why this is important is that niconico is a website that is technically inept (they haven't added HTTPS until very recently) and it is likely that they will be removing videos once flash player is deprecated, so it's important that old livestreams that they still host are archived.

All streams created prior to March 2019 use RTMP, while all ones after use HLS (with their own complex websocket heartbeat and manifest retrieval method).

All of the videos from the site will use UTF for titles as it's orientated for a Japanese audience. It's something that would be encountered 99% of the time.

while there are few solutions that has been presented here.

None of them are user friendly. I either have to tell people who want to download from this site to either use a specific fork (which will not be obvious as I doubt this will be put in youtube-dl documentation, and would raise a lot of issues from people using the much more popular forks), or they have to rename every old stream that they download, which is very inconvenient when this should be automated by youtube-dl itself.

And if this is to be automated, the title has to be extracted from youtube-dl run a second time without the flag to be able to find out what the original filename would be.

And if RTMP is not being used anymore by most sites, it only means that there will not be many sites at risk of breaking.

This is a very difficult workaround for something that can be easily fixed with two lines of code, that will have minimal impact (if any) to the downloader that saves a bunch of people the headache of having to create a 50 line batch script just to get around this. I really don't understand why there is so much friction for this. I might as well just tell people to use my fork where this works out of the box without needing to jump through any hoops.

ytdl-org / youtube-dl