search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.25k stars 9.93k forks source link

Bad handling of stopped HLS stream #22870

Closed nkomarov closed 4 years ago

nkomarov commented 4 years ago

Verbose log

PASTE VERBOSE LOG HERE

youtube-dl https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8 [generic] chunks: Requesting header [generic] chunks: Downloading m3u8 information [download] Destination: chunks-chunks.mp4 ffmpeg version N-94257-g5a481b15bd Copyright (c) 2000-2019 the FFmpeg developers built with gcc 9.1.1 (GCC) 20190621 configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-li bdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpe g --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-lib wavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enab le-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libm fx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-l ibopenmpt libavutil 56. 30.100 / 56. 30.100 libavcodec 58. 53.101 / 58. 53.101 libavformat 58. 28.101 / 58. 28.101 libavdevice 58. 7.100 / 58. 7.100 libavfilter 7. 56.101 / 7. 56.101 libswscale 5. 4.101 / 5. 4.101 libswresample 3. 4.100 / 3. 4.100 libpostproc 55. 4.100 / 55. 4.100 [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [hls @ 0000000000449100] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_11226000_ 5612.ts' for reading [hls @ 0000000000449100] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_11228000_ 5613.ts' for reading Input #0, hls, from 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8': Duration: N/A, start: 11226.000000, bitrate: N/A Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1920x1080, 30 tbr, 90k tbn, 60 tbc Metadata: variant_bitrate : 0 Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp Metadata: variant_bitrate : 0 Output #0, mp4, to 'file:chunks-chunks.mp4.part': Metadata: encoder : Lavf58.28.101 Stream #0:0: Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080, q=2-31, 30 tbr, 90k tbn, 90k tbc Metadata: variant_bitrate : 0 Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp Metadata: variant_bitrate : 0 Stream mapping: Stream #0:0 -> #0:0 (copy) Stream #0:1 -> #0:1 (copy) Press [q] to stop, [?] for help [https @ 00000000032859c0] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_1123000 0_5614.ts' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000032859c0] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_1123200 0_5615.ts' for reading [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000032859c0] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_1123400 0_5616.ts' for reading [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000032859c0] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_1123600 0_5617.ts' for reading [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000032859c0] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_1131400 0_5656.ts' for reading [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000032859c0] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/l_272857_1131600 0_5657.ts' for reading

... (streaming stops here on the server side)

[https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading

...

[hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') [https @ 00000000037ce340] Opening 'https://ded7105-edge32.bcrncdn.com/hls/stream_SweetButtocks/public/stream_SweetButtocks/chunks.m3u8' for reading [hls @ 0000000000449100] Skip ('#EXT-X-VERSION:3') Error writing trailer of file:chunks-chunks.mp4.part: Immediate exit requested frame= 2806 fps= 25 q=-1.0 Lsize= 47872kB time=00:01:32.03 bitrate=4260.9kbits/s speed=0.811x video:45756kB audio:2165kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Exiting normally, received signal 2.

ERROR: Interrupted by user

(broken file)

ANOTHER URL ATTEMPT

[hls @ 0000000000489100] Skip ('#EXT-X-VERSION:3') [https @ 0000000003498000] Opening 'https://ded1163-edge30.bcrncdn.com/hls/stream_MargoEhos/public/stream_MargoEhos/l_314782_2582000_1289.ts ' for reading [https @ 0000000003333100] Opening 'https://ded1163-edge30.bcrncdn.com/hls/stream_MargoEhos/public/stream_MargoEhos/l_314782_2584000_1290.ts ' for reading

... (streaming stops here on the server side)

[https @ 0000000004446e40] Opening 'https://ded1163-edge30.bcrncdn.com/hls/stream_MargoEhos/public/stream_MargoEhos/chunks.m3u8' for reading

[hls @ 0000000000489100] Skip ('#EXT-X-VERSION:3') [https @ 0000000003333100] Opening 'https://ded1163-edge30.bcrncdn.com/hls/stream_MargoEhos/public/stream_MargoEhos/l_314782_2586000_1291.ts ' for reading [https @ 0000000004446e40] Opening 'https://ded1163-edge30.bcrncdn.com/hls/stream_MargoEhos/public/stream_MargoEhos/chunks.m3u8' for reading

[hls @ 0000000000489100] Skip ('#EXT-X-VERSION:3') [https @ 0000000004446e40] Opening 'https://ded1163-edge30.bcrncdn.com/hls/stream_MargoEhos/public/stream_MargoEhos/chunks.m3u8' for reading

[hls @ 0000000000489100] Skip ('#EXT-X-VERSION:3') [https @ 0000000004446e40] Opening 'https://ded1163-edge30.bcrncdn.com/hls/stream_MargoEhos/public/stream_MargoEhos/chunks.m3u8' for reading

(broken file)

ANOTHER URL ATTEMPT

[hls @ 0000000000109140] Skip ('#EXT-X-VERSION:3') [https @ 0000000003a0b500] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/l_1163992_162600 0_811.ts' for reading [https @ 00000000046b3ac0] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/chunks.m3u8' for reading [hls @ 0000000000109140] Skip ('#EXT-X-VERSION:3') [https @ 0000000003a0b500] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/l_1163992_162800 0_812.ts' for reading

... (streaming stops here on the server side)

[https @ 00000000046b3ac0] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/chunks.m3u8' for reading [hls @ 0000000000109140] Skip ('#EXT-X-VERSION:3') [https @ 00000000046b3ac0] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/chunks.m3u8' for reading

[https @ 00000000046b3ac0] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/chunks.m3u8' for reading [hls @ 0000000000109140] Skip ('#EXT-X-VERSION:3') [https @ 00000000046b3ac0] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/chunks.m3u8' for reading [hls @ 0000000000109140] Skip ('#EXT-X-VERSION:3') [https @ 00000000046b3ac0] Opening 'https://ded6443-edge12.bcrncdn.com/hls/stream_Goldenfleece9/public/stream_Goldenfleece9/chunks.m3u8' for reading [hls @ 0000000000109140] Skip ('#EXT-X-VERSION:3') Error writing trailer of file:chunks-chunks.mp4.part: Immediate exit requested frame= 6720 fps= 30 q=-1.0 Lsize= 60672kB time=00:03:44.00 bitrate=2218.8kbits/s speed=0.989x video:56328kB audio:4388kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Exiting normally, received signal 2.

ERROR: Interrupted by user

(result - broken file)

Description

Trying on Windows 7 from the command-line. Tried on several versions including the most recent.

There is a website bongacams.com where models stream their video. HLS stream is being downloaded successfully from command-line. Recording can be stopped upon pressing Ctrl+C. But there is a use case where streaming is stopped on the server side. In this case youtube-dl should stop recording but it shows "skipped" for eternity. Pressing Ctrl+C for the first time in this case doesn't help, the program ignores it. Pressing Ctrl+C for the second time stops the program but the trailer data won't be written into the file so it remains corrupted.

I can't give a URL where you could see an interrupted stream. Bu I think the problem is the same on any site sending and stopping HLS stream from its side, so please try to reproduce by stopping HLS stream on any of your sources.

What is expected from a fixed release: either stop recording upon detection of stopped streaming (empty chunk from server received). Or normal handling on the first pressed Ctrl+C, normal finishing file and normal exit. Like it happens when the streaming is going on.

dstftw commented 4 years ago

Report this to ffmpeg.

theultramage commented 3 years ago

I'm adding some relevant info based on testing and consulting with ffmpeg.

Youtube livestreams either don't signal end-of-stream, or ffmpeg doesn't notice/detect it. This causes it to get stuck in a loop, re-downloading the last chunk's playlist over and over without seeing any new data. In this state, ffmpeg does not respond to keyboard commands (like 'q') or sigterm (ctrl-c). Pressing ctrl-c twice will invoke the top-level signal handler and kill the ffmpeg process, not letting it finalize the mp4 file.

FFmpeg is presumably ignoring inputs because its input/signal handler is in its main loop, and because it's doing a simple synchronous blocking function call to its HLS demuxer to fetch data for the encoder. And the call will not return until it gets data or declares EOF. This structural shortcoming is still present in today's master branch, and is made worse by the abovementioned stream-end detection issue. I did find some workarounds:

  1. Using ytdl --hls-use-mpegts produces a raw file that does not need a trailing mp4 header and is not affected by this issue.
  2. Since 2019-11 ffmpeg's HLS demuxer implements the m3u8_hold_counters setting, which limits the amount of looping. Default value is 1000, which means 10-15 extra minutes of polling. So if you wait long enough, it will stop on its own. Lowering that overly generous value helps. It's unfortunate that ffmpeg doesn't print any status information to indicate that there's a loop counter.