search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.15k stars 10.03k forks source link

Streaming to stdout with a dash format downloads a whole fragment before streaming. #31792

Closed brandon-doornbos closed 1 year ago

brandon-doornbos commented 1 year ago

Checklist

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU', '--output', '-', '--format', 'bestaudio', 'L-4IuD75pCE', '--rate-limit', '100K']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 27d41d736
[debug] Python version 3.10.9 (CPython) - Linux-6.2.2-zen1-1-zen-x86_64-with-glibc2.37
[debug] exe versions: ffmpeg 5.1.2, ffprobe 5.1.2
[debug] Proxy map: {}
It looks like you installed youtube-dl with a package manager, pip, setup.py or a tarball. Please use that to update.
[youtube] L-4IuD75pCE: Downloading webpage
[debug] [youtube] Decrypted nsig AFDraWkdzyVnrUd => Y0BxV-KMrK4frw
[debug] [youtube] Decrypted nsig uc-oaA7Os00FStD => VGnaLqKh0rZ8oA
[debug] Invoking downloader on 'https://rr2---sn-5hnednss.googlevideo.com/videoplayback?expire=1678257678&ei=rtkHZO6AC4i21gK3s4WwAw&ip=84.31.14.209&id=o-AJ7WgvkPAR8DwaEyq3OGDjTfvrciESMVUqvX8zXMAVDC&itag=140&source=youtube&requiressl=yes&mh=-I&mm=31%2C26&mn=sn-5hnednss%2Csn-4g5lznls&ms=au%2Conr&mv=m&mvi=2&pl=18&initcwndbps=2413750&vprv=1&mime=audio%2Fmp4&ns=RqgP5sJXZr5Y0ZZ-dTS2ixAL&gir=yes&clen=3973222&dur=245.458&lmt=1668326838841162&mt=1678235615&fvip=1&keepalive=yes&fexp=24007246&c=WEB&txp=4532434&n=Y0BxV-KMrK4frw&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhANYRAZjt1oK2hGttL6QhvQFIJnSozXJoCRyJu7YLf2tHAiEAtZeYH07fLgcl1UmGKaSOWruu3f4W8OwbxdDUGKFTZxc%3D&sig=AOq0QJ8wRAIgQhPWwpbpxn_9bwwl499BERmiRVXmPz455qU0IcGTw-8CIF9h3EmW-BX-miW9-s_Sc9Nj3RndqhVuIifOlfcB22mc'
[dashsegments] Total fragments: 1
[download] Destination: -
[download] 100.0% of ~3.79MiB at 100.00KiB/s ETA 00:00

Description

This happens on git master, not the latest release!

Whenever I stream a video with a dash format (e.g. YouTube 251) to stdout (--output -), youtube-dl downloads the entire first dash fragment first and then streams it instead of immediately starting to stream.

I've tested it on YouTube and Vimeo, with audio and video formats.

I use a rate limit to reduce stress while streaming, and for videos with one dash fragment, it downloads the entire video before starting to stream to stdout. This increases the latency between requesting and starting a stream dramatically. In the case of the example output (python -m youtube_dl -vU --output - --format bestaudio L-4IuD75pCE --rate-limit 100K | ffplay -loglevel quiet -) it's about 30 seconds.

Reverting https://github.com/ytdl-org/youtube-dl/commit/3da17834a49fad2a97c308fdd89aa26781ef4d60 fixes it, but that is required to fix throttling(?), so most likely not a permanent fix.

Could we just stream the fragments while downloading? After some watermark has been reached?

If I need to add information or can assist further, please tell me. I'm not familiar enough with python, dash or these kind of streams to dive in myself.

dirkf commented 1 year ago

Try --external_downloader ffmpeg, as you do have ffmpeg.

brandon-doornbos commented 1 year ago

I don't know if I'm doing something wrong, but ffmpeg doesn't even get invoked when I use it with various dash formats.

Only with a non-dash format, where it crashes.

Crash output ``` python -m youtube_dl -vU --output - --format 18 L-4IuD75pCE --external-downloader ffmpeg --limit-rate 100K | ffplay -loglevel quiet - [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU', '--output', '-', '--format', '18', 'L-4IuD75pCE', '--external-downloader', 'ffmpeg', '--limit-rate', '100K'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2021.12.17 [debug] Git HEAD: 27d41d736 [debug] Python version 3.10.9 (CPython) - Linux-6.2.2-zen1-1-zen-x86_64-with-glibc2.37 [debug] exe versions: ffmpeg 5.1.2, ffprobe 5.1.2 [debug] Proxy map: {} It looks like you installed youtube-dl with a package manager, pip, setup.py or a tarball. Please use that to update. [youtube] L-4IuD75pCE: Downloading webpage [debug] [youtube] Decrypted nsig RTymv5Nb8SkNOvfm7 => f5AxLg-DaBUuUw [debug] [youtube] Decrypted nsig 67p97dU6ww7JtfsWL => I-kz7HsyVlEWLQ [debug] Invoking downloader on 'https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678269379&ei=YwcIZOuFG7XFx_AP27-HmA8&ip=84.31.14.209&id=o-AEqh4BcXH40akTYlu0x5WQ4qaMh0fBwglkJTY-nUNPsR&itag=18&source=youtube&requiressl=yes&mh=-I&mm=31%2C26&mn=sn-5hne6nzd%2Csn-4g5lznls&ms=au%2Conr&mv=m&mvi=3&pl=18&initcwndbps=2190000&vprv=1&mime=video%2Fmp4&ns=h9ICuIRatKUUCiaWBKKUCdwL&gir=yes&clen=11778885&ratebypass=yes&dur=245.458&lmt=1668332195803883&mt=1678247379&fvip=1&fexp=24007246&c=WEB&txp=4538434&n=I-kz7HsyVlEWLQ&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cratebypass%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgeyYzApKXbMGXchGC6vbpaqiTSVUN8XQN7ESaMjG_YtQCIG0zxgGn6LFADdLSJlhSakLhQHA5zK_mT7JMSuXtGLle&sig=AOq0QJ8wRAIgV3xZ0MI91MWsYlTXtwWVzlLDZa32EBZ1uaH7qQGtVr4CIE7MYCBOn_Lttv3kdq-xdLs6RU141QNjw2ek9ilpsrYU' [download] Destination: - [debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.51 Safari/537.36 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Encoding: gzip, deflate Accept-Language: en-us,en;q=0.5 ' -i 'https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678269379&ei=YwcIZOuFG7XFx_AP27-HmA8&ip=84.31.14.209&id=o-AEqh4BcXH40akTYlu0x5WQ4qaMh0fBwglkJTY-nUNPsR&itag=18&source=youtube&requiressl=yes&mh=-I&mm=31%2C26&mn=sn-5hne6nzd%2Csn-4g5lznls&ms=au%2Conr&mv=m&mvi=3&pl=18&initcwndbps=2190000&vprv=1&mime=video%2Fmp4&ns=h9ICuIRatKUUCiaWBKKUCdwL&gir=yes&clen=11778885&ratebypass=yes&dur=245.458&lmt=1668332195803883&mt=1678247379&fvip=1&fexp=24007246&c=WEB&txp=4538434&n=I-kz7HsyVlEWLQ&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cratebypass%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgeyYzApKXbMGXchGC6vbpaqiTSVUN8XQN7ESaMjG_YtQCIG0zxgGn6LFADdLSJlhSakLhQHA5zK_mT7JMSuXtGLle&sig=AOq0QJ8wRAIgV3xZ0MI91MWsYlTXtwWVzlLDZa32EBZ1uaH7qQGtVr4CIE7MYCBOn_Lttv3kdq-xdLs6RU141QNjw2ek9ilpsrYU' -c copy -f mp4 - ffmpeg version n5.1.2 Copyright (c) 2000-2022 the FFmpeg developers built with gcc 12.2.1 (GCC) 20230201 configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan libavutil 57. 28.100 / 57. 28.100 libavcodec 59. 37.100 / 59. 37.100 libavformat 59. 27.100 / 59. 27.100 libavdevice 59. 7.100 / 59. 7.100 libavfilter 8. 44.100 / 8. 44.100 libswscale 6. 7.100 / 6. 7.100 libswresample 4. 7.100 / 4. 7.100 libpostproc 56. 6.100 / 56. 6.100 [tcp @ 0x55aef13cb600] Starting connection attempt to 74.125.100.232 port 443 [tcp @ 0x55aef13cb600] Successfully connected to 74.125.100.232 port 443 [h264 @ 0x55aef19f94c0] Reinit context to 640x272, pix_fmt: yuv420p Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678269379&ei=YwcIZOuFG7XFx_AP27-HmA8&ip=84.31.14.209&id=o-AEqh4BcXH40akTYlu0x5WQ4qaMh0fBwglkJTY-nUNPsR&itag=18&source=youtube&requiressl=yes&mh=-I&mm=31%2C26&mn=sn-5hne6nzd%2Csn-4g5lznls&ms=au%2Conr&mv=m&mvi=3&pl=18&initcwndbps=2190000&vprv=1&mime=video%2Fmp4&ns=h9ICuIRatKUUCiaWBKKUCdwL&gir=yes&clen=11778885&ratebypass=yes&dur=245.458&lmt=1668332195803883&mt=1678247379&fvip=1&fexp=24007246&c=WEB&txp=4538434&n=I-kz7HsyVlEWLQ&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cratebypass%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgeyYzApKXbMGXchGC6vbpaqiTSVUN8XQN7ESaMjG_YtQCIG0zxgGn6LFADdLSJlhSakLhQHA5zK_mT7JMSuXtGLle&sig=AOq0QJ8wRAIgV3xZ0MI91MWsYlTXtwWVzlLDZa32EBZ1uaH7qQGtVr4CIE7MYCBOn_Lttv3kdq-xdLs6RU141QNjw2ek9ilpsrYU': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: isommp42 encoder : Google Duration: 00:04:05.46, start: 0.000000, bitrate: 383 kb/s Stream #0:0[0x1](und): Video: h264 (Main), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, bt709, progressive, left), 640x268 (640x272) [SAR 1:1 DAR 160:67], 285 kb/s, 25 fps, 25 tbr, 12800 tbn (default) Metadata: handler_name : ISO Media file produced by Google Inc. vendor_id : [0][0][0][0] Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 96 kb/s (default) Metadata: handler_name : ISO Media file produced by Google Inc. vendor_id : [0][0][0][0] [mp4 @ 0x55aef19c97c0] muxer does not support non seekable output Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument Error initializing output stream 0:1 -- Stream mapping: Stream #0:0 -> #0:0 (copy) Stream #0:1 -> #0:1 (copy) Last message repeated 1 times [AVIOContext @ 0x55aef1ab9180] Statistics: 0 bytes written, 0 seeks, 0 writeouts [AVIOContext @ 0x55aef1466b00] Statistics: 98304 bytes read, 0 seeks ERROR: ffmpeg exited with code 1 File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/brandond/tmp/youtube-dl/youtube_dl/__main__.py", line 19, in youtube_dl.main() File "/home/brandond/tmp/youtube-dl/youtube_dl/__init__.py", line 475, in main _real_main(argv) File "/home/brandond/tmp/youtube-dl/youtube_dl/__init__.py", line 465, in _real_main retcode = ydl.download(all_urls) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 2080, in download res = self.extract_info( File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 809, in extract_info return self.__extract_info(url, ie, download, extra_info, process) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 816, in wrapper return func(self, *args, **kwargs) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 848, in __extract_info return self.process_ie_result(ie_result, download, extra_info) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 882, in process_ie_result return self.process_video_result(ie_result, download=download) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 1693, in process_video_result self.process_info(new_info) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 1985, in process_info success = dl(filename, info_dict) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 1924, in dl return fd.download(name, info) File "/home/brandond/tmp/youtube-dl/youtube_dl/downloader/common.py", line 366, in download return self.real_download(filename, info_dict) File "/home/brandond/tmp/youtube-dl/youtube_dl/downloader/external.py", line 64, in real_download self.report_error('%s exited with code %d' % ( File "/home/brandond/tmp/youtube-dl/youtube_dl/downloader/common.py", line 165, in report_error self.ydl.report_error(*args, **kargs) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 629, in report_error self.trouble(error_message, tb) File "/home/brandond/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 591, in trouble tb_data = traceback.format_list(traceback.extract_stack()) ```
dirkf commented 1 year ago

For me, this works with _ytdlp: wait for fragment without ffmpeg, straight to stdout with it, but ffmpeg is ignored in yt-dl.

There are two issues:

  1. tell FFmpegFD to handle http_dash_segments
  2. select a container format that can be streamed.

For (1):

--- old/youtube_dl/downloader/external.py
+++ new/youtube_dl/downloader/external.py
@@ -273,7 +273,7 @@ class HttpieFD(ExternalFD):
 class FFmpegFD(ExternalFD):
     @classmethod
     def supports(cls, info_dict):
-        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
+        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms', 'http_dash_segments')

     @classmethod
     def available(cls):

For (2), select a webm format:

$ python -m youtube_dl -v --output - --format bestaudio[ext=webm] 'L-4IuD75pCE' --external-downloader ffmpeg | mpv -
[file] Reading from stdin...
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--output', u'-', u'--format', u'bestaudio[ext=webm]', u'L-4IuD75pCE', u'--external-downloader', u'ffmpeg']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 040271022
[debug] Python version 2.7.18 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[youtube] L-4IuD75pCE: Downloading webpage
[debug] [youtube] Decrypted nsig PLUtfn6oRFl-4PVGv => 4IdEvLLJv56m2A
[debug] [youtube] Decrypted nsig o1PCOZDbiw_vDet-f => 8-xcfMCkd-uQxA
[debug] Invoking downloader on u'https://rr5---sn-cu-aigsl.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=_3YIZPuHNq-Lp-oPk5KA-AE&ip=46.208.6.20&clen=3953740&id=o-ACoYnNmTVTWXtKHp48s15Y_wvJjB6q4qxF7x6QQZ8N_C&txp=4532434&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRQIhAMTsOmAowuqHJ1msAhgejuNPmgGlXrsTJcL3bIeEciS6AiAL7OYTlUf1NjX5-Pnz0rcdB1OQCtpPEhYDjTCOMY7V9Q%3D%3D&pcm2cms=yes&dur=245.421&ns=rUz-oCAkasofphDvDrz4y4UL&initcwndbps=1318750&vprv=1&lsig=AG3C_xAwRgIhALEitcmjDS7MqFB0mIiae94K_szjR3SJoriE3pEUqrQoAiEAg6wRvExxXMZ4Jya1-jyQr2hbGCR_eulLK-Mxy2XvEeQ%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lmt=1668326808849893&c=WEB&expire=1678297951&mime=audio%2Fwebm&fvip=2&itag=251&mm=31%2C29&mn=sn-cu-aigsl%2Csn-cu-c9i6&mh=-I&n=4IdEvLLJv56m2A&mt=1678276141&mv=m&pl=25&ms=au%2Crdu&mvi=5'
[download] Destination: -
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3765.1 Safari/537.36
' -i 'https://rr5---sn-cu-aigsl.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=_3YIZPuHNq-Lp-oPk5KA-AE&ip=46.208.6.20&clen=3953740&id=o-ACoYnNmTVTWXtKHp48s15Y_wvJjB6q4qxF7x6QQZ8N_C&txp=4532434&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRQIhAMTsOmAowuqHJ1msAhgejuNPmgGlXrsTJcL3bIeEciS6AiAL7OYTlUf1NjX5-Pnz0rcdB1OQCtpPEhYDjTCOMY7V9Q%3D%3D&pcm2cms=yes&dur=245.421&ns=rUz-oCAkasofphDvDrz4y4UL&initcwndbps=1318750&vprv=1&lsig=AG3C_xAwRgIhALEitcmjDS7MqFB0mIiae94K_szjR3SJoriE3pEUqrQoAiEAg6wRvExxXMZ4Jya1-jyQr2hbGCR_eulLK-Mxy2XvEeQ%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lmt=1668326808849893&c=WEB&expire=1678297951&mime=audio%2Fwebm&fvip=2&itag=251&mm=31%2C29&mn=sn-cu-aigsl%2Csn-cu-c9i6&mh=-I&n=4IdEvLLJv56m2A&mt=1678276141&mv=m&pl=25&ms=au%2Crdu&mvi=5' -c copy -f webm -
ffmpeg version 4.3-2ubuntu0~ppa16.04+8 Copyright (c) 2000-2020 the FFmpeg developers
...
[tcp @ 0xa75f80] Starting connection attempt to 86.189.112.16 port 443
[tcp @ 0xa75f80] Successfully connected to 86.189.112.16 port 443
Input #0, matroska,webm, from 'https://rr5---sn-cu-aigsl.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=_3YIZPuHNq-Lp-oPk5KA-AE&ip=46.208.6.20&clen=3953740&id=o-ACoYnNmTVTWXtKHp48s15Y_wvJjB6q4qxF7x6QQZ8N_C&txp=4532434&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRQIhAMTsOmAowuqHJ1msAhgejuNPmgGlXrsTJcL3bIeEciS6AiAL7OYTlUf1NjX5-Pnz0rcdB1OQCtpPEhYDjTCOMY7V9Q%3D%3D&pcm2cms=yes&dur=245.421&ns=rUz-oCAkasofphDvDrz4y4UL&initcwndbps=1318750&vprv=1&lsig=AG3C_xAwRgIhALEitcmjDS7MqFB0mIiae94K_szjR3SJoriE3pEUqrQoAiEAg6wRvExxXMZ4Jya1-jyQr2hbGCR_eulLK-Mxy2XvEeQ%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lmt=1668326808849893&c=WEB&expire=1678297951&mime=audio%2Fwebm&fvip=2&itag=251&mm=31%2C29&mn=sn-cu-aigsl%2Csn-cu-c9i6&mh=-I&n=4IdEvLLJv56m2A&mt=1678276141&mv=m&pl=25&ms=au%2Crdu&mvi=5':
  Metadata:
    encoder         : google/video-file
  Duration: 00:04:05.42, start: -0.007000, bitrate: 128 kb/s
    Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default)
Output #0, webm, to 'pipe:':
  Metadata:
    encoder         : Lavf58.45.100
    Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default)
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
 (+) Audio --aid=1 --alang=eng (*) (opus 2ch 48000Hz)
AO: [pulse] 48000Hz stereo 2ch floatrate= 114.2kbits/s speed=2.34x    
A: 00:00:05 / 00:00:14 (36%) Cache: 9.3s/291KB
...
brandon-doornbos commented 1 year ago

Thanks for the replies.

I'm guessing this is a separate issue, but it is preventing me from using your solution.

When I use ffmpeg for downloading, it firstly doesn't respect youtube-dl's rate limiting, so when using it with mpv it first downloads the whole file before playing. With ffplay it does its own rate limiting, which is fine by me, however when a video is longer than about 2 minutes it crashes with this error. I assume it finished downloading the file, so youtube closes the connection, and ffmpeg sees this as an error when it hasn't finished streaming.

crash ``` $ python -m youtube_dl -vU --output - --format bestaudio[ext=webm] L-4IuD75pCE --external-downloader ./ffmpeg --limit-rate 100K | ffplay -loglevel quiet - [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU', '--output', '-', '--format', 'bestaudio[ext=webm]', 'L-4IuD75pCE', '--external-downloader', './ffmpeg', '--limit-rate', '100K'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2021.12.17 [debug] Git HEAD: 27d41d736 [debug] Python version 3.10.9 (CPython) - Linux-6.2.2-zen1-1-zen-x86_64-with-glibc2.37 [debug] exe versions: ffmpeg 5.1.2, ffprobe 5.1.2 [debug] Proxy map: {} It looks like you installed youtube-dl with a package manager, pip, setup.py or a tarball. Please use that to update. [youtube] L-4IuD75pCE: Downloading webpage [debug] [youtube] Decrypted nsig w4oD-AoEOsSK8B8tO => nAVnuJaOZrSTKg [debug] [youtube] Decrypted nsig zsOltyBRt1qWayHOz => HmAGTtsHrmJHwg [debug] Invoking downloader on 'https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678310040&ei=OKYIZKjeDY-wgAfw66WICQ&ip=84.31.14.209&id=o-AMRTiZU4iXhr-JQIVLy7oirLaj5c09tjazn-9aVs78mb&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hne6nzd%2Csn-5hnednss&ms=au%2Crdu&mv=m&mvi=3&pl=18&initcwndbps=2428750&vprv=1&mime=audio%2Fwebm&ns=rZabYS2Vte5G6s8qpJAxbuoL&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678288150&fvip=2&keepalive=yes&fexp=24007246&c=WEB&txp=4532434&n=nAVnuJaOZrSTKg&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgaG6Zm8U2HOeA3B7Ro74K2yNqGes-8Bpm5PzG9qXIQBwCIAWpcPdTjs_fbSeKs7u4g-OaBypY6wAoxsA9gbi9k75t&sig=AOq0QJ8wRAIgVG9H8WjttzZ9lHsGjVzMp0TT7pgwbegbYHm198CH2akCICu6bbA5lsqroJnwqU1G1kLa1wkNvW-kmE66qdTJIB_8' [download] Destination: - [debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.30 Safari/537.36 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Encoding: gzip, deflate Accept-Language: en-us,en;q=0.5 ' -i 'https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678310040&ei=OKYIZKjeDY-wgAfw66WICQ&ip=84.31.14.209&id=o-AMRTiZU4iXhr-JQIVLy7oirLaj5c09tjazn-9aVs78mb&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hne6nzd%2Csn-5hnednss&ms=au%2Crdu&mv=m&mvi=3&pl=18&initcwndbps=2428750&vprv=1&mime=audio%2Fwebm&ns=rZabYS2Vte5G6s8qpJAxbuoL&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678288150&fvip=2&keepalive=yes&fexp=24007246&c=WEB&txp=4532434&n=nAVnuJaOZrSTKg&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgaG6Zm8U2HOeA3B7Ro74K2yNqGes-8Bpm5PzG9qXIQBwCIAWpcPdTjs_fbSeKs7u4g-OaBypY6wAoxsA9gbi9k75t&sig=AOq0QJ8wRAIgVG9H8WjttzZ9lHsGjVzMp0TT7pgwbegbYHm198CH2akCICu6bbA5lsqroJnwqU1G1kLa1wkNvW-kmE66qdTJIB_8' -c copy -f webm - ffmpeg version n5.1.2 Copyright (c) 2000-2022 the FFmpeg developers built with gcc 12.2.1 (GCC) 20230201 configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan libavutil 57. 28.100 / 57. 28.100 libavcodec 59. 37.100 / 59. 37.100 libavformat 59. 27.100 / 59. 27.100 libavdevice 59. 7.100 / 59. 7.100 libavfilter 8. 44.100 / 8. 44.100 libswscale 6. 7.100 / 6. 7.100 libswresample 4. 7.100 / 4. 7.100 libpostproc 56. 6.100 / 56. 6.100 [tcp @ 0x55dc0f2485c0] Starting connection attempt to 74.125.100.232 port 443 [tcp @ 0x55dc0f2485c0] Successfully connected to 74.125.100.232 port 443 Input #0, matroska,webm, from 'https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678310040&ei=OKYIZKjeDY-wgAfw66WICQ&ip=84.31.14.209&id=o-AMRTiZU4iXhr-JQIVLy7oirLaj5c09tjazn-9aVs78mb&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hne6nzd%2Csn-5hnednss&ms=au%2Crdu&mv=m&mvi=3&pl=18&initcwndbps=2428750&vprv=1&mime=audio%2Fwebm&ns=rZabYS2Vte5G6s8qpJAxbuoL&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678288150&fvip=2&keepalive=yes&fexp=24007246&c=WEB&txp=4532434&n=nAVnuJaOZrSTKg&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgaG6Zm8U2HOeA3B7Ro74K2yNqGes-8Bpm5PzG9qXIQBwCIAWpcPdTjs_fbSeKs7u4g-OaBypY6wAoxsA9gbi9k75t&sig=AOq0QJ8wRAIgVG9H8WjttzZ9lHsGjVzMp0TT7pgwbegbYHm198CH2akCICu6bbA5lsqroJnwqU1G1kLa1wkNvW-kmE66qdTJIB_8': Metadata: encoder : google/video-file Duration: 00:04:05.42, start: -0.007000, bitrate: 128 kb/s Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default) Output #0, webm, to 'pipe:': Metadata: encoder : Lavf59.27.100 Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default) Stream mapping: Stream #0:0 -> #0:0 (copy) Press [q] to stop, [?] for help [tls @ 0x55dc0f247f80] Error in the pull function.ts/s speed=1.04x [tls @ 0x55dc0f247f80] IO error: Connection reset by peer [tls @ 0x55dc0f247f80] The specified session has been invalidated for some reason. Last message repeated 1 times https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678310040&ei=OKYIZKjeDY-wgAfw66WICQ&ip=84.31.14.209&id=o-AMRTiZU4iXhr-JQIVLy7oirLaj5c09tjazn-9aVs78mb&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hne6nzd%2Csn-5hnednss&ms=au%2Crdu&mv=m&mvi=3&pl=18&initcwndbps=2428750&vprv=1&mime=audio%2Fwebm&ns=rZabYS2Vte5G6s8qpJAxbuoL&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678288150&fvip=2&keepalive=yes&fexp=24007246&c=WEB&txp=4532434&n=nAVnuJaOZrSTKg&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgaG6Zm8U2HOeA3B7Ro74K2yNqGes-8Bpm5PzG9qXIQBwCIAWpcPdTjs_fbSeKs7u4g-OaBypY6wAoxsA9gbi9k75t&sig=AOq0QJ8wRAIgVG9H8WjttzZ9lHsGjVzMp0TT7pgwbegbYHm198CH2akCICu6bbA5lsqroJnwqU1G1kLa1wkNvW-kmE66qdTJIB_8: Input/output error No more output streams to write to, finishing. size= 2449kB time=00:02:32.38 bitrate= 131.7kbits/s speed=1.05x video:0kB audio:2395kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.260841% Input file #0 (https://rr3---sn-5hne6nzd.googlevideo.com/videoplayback?expire=1678310040&ei=OKYIZKjeDY-wgAfw66WICQ&ip=84.31.14.209&id=o-AMRTiZU4iXhr-JQIVLy7oirLaj5c09tjazn-9aVs78mb&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hne6nzd%2Csn-5hnednss&ms=au%2Crdu&mv=m&mvi=3&pl=18&initcwndbps=2428750&vprv=1&mime=audio%2Fwebm&ns=rZabYS2Vte5G6s8qpJAxbuoL&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678288150&fvip=2&keepalive=yes&fexp=24007246&c=WEB&txp=4532434&n=nAVnuJaOZrSTKg&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgaG6Zm8U2HOeA3B7Ro74K2yNqGes-8Bpm5PzG9qXIQBwCIAWpcPdTjs_fbSeKs7u4g-OaBypY6wAoxsA9gbi9k75t&sig=AOq0QJ8wRAIgVG9H8WjttzZ9lHsGjVzMp0TT7pgwbegbYHm198CH2akCICu6bbA5lsqroJnwqU1G1kLa1wkNvW-kmE66qdTJIB_8): Input stream #0:0 (audio): 7620 packets read (2452273 bytes); Total: 7620 packets (2452273 bytes) demuxed Output file #0 (pipe:): Output stream #0:0 (audio): 7620 packets muxed (2452273 bytes); Total: 7620 packets (2452273 bytes) muxed [AVIOContext @ 0x55dc0f842fc0] Statistics: 2507715 bytes written, 0 seeks, 154 writeouts [AVIOContext @ 0x55dc0f2e3b40] Statistics: 2506736 bytes read, 0 seeks [download] 100% in 02:25 ```
dirkf commented 1 year ago

When streaming to mpv as in my example, it's only downloading fast enough to fill the various buffers in ffmpeg and mpv while playing the stream, so there shouldn't be much point in rate-limiting. But this doesn't happen without the change (1). A pretty plangent song comes out here, with my modified FFmpegFD.

$ python -m youtube_dl -v --output - --format bestaudio[ext=webm] 'L-4IuD75pCE' --external-downloader ffmpeg | ffplay -loglevel quiet -
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--output', u'-', u'--format', u'bestaudio[ext=webm]', u'L-4IuD75pCE', u'--external-downloader', u'ffmpeg']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 040271022
[debug] Python version 2.7.18 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[youtube] L-4IuD75pCE: Downloading webpage
[debug] [youtube] Decrypted nsig zc3oQz4-UcF7psmV3 => OiYN3mfylZ93ZA
[debug] [youtube] Decrypted nsig S0cQ_Zbf6DfYxdAUR => qaWSRALgvw4P-A
[debug] Invoking downloader on u'https://rr5---sn-cu-aigsl.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=-xQJZNGJJ87PxwLklYDwAg&ip=46.208.6.20&clen=3953740&id=o-APbNL3mhSwjI8erJqqNI4Bl8PIvcmBXFEujSBC7eGQMK&txp=4532434&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRAIgXwpMEtgAQFxNpRutFs37sl_mux69Kj-YJDqoqMQnWwICIHsI4VgAjhJk7EAYzTnzl99P5Lyu-7e-3MwH77Bh5QBi&dur=245.421&ns=OUeD5RQKBhUT9pwJhACNJtAL&initcwndbps=1608750&vprv=1&lsig=AG3C_xAwRQIgR_gmS9-lKDc4L3g8DCUEyNKCZSkAf_4dcP9uxuQsHX4CIQDspCg7phY-VIuDCas19dKMDtzfrlJDIRo_-qeTfFGBVA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lmt=1668326808849893&c=WEB&expire=1678338395&mime=audio%2Fwebm&fvip=2&itag=251&mm=31%2C29&mn=sn-cu-aigsl%2Csn-cu-c9i6&mh=-I&n=OiYN3mfylZ93ZA&mt=1678316229&mv=m&pl=25&ms=au%2Crdu&mvi=5'
[download] Destination: -
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.2 Safari/537.36
' -i 'https://rr5---sn-cu-aigsl.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=-xQJZNGJJ87PxwLklYDwAg&ip=46.208.6.20&clen=3953740&id=o-APbNL3mhSwjI8erJqqNI4Bl8PIvcmBXFEujSBC7eGQMK&txp=4532434&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRAIgXwpMEtgAQFxNpRutFs37sl_mux69Kj-YJDqoqMQnWwICIHsI4VgAjhJk7EAYzTnzl99P5Lyu-7e-3MwH77Bh5QBi&dur=245.421&ns=OUeD5RQKBhUT9pwJhACNJtAL&initcwndbps=1608750&vprv=1&lsig=AG3C_xAwRQIgR_gmS9-lKDc4L3g8DCUEyNKCZSkAf_4dcP9uxuQsHX4CIQDspCg7phY-VIuDCas19dKMDtzfrlJDIRo_-qeTfFGBVA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lmt=1668326808849893&c=WEB&expire=1678338395&mime=audio%2Fwebm&fvip=2&itag=251&mm=31%2C29&mn=sn-cu-aigsl%2Csn-cu-c9i6&mh=-I&n=OiYN3mfylZ93ZA&mt=1678316229&mv=m&pl=25&ms=au%2Crdu&mvi=5' -c copy -f webm -
ffmpeg version 4.3-2ubuntu0~ppa16.04+8 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.12) 20160609
  configuration: ...
[tcp @ 0x1ed4080] Starting connection attempt to 86.189.112.16 port 443
[tcp @ 0x1ed4080] Successfully connected to 86.189.112.16 port 443
Input #0, matroska,webm, from 'https://rr5---sn-cu-aigsl.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=-xQJZNGJJ87PxwLklYDwAg&ip=46.208.6.20&clen=3953740&id=o-APbNL3mhSwjI8erJqqNI4Bl8PIvcmBXFEujSBC7eGQMK&txp=4532434&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRAIgXwpMEtgAQFxNpRutFs37sl_mux69Kj-YJDqoqMQnWwICIHsI4VgAjhJk7EAYzTnzl99P5Lyu-7e-3MwH77Bh5QBi&dur=245.421&ns=OUeD5RQKBhUT9pwJhACNJtAL&initcwndbps=1608750&vprv=1&lsig=AG3C_xAwRQIgR_gmS9-lKDc4L3g8DCUEyNKCZSkAf_4dcP9uxuQsHX4CIQDspCg7phY-VIuDCas19dKMDtzfrlJDIRo_-qeTfFGBVA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lmt=1668326808849893&c=WEB&expire=1678338395&mime=audio%2Fwebm&fvip=2&itag=251&mm=31%2C29&mn=sn-cu-aigsl%2Csn-cu-c9i6&mh=-I&n=OiYN3mfylZ93ZA&mt=1678316229&mv=m&pl=25&ms=au%2Crdu&mvi=5':
  Metadata:
    encoder         : google/video-file
  Duration: 00:04:05.42, start: -0.007000, bitrate: 128 kb/s
    Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default)
Output #0, webm, to 'pipe:':
  Metadata:
    encoder         : Lavf58.45.100
    Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default)
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
size=     833kB time=00:00:53.02 bitrate= 128.7kbits/s speed= 1.1x
brandon-doornbos commented 1 year ago

I should explain myself better, the rate-limiting is to reduce network usage. But that is besides the point.

The song does play, but after 2 minutes or so, the connection gets cut, ffmpeg closes the pipe and the stream stops. So this solution does not work for me, neither in youtube-dl nor yt-dlp.

crash ``` yt-dlp -v --output - --format bestaudio[ext=webm] 'L-4IuD75pCE' --external-downloader ffmpeg | ffplay -loglevel quiet - [debug] Command-line config: ['-v', '--output', '-', '--format', 'bestaudio[ext=webm]', 'L-4IuD75pCE', '--external-downloader', 'ffmpeg'] [debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version stable@2023.03.04 [392389b7d] [debug] Python 3.10.9 (CPython x86_64 64bit) - Linux-6.2.2-zen1-1-zen-x86_64-with-glibc2.37 (OpenSSL 3.0.8 7 Feb 2023, glibc 2.37) [debug] exe versions: ffmpeg 6.0 (setts), ffprobe 6.0 [debug] Optional libraries: certifi-2022.12.07, no_Cryptodome-None, sqlite3-2.6.0 [debug] Proxy map: {} [debug] Loaded 1786 extractors [youtube] Extracting URL: L-4IuD75pCE [youtube] L-4IuD75pCE: Downloading webpage [youtube] L-4IuD75pCE: Downloading android player API JSON [debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto [debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id [info] L-4IuD75pCE: Downloading 1 format(s): 251 [debug] Invoking ffmpeg downloader on "https://rr2---sn-5hnednss.googlevideo.com/videoplayback?expire=1678495323&ei=-3kLZN3yK42J6dsPyOGQyA8&ip=84.31.14.209&id=o-AAbioqdNo0U7BDExnsfpRbWzAc7t0KC493IDyi7wP76E&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hnednss%2Csn-5hne6nzd&ms=au%2Crdu&mv=m&mvi=2&pl=18&initcwndbps=2305000&spc=H3gIhoMFXCB1rCBjpaZSuSvXVIU6nh4&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678473449&fvip=3&keepalive=yes&fexp=24007246&c=ANDROID&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgKkhWqWIDZZHudP-ZdE3Nk1u4165FR4D_dmOTFWH4nCUCIQCVVN-fWkNfyrka7FTr4eL2-jC9uduFYFD4lkI1LPQ9fw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAL8e7KemhqG-7PSgZogiGMU_oEWPFKN4GmTogXBAPhtRAiEAxFlbBwzS8jKV3CwqLjZII2z1WJCHSL8GEh_xnoZTpy8%3D" [download] Destination: - [debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Sec-Fetch-Mode: navigate ' -i 'https://rr2---sn-5hnednss.googlevideo.com/videoplayback?expire=1678495323&ei=-3kLZN3yK42J6dsPyOGQyA8&ip=84.31.14.209&id=o-AAbioqdNo0U7BDExnsfpRbWzAc7t0KC493IDyi7wP76E&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hnednss%2Csn-5hne6nzd&ms=au%2Crdu&mv=m&mvi=2&pl=18&initcwndbps=2305000&spc=H3gIhoMFXCB1rCBjpaZSuSvXVIU6nh4&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678473449&fvip=3&keepalive=yes&fexp=24007246&c=ANDROID&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgKkhWqWIDZZHudP-ZdE3Nk1u4165FR4D_dmOTFWH4nCUCIQCVVN-fWkNfyrka7FTr4eL2-jC9uduFYFD4lkI1LPQ9fw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAL8e7KemhqG-7PSgZogiGMU_oEWPFKN4GmTogXBAPhtRAiEAxFlbBwzS8jKV3CwqLjZII2z1WJCHSL8GEh_xnoZTpy8%3D' -c copy -map 0:0 -f webm - ffmpeg version n6.0 Copyright (c) 2000-2023 the FFmpeg developers built with gcc 12.2.1 (GCC) 20230201 configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 [tcp @ 0x55f64fcb4340] Starting connection attempt to 172.217.132.199 port 443 [tcp @ 0x55f64fcb4340] Successfully connected to 172.217.132.199 port 443 Input #0, matroska,webm, from 'https://rr2---sn-5hnednss.googlevideo.com/videoplayback?expire=1678495323&ei=-3kLZN3yK42J6dsPyOGQyA8&ip=84.31.14.209&id=o-AAbioqdNo0U7BDExnsfpRbWzAc7t0KC493IDyi7wP76E&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hnednss%2Csn-5hne6nzd&ms=au%2Crdu&mv=m&mvi=2&pl=18&initcwndbps=2305000&spc=H3gIhoMFXCB1rCBjpaZSuSvXVIU6nh4&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678473449&fvip=3&keepalive=yes&fexp=24007246&c=ANDROID&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgKkhWqWIDZZHudP-ZdE3Nk1u4165FR4D_dmOTFWH4nCUCIQCVVN-fWkNfyrka7FTr4eL2-jC9uduFYFD4lkI1LPQ9fw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAL8e7KemhqG-7PSgZogiGMU_oEWPFKN4GmTogXBAPhtRAiEAxFlbBwzS8jKV3CwqLjZII2z1WJCHSL8GEh_xnoZTpy8%3D': Metadata: encoder : google/video-file Duration: 00:04:05.42, start: -0.007000, bitrate: 128 kb/s Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default) Output #0, webm, to 'pipe:': Metadata: encoder : Lavf60.3.100 Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default) Stream mapping: Stream #0:0 -> #0:0 (copy) Press [q] to stop, [?] for help [tls @ 0x55f64fcb3cc0] Error in the pull function.ts/s speed=1.04x [tls @ 0x55f64fcb3cc0] IO error: Connection reset by peer [tls @ 0x55f64fcb3cc0] The specified session has been invalidated for some reason. Last message repeated 1 times Error demuxing input file 0: Input/output error Terminating demuxer thread 0 https://rr2---sn-5hnednss.googlevideo.com/videoplayback?expire=1678495323&ei=-3kLZN3yK42J6dsPyOGQyA8&ip=84.31.14.209&id=o-AAbioqdNo0U7BDExnsfpRbWzAc7t0KC493IDyi7wP76E&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hnednss%2Csn-5hne6nzd&ms=au%2Crdu&mv=m&mvi=2&pl=18&initcwndbps=2305000&spc=H3gIhoMFXCB1rCBjpaZSuSvXVIU6nh4&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678473449&fvip=3&keepalive=yes&fexp=24007246&c=ANDROID&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgKkhWqWIDZZHudP-ZdE3Nk1u4165FR4D_dmOTFWH4nCUCIQCVVN-fWkNfyrka7FTr4eL2-jC9uduFYFD4lkI1LPQ9fw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAL8e7KemhqG-7PSgZogiGMU_oEWPFKN4GmTogXBAPhtRAiEAxFlbBwzS8jKV3CwqLjZII2z1WJCHSL8GEh_xnoZTpy8%3D: Input/output error No more output streams to write to, finishing. [out#0/webm @ 0x55f6502e0840] All streams finished [out#0/webm @ 0x55f6502e0840] Terminating muxer thread [AVIOContext @ 0x55f6502af000] Statistics: 2688163 bytes written, 0 seeks, 165 writeouts size= 2625kB time=00:02:43.54 bitrate= 131.5kbits/s speed=1.04x video:0kB audio:2567kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.262302% Input file #0 (https://rr2---sn-5hnednss.googlevideo.com/videoplayback?expire=1678495323&ei=-3kLZN3yK42J6dsPyOGQyA8&ip=84.31.14.209&id=o-AAbioqdNo0U7BDExnsfpRbWzAc7t0KC493IDyi7wP76E&itag=251&source=youtube&requiressl=yes&mh=-I&mm=31%2C29&mn=sn-5hnednss%2Csn-5hne6nzd&ms=au%2Crdu&mv=m&mvi=2&pl=18&initcwndbps=2305000&spc=H3gIhoMFXCB1rCBjpaZSuSvXVIU6nh4&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=3953740&dur=245.421&lmt=1668326808849893&mt=1678473449&fvip=3&keepalive=yes&fexp=24007246&c=ANDROID&txp=4532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgKkhWqWIDZZHudP-ZdE3Nk1u4165FR4D_dmOTFWH4nCUCIQCVVN-fWkNfyrka7FTr4eL2-jC9uduFYFD4lkI1LPQ9fw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAL8e7KemhqG-7PSgZogiGMU_oEWPFKN4GmTogXBAPhtRAiEAxFlbBwzS8jKV3CwqLjZII2z1WJCHSL8GEh_xnoZTpy8%3D): Input stream #0:0 (audio): 8178 packets read (2628694 bytes); Total: 8178 packets (2628694 bytes) demuxed Output file #0 (pipe:): Output stream #0:0 (audio): 8178 packets muxed (2628694 bytes); Total: 8178 packets (2628694 bytes) muxed [AVIOContext @ 0x55f64fd4f840] Statistics: 2686960 bytes read, 0 seeks [download] 100% in 00:02:37 ```

This would be solved if the native downloader streamed DASH fragments to stdout while downloading, or fixing ffmpeg somehow to not crash while streaming.

Should I file a bug report with ffmpeg, or continue to work with you here to solve it in youtube-dl? I assume downloading whole fragments before streaming is not intended behavior.

dirkf commented 1 year ago

... the rate-limiting is to reduce network usage.

It's pointless if you're streaming to a player because the network usage will be as much as needed to play the media and little more. In my log you can see that the song is downloading at ~1/6 of your limit.

Should I ... continue to work with you here to solve it in youtube-dl?

I assume you've applied the patch (1), or ffmpeg wouldn't be running.

This seems to imply some network configuration problem

[tls @ 0x55f64fcb3cc0] Error in the pull function.ts/s speed=1.04x
[tls @ 0x55f64fcb3cc0] IO error: Connection reset by peer
[tls @ 0x55f64fcb3cc0] The specified session has been invalidated for some reason.

G/SO indicates that one or other of these may help:

Or try an old ffmpeg, if you have one. I'm not seeing the same problems, playing through 2.5 minutes with ffplay.

brandon-doornbos commented 1 year ago

It's pointless if you're streaming to a player because the network usage will be as much as needed ...

Ah fair enough.

G/SO indicates that one or other of these may help ...

I appreciate the effort, I tried all of these before and none worked. However, just -reconnect 1 did solve it for me which is not documented by ffmpeg for some reason.

My problem is solved now, however I still think the native downloader should stream the fragments while downloading, but I guess that is for you to decide.

Thank you for all the help. Should I close this since it's solved, or keep it open since it is technically still a bug?

dirkf commented 1 year ago

As it's a bug/enhancement resulting from https://github.com/ytdl-org/youtube-dl/commit/3da17834a49fad2a97c308fdd89aa26781ef4d60, I'll close it when the 1-line patch is committed, thanks.