[youtube] Intermittent `[requests] Unexpected error: EncoderStreamError: unable to encode headers`

ehoogeveen-medweb commented 2 weeks ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting that yt-dlp is broken on a supported site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region

No response

Provide a description that is worded well enough to be understood

I've been having this issue intermittently on YouTube recently, mostly with recently finished livestreams. The videos still seem to download okay despite the error (but it says to report the issue, so here we are).

Without --no-check-certificates I've seen the following warnings: WARNING: [youtube] TLS over QUIC did not succeed. Chain certificate verification failed or client cert validation failed

However I don't know for sure if those warnings are related to the error, or if they've just happened to coincide (as I normally run with --no-check-certificates enabled and only disabled it for testing). Those warnings don't seem to include any additional information.

Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', 'https://www.youtube.com/watch?v=CcBvK84kdJc']
[debug] User config "C:\Users\[username]\AppData\Roaming\yt-dlp\config.txt": ['--no-check-certificates', '--cookies-from-browser', 'firefox']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds [ac817bc83] (pip)
[debug] Python 3.12.3 (CPython AMD64 64bit) - Windows-11-10.0.22631-SP0 (OpenSSL 3.0.13 30 Jan 2024)
[debug] exe versions: ffmpeg N-115009-g2d5fa816fb-20240426 (setts), ffprobe N-115009-g2d5fa816fb-20240426, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.20.0, brotlicffi-1.1.0.0, certifi-2024.02.02, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.31.0, sqlite3-3.45.1, urllib3-2.7.906, websockets-12.0
[debug] Proxy map: {}
Extracting cookies from firefox
[debug] Extracting cookies from: "C:\Users\[username]\AppData\Roaming\Mozilla\Firefox\Profiles\132vx5df.default-nightly\cookies.sqlite"
Extracted 1051 cookies from firefox
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1810 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.04.28.232723 from yt-dlp/yt-dlp-nightly-builds)
[youtube] Extracting URL: https://www.youtube.com/watch?v=CcBvK84kdJc
[youtube] CcBvK84kdJc: Downloading webpage
[debug] [youtube] Extracted SAPISID cookie
[youtube] CcBvK84kdJc: Downloading ios player API JSON
ERROR: [requests] Unexpected error: EncoderStreamError: unable to encode headers; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
Traceback (most recent call last):
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\yt_dlp\networking\common.py", line 115, in send
    response = handler.send(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\yt_dlp\networking\_helper.py", line 208, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\yt_dlp\networking\common.py", line 335, in send
    return self._send(request)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\yt_dlp\networking\_requests.py", line 313, in _send
    requests_res = session.request(
                   ^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 1383, in urlopen
    response = self._make_request(  # type: ignore[call-overload,misc]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 1009, in _make_request
    rp = conn.request(
         ^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\connection.py", line 454, in request
    rp = self.endheaders(expect_body_afterward=chunks is not None)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\backend\hface.py", line 861, in endheaders
    self._protocol.submit_headers(
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\urllib3\contrib\hface\protocols\http3\_qh3.py", line 161, in submit_headers
    self._http.send_headers(stream_id, list(headers), end_stream)
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\qh3\h3\connection.py", line 481, in send_headers
    frame_data = self._encode_headers(stream_id, headers)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[username]\AppData\Local\Programs\Python\Python312\Lib\site-packages\qh3\h3\connection.py", line 555, in _encode_headers
    encoder, frame_data = self._encoder.encode(stream_id, headers)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_hazmat.EncoderStreamError: unable to encode headers

[youtube] CcBvK84kdJc: Downloading android player API JSON
WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "CcBvK84kdJc")
[youtube] CcBvK84kdJc: Downloading MPD manifest
[debug] [youtube] CcBvK84kdJc: Video is in Post-Live Manifestless mode
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] CcBvK84kdJc: Downloading 1 format(s): 303+140
[debug] Invoking dashsegments downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1714774426/ei/Og01ZunTFa-E6dsPk-CD-Ag/ip/2a10%3A3781%3A2c8e%3A1%3Adcce%3A9791%3A543e%3Afd32/id/CcBvK84kdJc.2/source/yt_live_broadcast/requiressl/yes/xpc/EgVo2aDSNQ%3D%3D/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/siu/1/spc/UWF9f4GVpSSLwTLrJIrP1-907dZwyA25O2071MLJ0xVOkNuNhL4OgSUN1_UtRREXQl74kDLnew/vprv/1/pacing/0/keepalive/yes/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cxpc%2Chfr%2Cas%2Cforce_finished%2Csiu%2Cspc%2Cvprv%2Citag%2Cplaylist_type/sig/AJfQdSswRAIgC6gTjHUxqrMoBZHuJkIfo3Zot12h3VwLGPRu2DPxOHcCIBk1cBSkfaniadht9DK4niWc_MKN7nhvaGVKyMUREAjr"
[debug] [youtube] [CcBvK84kdJc] Generating fragments for format 303
[dashsegments] Total fragments: 13446
[download] Destination: [PC] Warcraft III： The Frozen Throne (2003) #9 [CcBvK84kdJc].f303.webm
[debug] File locking is not supported. Proceeding without locking
[download]   0.2% of ~   4.36GiB at    3.01MiB/s ETA 25:28 (frag 26/13446)
ERROR: Interrupted by user

bashonly commented 2 weeks ago

cc @coletdjnz

ehoogeveen-medweb commented 2 weeks ago

I added a print statement for request.headers and got the following example:

{
  'Content-Type': 'application/json',
  'X-Youtube-Client-Name': '5',
  'X-Youtube-Client-Version': '19.09.3',
  'Origin': 'https://www.youtube.com',
  'User-Agent': 'com.google.ios.youtube/19.09.3 (iPhone14,3; U; CPU iOS 15_6 like Mac OS X)',
  'Authorization': 'SAPISIDHASH ***_***',
  'X-Origin': 'https://www.youtube.com',
  'X-Goog-Authuser': '0'
}

request.url was https://www.youtube.com/youtubei/v1/player?key=AIzaSyB-63vPrdThhKuerbB2N_l7Kwwcxj6yUAc&prettyPrint=false

bashonly commented 2 weeks ago

@ehoogeveen-medweb I'd suggest logging out of your current Youtube session in Firefox out of an abundance of caution. Your reply contained a potentially sensitive Authorization header. It has since been edited/redacted, but it still exists in repo subscription emails and github archive sites.

ehoogeveen-medweb commented 2 weeks ago

Ah - will do, thanks!

coletdjnz commented 2 weeks ago

You are running a modified/unofficial/unsupported version of urllib3. We only support the official urllib3 with no modifications.

Grub4K commented 2 weeks ago

Looking at the version string, this seems like urllib3.future. CC @Ousret if you want to migrate this issue to jawah/urllib3.future

ehoogeveen-medweb commented 2 weeks ago

Oh! I didn't realize at all. You're right, I had urllib3-future installed, although I don't remember installing it.

Ousret commented 2 weeks ago

Thank you for bringing this to my attention.

You were accessing YouTube using HTTP/3.

After digging into this, the issue here was a qpack encoder/decoder desync that can happen depending on your network. A fix has been written and published.

Sorry for the inconvenience.

If you would test it, I would be grateful pip install qh3 -U (v1.0.5) and even more if you could post a issue at https://github.com/jawah/urllib3.future if the issue persist.

although I don't remember installing it.

it is the core dependency of Niquests, you may have installed it directly or through another library that depend on it. while we're at it, could you say whether or not HTTP/3 improved your experience (outside of this incident..) (speed, latency, etc...)?

regards,

ehoogeveen-medweb commented 2 weeks ago

Ah, that explains it. I'll try using it again and let you know if it happens again. I've been on holiday with slow internet so I don't have a good point of reference regarding performance, sorry.

ehoogeveen-medweb commented 1 week ago

Update: It looks like the issues I was having have been fixed; I can run with urllib3.future without setting --no-check-certificates now without getting any errors or warnings.

yt-dlp / yt-dlp