[twitch] VODs subtitle (chat) failed to download (Error 401: Unauthorized) when using --cookies argument

dimasahmad commented 3 years ago

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.12.14
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--cookies', 'D:\\Twitch\\twitch.tv_cookies.txt', '--all-subs', '--skip-download', 'https://www.twitch.tv/juicylucy72/videos?filter=archives&sort=time']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2020.12.14
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.3.1-2020-11-19-full_build-www.gyan.dev, ffprobe 4.3.1-2020-11-19-full_build-www.gyan.dev
[debug] Proxy map: {}
[download] Downloading playlist: juicylucy72 - Past Broadcasts sorted by Date
[TwitchVideos] juicylucy72: Downloading Videos GraphQL page 1
[TwitchVideos] playlist juicylucy72 - Past Broadcasts sorted by Date: Downloading 15 videos
[download] Downloading video 1 of 15
[twitch:vod] 842712286: Downloading video info JSON
[twitch:vod] 842712286: Downloading vod access token
[twitch:vod] 842712286: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 힐링노래방 YES chillin' singing Stream-v842712286.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 2 of 15
[twitch:vod] 840509445: Downloading video info JSON
[twitch:vod] 840509445: Downloading vod access token
[twitch:vod] 840509445: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 힐링노래방 YES chillin' singing Stream-v840509445.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 3 of 15
[twitch:vod] 839383244: Downloading video info JSON
[twitch:vod] 839383244: Downloading vod access token
[twitch:vod] 839383244: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] YES short PC Stream-v839383244.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 4 of 15
[twitch:vod] 839215211: Downloading video info JSON
[twitch:vod] 839215211: Downloading vod access token
[twitch:vod] 839215211: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] Dance practice stream-v839215211.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 5 of 15
[twitch:vod] 839182139: Downloading video info JSON
[twitch:vod] 839182139: Downloading vod access token
[twitch:vod] 839182139: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 테스트! test!-v839182139.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 6 of 15
[twitch:vod] 837028668: Downloading video info JSON
[twitch:vod] 837028668: Downloading vod access token
[twitch:vod] 837028668: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 12시까지 짧뱅! YES short Singing Stream! -v837028668.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 7 of 15
[twitch:vod] 835931201: Downloading video info JSON
[twitch:vod] 835931201: Downloading vod access token
[twitch:vod] 835931201: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 12시까지 짧뱅! YES short Singing Stream! -v835931201.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 8 of 15
[twitch:vod] 834658483: Downloading video info JSON
[twitch:vod] 834658483: Downloading vod access token
[twitch:vod] 834658483: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 오늘도 노래뱅! YES Singing Stream!-v834658483.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 9 of 15
[twitch:vod] 833629621: Downloading video info JSON
[twitch:vod] 833629621: Downloading vod access token
[twitch:vod] 833629621: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 새벽노래방-3- Singing Lucy! -v833629621.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 10 of 15
[twitch:vod] 831201980: Downloading video info JSON
[twitch:vod] 831201980: Downloading vod access token
[twitch:vod] 831201980: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 새벽노래방-3- Singing Lucy! -v831201980.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 11 of 15
[twitch:vod] 829906936: Downloading video info JSON
[twitch:vod] 829906936: Downloading vod access token
[twitch:vod] 829906936: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 오늘도 노래뱅! YES Singing Stream!-v829906936.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 12 of 15
[twitch:vod] 829850048: Downloading video info JSON
[twitch:vod] 829850048: Downloading vod access token
[twitch:vod] 829850048: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] ! Yes Stream!-v829850048.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 13 of 15
[twitch:vod] 828808045: Downloading video info JSON
[twitch:vod] 828808045: Downloading vod access token
[twitch:vod] 828808045: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] ! Yes Stream!-v828808045.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 14 of 15
[twitch:vod] 828753355: Downloading video info JSON
[twitch:vod] 828753355: Downloading vod access token
[twitch:vod] 828753355: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] ! Yes Stream!-v828753355.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Downloading video 15 of 15
[twitch:vod] 827652373: Downloading video info JSON
[twitch:vod] 827652373: Downloading vod access token
[twitch:vod] 827652373: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: [KOR _ ENG] 루씨! 늦어서 미안..!-v827652373.rechat.json
WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[download] Finished downloading playlist: juicylucy72 - Past Broadcasts sorted by Date

Description

Downloading subtitles (the chat) for Twitch video-on-demands failed when using --cookies "cookies.txt" argument with this error:

WARNING: Unable to download subtitle for "rechat": Unable to download webpage: HTTP Error 401: Unauthorized (caused by HTTPError())

It works without the --cookies argument for public VODs, but then I can't download subscribers-only VODs. Without --cookies, I would get ERROR: Failed to download m3u8 information: HTTP Error 403: Forbidden for subs-only VODs.

dimasahmad commented 3 years ago

I don't really familiar with youtube-dl code, but it seems the Twitch rechat endpoint doesn't need any cookies at all to work. So maybe, we need to change the behaviour to download the rechat file without cookies or replicate the proper authentication for the endpoint like the way Twitch web access them.

burtek commented 3 years ago

youtube-dl won't help you here, it doesn't download chat (only first 30 secs). Workaround here: https://github.com/ytdl-org/youtube-dl/issues/12266#issuecomment-465359916 but not sure if it works for sub-only VODs

dimasahmad commented 3 years ago

it doesn't download chat (only first 30 secs)

I think that's a different issue.

Youtube-dl can download the chat in .rechat.json format. You can try it yourself with clips or public VODs and look at the .description.json file (you need to use some flags to activate the description and chat file) there is rechat URL for the rechat file.

This issue is completely different from yours. We can download chat (subs) from public videos, but won't work with subscribers-only VODs.

The problem is, twitch endpoint refuses the request from youtube-dl because of the way ytdl handles the rechat URL when we using cookies.

ytdl-org / youtube-dl