Fix Picarto.tv video downloads (not stream)

jbruchon commented 2 years ago

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.02.04. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region

USA

Description

When trying to download this stream recording that is going to be auto-deleted soon I found that yt-dlp and JDownloader2 both can't fetch the stream. I found issue #1373 but it's a little old and the error is different, plus I've figured out roughly how to fix this, so I'm opening this new issue to put all the info in one place.

The actual player data is hiding behind a blob.

It loads the video info through a Websocket.

Swap wss:// to https:// and you get playlist info, BUT the playlists only contain a reference to ANOTHER playlist.

This is what's inside the HLS index.m3u8 file:

If you use the relative URL in that file, you'll get the ACTUAL m3u8 playlist for the MPEG-TS segments.

From there, it's a straightforward "download all segments and concat with ffmpeg" strategy. I'm fetching it with wget as I type this, and I've verified the segments are playable.

Verbose log

FYI: I rename to youtube-dl.exe so I don't have to change my scripts; it's actually yt-dlp latest master.

X:\>youtube-dl --throttled-rate 20k --match-filter "!is_live & !live" --cookies "Y:\cookies.txt" --add-metadata --write-description --write-info-json --write-thumbnail --write-subs --write-auto-subs --sub-lang en,en_US --convert-subs srt -ciw -o "%(title)s.%(ext)s" -v https://picarto.tv/DarkCookie/videos/616575
[debug] Command-line config: ['--throttled-rate', '20k', '--match-filter', '!is_live & !live', '--cookies', 'Y:\\cookies.txt', '--add-metadata', '--write-description', '--write-info-json', '--write-thumbnail', '--write-subs', '--write-auto-subs', '--sub-lang', 'en,en_US', '--convert-subs', 'srt', '-ciw', '-o', '%(title)s.%(ext)s', '-v', 'https://picarto.tv/DarkCookie/videos/616575']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.02.04 [c1653e9ef] (win_exe)
[debug] Lazy loading extractors is disabled
[debug] Python version 3.9.0 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg 2021-01-12-git-ca21cb1e36-full_build-www.gyan.dev, ffprobe 4.3.1-2020-11-19-full_build-www.gyan.dev
[debug] Optional libraries: Crypto, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [Picarto] Extracting URL: https://picarto.tv/DarkCookie/videos/616575
[Picarto] DarkCookie: Downloading JSON metadata
ERROR: [Picarto] DarkCookie: Stream is offline
  File "yt_dlp\extractor\common.py", line 616, in extract
  File "yt_dlp\extractor\picarto.py", line 50, in _real_extract

pukkandan commented 2 years ago

As a workaround, once u have the m3u8 url, you can just pass it to yt-dlp. No need to manually download and concat

❯ yt-dlp "https://recording-eu-1.picarto.tv/stream/hls/govod%2bgolive+DarkCookie_2022.01.13.20.05.50_nsfw.mkv/index.m3u8?video=maxbps"
[generic] index: Requesting header
[generic] index: Downloading m3u8 information
[info] index: Downloading 1 format(s): 2128
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 3684
[download] Destination: index [index].mp4
[download]   0.1% of ~978.54MiB at 50.17KiB/s ETA 06:22:47 (frag 1/3684)
ERROR: Interrupted by user

jbruchon commented 2 years ago

As a workaround, once u have the m3u8 url, you can just pass it to yt-dlp. No need to manually download and concat

Sometimes I like to see things through the hard way...just to make sure ;-)

jbruchon commented 2 years ago

Something to add: I've discovered that when a live stream is broken (stops due to streamer internet disconnection etc.), the video (recording) segments end prematurely, with the m3u8 handing out segments that don't exist on the server. If this gets added, it'll need to handle a consistently failed segment (usually in the 90th percentile completion) as the end of the video rather than an error.

yt-dlp / yt-dlp