search

animelover1984 / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
Other
290 stars 47 forks source link

NicoNico livestream timeshift websocket protocol change breaking youtube-dl #3

Closed nolanlum closed 4 years ago

nolanlum commented 4 years ago

I have reason to believe that either 1) NicoNico recently introduced a new protocol to their websocket-based HLS stream delivery, or 2) some NicoNico timeshifts were never properly supported. Downloading livestreams worked as recently as May 31, but are now broken.

The video I'm trying to download is members-only, so I get the sense I shouldn't provide it in this bug report.

The current behavior of youtube-dl is as follows:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--print-traffic', '--cookies', 'cookies.txt', 'https://live2.nicovideo.jp/watch/lv****']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.05.08
[debug] Python version 3.8.3 (CPython) - Windows-10-10.0.18362-SP0
[debug] exe versions: ffmpeg git-2020-06-01-dd76226, ffprobe git-2020-06-01-dd76226
[debug] Proxy map: {}
[NiconicoLive] lv****: Downloading webpage
[NiconicoLive] Detected post-March 2019 HLS-based stream
**** HANG ****

I suspect this is due to a change in the command and response types used for this particular websocket endpoint (URIs like wss://a.live2.nicovideo.jp/unama/wsapi/v2/watch/****/timeshift). Sniffing the websocket traffic in my browser (Firefox 77), the initial handshake looks like:

{"type":"startWatching","data":{"stream":{"quality":"high","protocol":"hls","latency":"high","chasePlay":false},"room":{"protocol":"webSocket","commentable":true},"reconnect":false}}

The server returns (among other messages):

{"type":"seat","data":{"keepIntervalSec":30}}
{"type":"stream","data":{"uri":"https://pa039327eda.dmc.nico/hlsarchive/ht2_nicolive/[REDACTED]/master.m3u8","syncUri":"https://pa039327eda.dmc.nico/hlsarchive/ht2_nicolive/[REDACTED]/stream_sync.json","quality":"high","availableQualities":["abr","super_high","high","normal","low","super_low","audio_high"],"protocol":"hls"}}

In addition to the differences in message types ("watch" vs "startWatching", "currentstream" vs "stream"), the browser sends a keepSeat request as a heartbeat, rather than a watch frame.

Looking at the niconico extractor code, the changes should be pretty simple, so I'm going to try to fix the issue and submit a pull request in the near future.

bbepis commented 4 years ago

Thanks for this. You are right in that they changed the websocket protocol for livesteams, however as far as I know, it started breaking back in mid May for streams that were currently live whilst timeshifted streams still worked.

As for some NicoNico timeshifts were never properly supported, this is partially correct. As you've probably already seen in the code, old RTMP-based streams uploaded by users require a custom build of rtmpdump that conforms to NicoNico's weirdly modified implementation of RTMP. Truth be told though, I haven't actually encountered a livestream that's still available for timeshifting that fits this criteria, so there's been zero investigation into it.

Other than those kinds of streams, I intend to support all (timeshifted) livestreams.