SlowLogicBoy
commented
5 years ago Maybe this would help to get a stream of well live stream?
Closed Tyrrrz closed 1 year ago
SlowLogicBoy
commented
5 years ago Maybe this would help to get a stream of well live stream?
Tyrrrz
commented
5 years ago I think DASH streams work in a way that requires to refresh the manifest every second or so.
Tyrrrz
commented
5 years ago Just to clarify, we're already parsing DASH streams here. The problem is that they come in two forms:
Tyrrrz
commented
5 years ago Here's an example DASH manifest
You can see there are representations like this one:
<Representation id="140" codecs="mp4a.40.2" audioSamplingRate="44100" startWithSAP="1" bandwidth="129223">
<AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2" />
<BaseURL>https://r6---sn-bpb5oxu-3c2d.googlevideo.com/videoplayback/id/e551a6d1d7339877/itag/140/source/youtube/requiressl/yes/ei/nVTYXJPML4yuyQXqjITIDA/mm/31,29/mn/sn-bpb5oxu-3c2d,sn-ug5onuxaxjvh-n8v6/ms/au,rdu/mv/m/pl/24/initcwndbps/1148750/ratebypass/yes/mime/audio%2Fmp4/otfp/1/gir/yes/clen/1721949/lmt/1462747880588197/dur/108.297/mt/1557681202/fvip/6/beids/9466587/ip/ip_hidden/ipbits/0/expire/1557702909/sparams/ip,ipbits,expire,id,itag,source,requiressl,ei,mm,mn,ms,mv,pl,initcwndbps,ratebypass,mime,otfp,gir,clen,lmt,dur/signature/263373FCE278972F5F618F7966130F575EF337D9.30AB7F7746E0B3C6D16AB51B74502BA1358060D7/key/dg_yt0/</BaseURL>
<SegmentList>
<Initialization sourceURL="range/0-591" />
<SegmentURL media="range/888-81022" />
<SegmentURL media="range/81023-160334" />
<SegmentURL media="range/160335-239610" />
<SegmentURL media="range/239611-319249" />
<SegmentURL media="range/319250-398680" />
<SegmentURL media="range/398681-478079" />
<SegmentURL media="range/478080-557672" />
<SegmentURL media="range/557673-637068" />
<SegmentURL media="range/637069-716346" />
<SegmentURL media="range/716347-796066" />
<SegmentURL media="range/796067-875508" />
<SegmentURL media="range/875509-954737" />
<SegmentURL media="range/954738-1034467" />
<SegmentURL media="range/1034468-1113823" />
<SegmentURL media="range/1113824-1193210" />
<SegmentURL media="range/1193211-1272833" />
<SegmentURL media="range/1272834-1352180" />
<SegmentURL media="range/1352181-1431566" />
<SegmentURL media="range/1431567-1511185" />
<SegmentURL media="range/1511186-1590528" />
<SegmentURL media="range/1590529-1669515" />
<SegmentURL media="range/1669516-1721948" />
</SegmentList>
</Representation>And like this one:
<Representation id="133" codecs="avc1.4d400c" width="426" height="240" startWithSAP="1" maxPlayoutRate="1" bandwidth="242000" frameRate="30">
<BaseURL>https://r6---sn-bpb5oxu-3c2d.googlevideo.com/videoplayback/id/e551a6d1d7339877/itag/133/source/yt_otf/requiressl/yes/ei/nVTYXJPML4yuyQXqjITIDA/mm/31,29/mn/sn-bpb5oxu-3c2d,sn-ug5onuxaxjvh-n8v6/ms/au,rdu/mv/m/pl/24/initcwndbps/1148750/ratebypass/yes/mime/video%2Fmp4/otfp/1/otf/1/lmt/1462747952179492/mt/1557681202/fvip/6/beids/9466587/ip/ip_hidden/ipbits/0/expire/1557702909/sparams/ip,ipbits,expire,id,itag,source,requiressl,ei,mm,mn,ms,mv,pl,initcwndbps,ratebypass,mime,otfp,otf,lmt/signature/8DE65CD8F30555AA510337F308C36FB206E11C6D.0AF9F74E7E40B0175FD198B9CDD38073361157F5/key/dg_yt0/</BaseURL>
<SegmentList>
<Initialization sourceURL="sq/0" />
<SegmentURL media="sq/1/dur/5.067" />
<SegmentURL media="sq/2/dur/5.066" />
<SegmentURL media="sq/3/dur/5.067" />
<SegmentURL media="sq/4/dur/5.067" />
<SegmentURL media="sq/5/dur/5.066" />
<SegmentURL media="sq/6/dur/5.067" />
<SegmentURL media="sq/7/dur/5.067" />
<SegmentURL media="sq/8/dur/5.066" />
<SegmentURL media="sq/9/dur/5.067" />
<SegmentURL media="sq/10/dur/5.067" />
<SegmentURL media="sq/11/dur/5.066" />
<SegmentURL media="sq/12/dur/5.067" />
<SegmentURL media="sq/13/dur/5.067" />
<SegmentURL media="sq/14/dur/5.066" />
<SegmentURL media="sq/15/dur/5.067" />
<SegmentURL media="sq/16/dur/5.067" />
<SegmentURL media="sq/17/dur/5.066" />
<SegmentURL media="sq/18/dur/5.067" />
<SegmentURL media="sq/19/dur/5.067" />
<SegmentURL media="sq/20/dur/5.066" />
<SegmentURL media="sq/21/dur/5.067" />
<SegmentURL media="sq/22/dur/1.833" />
</SegmentList>
</Representation>Note: both URLs will not work because I removed my IP from the query string.
Tyrrrz
commented
5 years ago Tyrrrz
commented
5 years ago Ah yes. Most videos don't have DASH manifest. Some videos only sometimes have it. Some videos have it all the time.
For example, this video always has DASH manifest. It has 20 representations while playerconfig and videoinfo contain only 4.
Tyrrrz
commented
5 years ago As a side note, it looks like YouTube is killing muxed streams. For example, this video only has 1 muxed stream, both in playerconfig and videoinfo.
alexrainman
commented
3 years ago Is it possible to get live streams?
Tyrrrz
commented
3 years ago Using HLS, yes.
Tyrrrz
commented
3 years ago This is still relevant today because some videos only provide higher quality streams from DASH manifest with segmentation.
Here's an example video: https://www.youtube.com/watch?v=AI7ULzgf8RU Streams for 720p60 and 1080p60 are only available in DASH and they are segmented.
Here's the full DASH manifest for the above video:
From my basic understanding, it should be enough to use the BaseURL value and append SegmentURL/media to it. Doing so does seem to return 200 OK response of correct content type, but I wasn't able to get any actual video this way. I tried downloading all sq/ segments and merge them in a single file, but that didn't work either. So any help to understand this is appreciated.
Another challenge would be how to represent segment streams using the existing IStreamInfo interface, but that can be solved once it's clear how to download them in the first place.
Tyrrrz
commented
3 years ago @Tyrrrz can you please wrap that monster in a details block? Like this:
Thanks, done that 😄
On topic, it seems all you have to do is download the pieces and join them. Here is a working example in Go:
Interesting, I will take a look. I did that (manually) but the video ended up unplayable. It's possible I've screwed up somewhere though.
Tyrrrz
commented
3 years ago Don't remember exactly, maybe I didn't.
Tyrrrz
commented
2 years ago I noticed that YouTube seems to use DASH less and less. I wonder if there is a lot of worth in this issue, given how uncommon DASH streams are nowadays.
qtlin
commented
2 years ago I don't know if applicable to the topic, please advise. Here is the problem - I see plenty of ready to play streams:
[info] Available formats for xxxxxxxx:
format code extension resolution note
139 m4a audio only DASH audio 64k , m4a_dash container, mp4a.40.5 (22050Hz)
140 m4a audio only DASH audio 144k , m4a_dash container, mp4a.40.2 (44100Hz)
278 webm 256x144 DASH video 111k , webm_dash container, vp9, 30fps, video only
242 webm 426x240 DASH video 166k , webm_dash container, vp9, 30fps, video only
160 mp4 256x144 DASH video 212k , mp4_dash container, avc1.42c00b, 15fps, video only
243 webm 640x360 DASH video 292k , webm_dash container, vp9, 30fps, video only
133 mp4 426x240 DASH video 456k , mp4_dash container, avc1.4d4015, 30fps, video only
244 webm 854x480 DASH video 528k , webm_dash container, vp9, 30fps, video only
134 mp4 640x360 DASH video 1008k , mp4_dash container, avc1.4d401e, 30fps, video only
247 webm 1280x720 DASH video 1040k , webm_dash container, vp9, 30fps, video only
135 mp4 854x480 DASH video 1350k , mp4_dash container, avc1.4d401f, 30fps, video only
302 webm 1280x720 DASH video 1816k , webm_dash container, vp9, 60fps, video only
248 webm 1920x1080 DASH video 1816k , webm_dash container, vp9, 30fps, video only
136 mp4 1280x720 DASH video 2684k , mp4_dash container, avc1.4d401f, 30fps, video only
303 webm 1920x1080 DASH video 3016k , webm_dash container, vp9, 60fps, video only
298 mp4 1280x720 DASH video 4018k , mp4_dash container, avc1.4d4020, 60fps, video only
137 mp4 1920x1080 DASH video 5018k , mp4_dash container, avc1.640028, 30fps, video only
299 mp4 1920x1080 DASH video 6686k , mp4_dash container, avc1.64002a, 60fps, video only
91 mp4 256x144 290k , avc1.42c00b, 15.0fps, mp4a.40.5
92 mp4 426x240 546k , avc1.4d4015, 30.0fps, mp4a.40.5
93 mp4 640x360 1209k , avc1.4d401e, 30.0fps, mp4a.40.2
94 mp4 854x480 1568k , avc1.4d401f, 30.0fps, mp4a.40.2
300 mp4 1280x720 4370k , avc1.4d4020, 60.0fps, mp4a.40.2
301 mp4 1920x1080 7171k , avc1.64002a, 60.0fps, mp4a.40.2 (best)Trying DemoConsole
YoutubeExplode.DemoConsole]$ dotnet run
Enter YouTube video ID or URL: xxxxxxxx
Unhandled exception. YoutubeExplode.Exceptions.VideoUnplayableException: Video 'xxxxxxxx' does not contain any playable streams. Reason: 'This live event has ended.'.
at YoutubeExplode.Videos.Streams.StreamClient.GetManifestAsync(VideoId videoId, CancellationToken cancellationToken) in /home/eugene/git/YoutubeExplode/YoutubeExplode/Videos/Streams/StreamClient.cs:line 185
at YoutubeExplode.DemoConsole.Program.Main() in /home/eugene/git/YoutubeExplode/YoutubeExplode.DemoConsole/Program.cs:line 25
at YoutubeExplode.DemoConsole.Program.<Main>()
YoutubeExplode.DemoConsole]$ Am I missing something?
Tyrrrz
commented
2 years ago @qtlin it looks like it's unrelated because youtube-dl appears to find non-DASH streams as well. There is an issue that for some videos, sometimes (often retrying helps) we cannot retrieve any streams. I'm not sure what the reason is and because it's really inconsistent it's very difficult to debug.
qtlin
commented
2 years ago I did a download attempt using youtube-dl -f 91 -v xxxxxxxx and I see that a URL is fed to ffmpeg:
[debug] Invoking downloader on u'https://manifest.googlevideo.com/api/manifest/hls_playlist/expire/......../playlist/index.m3u8'It is not a single file as type 22 or 18.
Tyrrrz
commented
2 years ago Hmm, is the video you're trying to download a livestream?
qtlin
commented
2 years ago of course not. Did you see Reason: 'This live event has ended.'. above?. Based on my experience YouTube may eventually add formats 18 and 22 in the future though.
qtlin
commented
2 years ago As I just said above, 18 and 22 are now available:
[info] Available formats for xxxxxxxx:
format code extension resolution note
249 webm audio only tiny 50k , webm_dash container, opus @ 50k (48000Hz), 69.82MiB
250 webm audio only tiny 63k , webm_dash container, opus @ 63k (48000Hz), 87.91MiB
251 webm audio only tiny 115k , webm_dash container, opus @115k (48000Hz), 158.19MiB
140 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 178.03MiB
160 mp4 256x144 144p 57k , mp4_dash container, avc1.4d400c@ 57k, 30fps, video only, 78.43MiB
278 webm 256x144 144p 71k , webm_dash container, vp9@ 71k, 30fps, video only, 98.03MiB
242 webm 426x240 240p 104k , webm_dash container, vp9@ 104k, 30fps, video only, 143.85MiB
133 mp4 426x240 240p 130k , mp4_dash container, avc1.4d4015@ 130k, 30fps, video only, 179.89MiB
243 webm 640x360 360p 183k , webm_dash container, vp9@ 183k, 30fps, video only, 252.36MiB
134 mp4 640x360 360p 251k , mp4_dash container, avc1.4d401e@ 251k, 30fps, video only, 346.30MiB
244 webm 854x480 480p 288k , webm_dash container, vp9@ 288k, 30fps, video only, 396.53MiB
135 mp4 854x480 480p 463k , mp4_dash container, avc1.4d401f@ 463k, 30fps, video only, 636.85MiB
247 webm 1280x720 720p 566k , webm_dash container, vp9@ 566k, 30fps, video only, 779.60MiB
302 webm 1280x720 720p60 798k , webm_dash container, vp9@ 798k, 60fps, video only, 1.07GiB
136 mp4 1280x720 720p 902k , mp4_dash container, avc1.64001f@ 902k, 30fps, video only, 1.21GiB
298 mp4 1280x720 720p60 1239k , mp4_dash container, avc1.640020@1239k, 60fps, video only, 1.66GiB
303 webm 1920x1080 1080p60 1078k , webm_dash container, vp9@1078k, 60fps, video only, 1.45GiB
299 mp4 1920x1080 1080p60 2377k , mp4_dash container, avc1.64002a@2377k, 60fps, video only, 3.19GiB
18 mp4 640x360 360p 338k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz), 465.56MiB
22 mp4 1280x720 720p 1031k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)It looks like youtube-dl is getting streams from the HLS manifest and probably stitching them together. Interesting. I wonder if that's only relevant for past livestreams.
qtlin
commented
2 years ago Here is another view on the same object. youtube-dl does not show 144p but in reality it is there and even YoutubeExplode downloads it if instructed.
$ yt-dlp -F xxxxxxxx
[info] Available formats for xxxxxxxx:
ID EXT RESOLUTION FPS │ FILESIZE TBR PROTO │ VCODEC VBR ACODEC ABR ASR MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────
139 m4a audio only │ 67.08MiB 48k https │ mp4a.40.5 48k 22050Hz low, m4a_dash
249 webm audio only │ 69.82MiB 50k https │ opus 50k 48000Hz low, webm_dash
250 webm audio only │ 87.91MiB 63k https │ opus 63k 48000Hz low, webm_dash
140 m4a audio only │ 178.03MiB 129k https │ mp4a.40.2 129k 44100Hz medium, m4a_dash
251 webm audio only │ 158.19MiB 115k https │ opus 115k 48000Hz medium, webm_dash
17 3gp 176x144 8 │ 104.49MiB 75k https │ mp4v.20.3 75k mp4a.40.2 0k 22050Hz 144p
160 mp4 256x144 30 │ 78.43MiB 57k https │ avc1.4d400c 57k 144p, mp4_dash
278 webm 256x144 30 │ 98.03MiB 71k https │ vp9 71k 144p, webm_dash
133 mp4 426x240 30 │ 179.89MiB 130k https │ avc1.4d4015 130k 240p, mp4_dash
242 webm 426x240 30 │ 143.85MiB 104k https │ vp9 104k 240p, webm_dash
134 mp4 640x360 30 │ 346.30MiB 251k https │ avc1.4d401e 251k 360p, mp4_dash
18 mp4 640x360 30 │ 465.56MiB 338k https │ avc1.42001E 338k mp4a.40.2 0k 44100Hz 360p
243 webm 640x360 30 │ 252.36MiB 183k https │ vp9 183k 360p, webm_dash
135 mp4 854x480 30 │ 636.85MiB 463k https │ avc1.4d401f 463k 480p, mp4_dash
244 webm 854x480 30 │ 396.53MiB 288k https │ vp9 288k 480p, webm_dash
136 mp4 1280x720 30 │ 1.21GiB 902k https │ avc1.64001f 902k 720p, mp4_dash
22 mp4 1280x720 30 │ ~1.42GiB 1031k https │ avc1.64001F 1031k mp4a.40.2 0k 44100Hz 720p
247 webm 1280x720 30 │ 779.60MiB 566k https │ vp9 566k 720p, webm_dash
298 mp4 1280x720 60 │ 1.66GiB 1239k https │ avc1.640020 1239k 720p60, mp4_dash
302 webm 1280x720 60 │ 1.07GiB 798k https │ vp9 798k 720p60, webm_dash
299 mp4 1920x1080 60 │ 3.19GiB 2377k https │ avc1.64002a 2377k 1080p60, mp4_dash
303 webm 1920x1080 60 │ 1.45GiB 1078k https │ vp9 1078k 1080p60, webm_dashThis example is not anywhere near edge case. When a live stream ends, it takes some time for YouTube to process video and make single file formats like 17, 18 and 22 available. Only then YoutubeExplode is able to download. In some edge cases it may take a day or even more but eventually 17, 18 and 22 show up in the list.
Tyrrrz
commented
2 years ago @89z I think the challenge here is that this whole situation is only valid for a limited time after the livestream ends. So the video ID might be outdated by the time anyone looks at it.
qtlin
commented
2 years ago My comments are applicable to any completed live stream. I now understand that YoutubeExplode is not currently designed to download any format besides 17, 18 and 22. In order to download completed live stream one must wait for YouTube to produce supported formats.
Tyrrrz
commented
2 years ago I now understand that YoutubeExplode is not currently designed to download any format besides 17, 18 and 22.
That's not true. YoutubeExplode doesn't care which formats it downloads. What youtube-dl seems to be doing is using the HLS manifest to get livestream segments and stitch them together.
qtlin
commented
2 years ago To clarify, I am not requesting any fixes since I understood what follows by a live stream. If I come across any scheduled live stream in the future I'll announce it here ahead of time so all interested parties can monitor it.
qtlin
commented
2 years ago I now understand that YoutubeExplode is not currently designed to download any format besides 17, 18 and 22.
That's not true. YoutubeExplode doesn't care which formats it downloads. What youtube-dl seems to be doing is using the HLS manifest to get livestream segments and stitch them together.
youtube-dl does not parse HLS manifest. It passes HLS manifest to ffmpeg for processing.
Tyrrrz
commented
2 years ago It passes HLS manifest to ffmpeg for processing.
Which command are they using to do that?
qtlin
commented
2 years ago It passes HLS manifest to ffmpeg for processing.
Which command are they using to do that?
Have you ever tried to find any just finished live stream and invoke youtube-dl -v xxxxxxxx - how hard is it?
I glanced over YouTube and found one in almost no time. Giving this stream address is useless because it is already been processed and HLS elements removed.
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3763.0 Safari/537.36
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
' -i https://manifest.googlevideo.com/api/manifest/hls_playlist/expire/path/to/index.m3u8 -c copy -f mp4 'file:Title-@Channel-xxxxxxxx.mp4.part'I now understand that YoutubeExplode is not currently designed to download any format besides 17, 18 and 22.
That's not true. YoutubeExplode doesn't care which formats it downloads. What youtube-dl seems to be doing is using the HLS manifest to get livestream segments and stitch them together.
You probably meant to say that yt-dlp does it:
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1995
[download] 10.3% of ~274.14MiB at 496.95KiB/s ETA 03:34 (frag 205/1995)Just checking if above mentioned support is possible. Even returning a path to index.m3u8 allows caller to invoke ffmpeg and download.
Tyrrrz
commented
2 years ago Even returning a path to index.m3u8 allows caller to invoke ffmpeg and download.
This is unrelated to DASH and is already possible with StreamClient.GetHlsManifestUrlAsync()
Tyrrrz
commented
1 year ago It seems that YouTube all but phased out DASH manifests, at least on the Android client. I can't find a single video that still provides them. Closing this as it doesn't seem to be worth the effort anymore.
Related to #157. Partial streams are the first to appear and it might be useful when it comes to downloading videos as soon as they're uploaded.