search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.27k stars 10.03k forks source link

Don't pre-apprend baseurl to each fragment in DASH/other fragmented protocols #12990

Closed wiiaboo closed 7 years ago

wiiaboo commented 7 years ago

What is the purpose of your issue?

Example URL:

As it is now, with each fragment url being pre-appended by the baseURL from the manifest, the JSON exported from this URL reaches 31.1MB. If I change Youtube-DL to not do this, the resulting JSON is only 5.45MB.

Considering baseURL is already available in the ".requested_formats[*].url" key, the API user could check if the fragment URL starts with 'http' and pre-append the baseURL itself if not, or some other new key could be added for the baseURL.

yan12125 commented 7 years ago

Related: #9109, which removes fragment URLs completely

wiiaboo commented 7 years ago

Fragment URLs are still needed for mpv until FFmpeg gets a DASH demuxer, though. It's not like the huge JSON prohibits playback, it's just a lot of repeated text for no reasonable gain.

yan12125 commented 7 years ago

9109 is a step towards live DASH streams. (#10787) As fragment URLs change as time goes by, it's not a good idea for youtube-dl to place them in extractor outputs (-j). Do you know how mpv handles live DASH streams?

wiiaboo commented 7 years ago

It doesn't, most likely. Any example URLs?

mpv doesn't continually get the JSON from Youtube-DL, it just does a single query, so it's not essential to produce fragments list at all with live content. Things like streamlink/livestreamer or youtube-dl -o - | mpv - would be required to support livestreams. There hasn't been any demand for that on mpv's side, since there's still HLS streams, which do work fine.

yan12125 commented 7 years ago

Facebook live videos use DASH. A current example is https://www.facebook.com/cs50/videos/10154276859382127/. You can find more on https://www.facebook.com/livemap. Note that you'll need to login to get video URLs on the livemap.

On Sat, May 6, 2017 at 12:34 AM, RiCON notifications@github.com wrote:

It doesn't, most likely. Any example URLs?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rg3/youtube-dl/issues/12990#issuecomment-299513172, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2RGYFsJW2OwquvE3FkOfx_pYJJzIoOks5r20-NgaJpZM4NR_ZJ .

wiiaboo commented 7 years ago

That works fine though, that's not segmented dash.

Videos from the livemap though don't work, since the urls are just dash manifests.

wiiaboo commented 7 years ago

Youtube-DL doesn't do anything with it either. It just downloads the manifest.

yan12125 commented 7 years ago

Youtube-DL doesn't do anything with it either. It just downloads the manifest.

Live DASH streams are not supported yet, so I decide to just return the DASH URL so that users can feed it to capable players. For example:

MP4Client $(youtube-dl -g url_of_the_live_video)

I mention the case here because I hope there will be a good design so that both mpv and youtube-dl will work for live DASH in the future. Personally I don't like the youtube-dl -o - | mpv - way as it's not the same smooth as the youtube-dl integration in mpv.

wiiaboo commented 7 years ago

The future will be a working/valid DASH demuxer in FFmpeg. It's probably out-of-scope for mpv.

Trying the patch in the ML it seems to work at least with Youtube's and Vimeo's non-live .mpd.

yan12125 commented 7 years ago

Once DASH support lands in ffmpeg, mpv doesn't need fragment URLs from youtube-dl anymore, right?

For now, minimizing -j output is still a good feature for static (non-live) MPDs.

wiiaboo commented 7 years ago

Since it's a huge patch with potentially a few security issues it won't be merged so soon (maybe months). mpv would be fine with live DASH not working meanwhile, as far as my opinion goes.