Youtube-dl Not Detecting Some 1080p Streams

goldensun87 commented 11 years ago

The bug has occurred on both vevo and non-vevo videos. I can't name any specific videos. It is basically hit-or-miss. The program JDownloader has also had trouble detecting several 1080p streams, where the program manages to detect the 1080p stream after few tries, or fails to detect altogether.

phihag commented 11 years ago

Duplicate of #1578. YouTube seems to have introduced a bug in that they sometimes don't list all available formats.

goldensun87 commented 11 years ago

Ok, I think I found the solution. Thanks to Issue #1596 I learned how to list all available formats with youtube-dl. For the video I checked, on closer inspection, I noticed that the tag for the 1080p mp4 is now 137, NOT 37. So I tried "f137", and the 1080p mp4 stream downloaded for that video. If this is consistent, from now on everyone should list all available formats b4 attempting to download whichever resolution they want.

Edit: Never mind, the download is not playing. What does DASH video mean?

strex commented 11 years ago

no, 137 is DASH Video. See --list-formats 137 : mp4 [1080p](DASH Video) This file is not playable. You have to combine it with an dash audio file using ffmpeg.

goldensun87 commented 11 years ago

HUZZAH! Nothing MeGUI couldn't handle! >:-) So, solution found for people who know how to mux and demux their media.

seojoohyun commented 11 years ago

Well it's not optimal, but we may be stuck with it. Youtube did have both tags on the 1080p videos before, not sure if they've dropped 37 or if this is a temporary glitch. DASH video is supposed to dynamically change bit rate depending on network conditions, so if we download using tag 137 will we get video whose quality varies, as opposed to consistent 1080p quality we would get with 37?

Elite commented 11 years ago

Youtube is doing some changes which is breaking 1080p on all downloaders, lets pray this is temporary.

strex commented 11 years ago

It seems they also dropped webm 1080p (itag 46) and webm 720p (itag 45)

drewm27 commented 11 years ago

One thing I've been doing to get around this is downloading the tag 137 and tag 141 and mux-ing them back together with ffmpeg -i video.mp4 -i audio.mp4 -c copy output.mp4

strex commented 11 years ago

Yes, this is possible but annoying especially if you want to integrate this into an app. But I think there is still a solution to get the complete 1080p mp4 file (itag 37), because the youtube App on iOS on iPad is using this format when you select 1080p.

xanadu commented 11 years ago

@seojoohyun DASH is adaptive from itag 133 ~ 138, but if you download itag 137, you will always get 1080p video.

phihag commented 11 years ago

@strex How do you know that the App is indeed using format 37? Can you provide a pcap dump?

xanadu commented 11 years ago

YouTube app on my iphone still use format 18 at most time. For the content it plays, youtube-dl can also download itag 18.

Tried one content youtube-dl failed to download itag 18, YouTube app says "This content not available for mobile device".

strex commented 11 years ago

I am using the new YouTube app which was released few weeks ago. You can manually choose the quality and itag 37 is used if you choose 1080p. It is only possible on iPad. On the iPhone you can only select quality up to 720p.

@phihag I saw it in the log of my proxy. This is the url (contains itag=37): http://r7---sn-0oputhxqp5-q0ne.c.youtube.com/videoplayback?ipbits=0&ip=176.199.208.159&sparams=algorithm%2Cburst%2Cclen%2Cdur%2Cfactor%2Cgir%2Cid%2Cip%2Cipbits%2Citag%2Clmt%2Csource%2Cexpire&sver=3&mv=m&dur=209.095&source=youtube&signature=8EB345BF1B513FA234C618D39ED80171CC35509F.A0F79C87D73B12AC02280B2C3C74578AF14E5F68&itag=37&algorithm=throttle-factor&factor=1.25&fexp=906394%2C939108%2C914504%2C916613%2C936118%2C924616%2C924610%2C907231&id=6fabd27f4700f6a6&clen=98066351&cpn=wPKqO7GOIqx_JUH4&mime=video%2Fmp4&lmt=1379672322395240&ms=au&expire=1382147033&gir=yes&burst=40&mt=1382123199&dnc=1&key=yt5

I also downloaded this url using wget, and it is full 1080p with audio.

Full log here: http://pastebin.com/dzE2Bgyu

xanadu commented 11 years ago

The content mentioned by @strex doesn't have itag 37 on Desktop.

$ youtube-dl -F b6vSf0cA9qY
[youtube] Setting language
[youtube] b6vSf0cA9qY: Downloading video webpage
[youtube] b6vSf0cA9qY: Downloading video info webpage
[youtube] b6vSf0cA9qY: Extracting video information
Available formats:
22      :       mp4     [720x1280]
18      :       mp4     [360x640]
43      :       webm    [360x640]
5       :       flv     [240x400]
36      :       3gp     [240x320]
17      :       3gp     [144x176]
137     :       mp4     [1080p] (DASH Video)
136     :       mp4     [720p] (DASH Video)
135     :       mp4     [480p] (DASH Video)
134     :       mp4     [360p] (DASH Video)
133     :       mp4     [240p] (DASH Video)
160     :       mp4     [192p] (DASH Video)
141     :       mp4     [256k] (DASH Audio)
172     :       webm    [256k] (DASH Audio)
140     :       mp4     [128k] (DASH Audio)
171     :       webm    [128k] (DASH Audio)
139     :       mp4     [48k] (DASH Audio)

astral2k5 commented 11 years ago

@strex this is very interesting, is there a way to find out the user agent of this app ?

xanadu commented 11 years ago

YouTube verify more than just UA. Simply change UA won't get the job done.

Actually there is no watch page or things similar when video plays on YouTube app on IOS, and YouTube serves totally different page for YouTube in browser on IOS.

goldensun87 commented 11 years ago

I'm afraid the full 1080p mp4 stream is not being detected on ANY 1080p video anymore, old or recent. Is Youtube doing this intentionally to prevent people from DLing 1080p streams?

phihag commented 11 years ago

@goldensun87 1080p is available for our test video:

$ youtube-dl -F http://www.youtube.com/watch?v=BaW_jenozKc
[youtube] Setting language
[youtube] BaW_jenozKc: Downloading video webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
Available formats:
37      :       mp4     [1080x1920]
46      :       webm    [1080x1920]
22      :       mp4     [720x1280]
45      :       webm    [720x1280]
35      :       flv     [480x854]
44      :       webm    [480x854]
34      :       flv     [360x640]
18      :       mp4     [360x640]
43      :       webm    [360x640]
5       :       flv     [240x400]
36      :       3gp     [240x320]
17      :       3gp     [144x176]

Elite commented 11 years ago

@phihag Does that mean Youtube might be rolling back the new DASH video only update?

pmo-19 commented 11 years ago

@philhag I strongly disagree with your interpretation.

YouTube introduced for large files at higher resolution than 720p (your quoted excerpt is not large enough) the DASH format i.e. separate download of sound and video streams. For exemple I did not yet succeed to download (in Linux Debian) this video in 1080p since the change:

http://www.youtube.com/watch?v=OnoNITE-CLc

You can check in any browser that 1080p is available to viewers.

However your program (right now in Ubuntu 12.04) just goes up till 720p no more, as expected if you didn't implement the change:

$ youtube-dl -F http://www.youtube.com/watch?v=OnoNITE-CLc [youtube] Setting language [youtube] OnoNITE-CLc: Downloading video webpage [youtube] OnoNITE-CLc: Downloading video info webpage [youtube] OnoNITE-CLc: Extracting video information Available formats: 22 : mp4 [720x1280] 18 : mp4 [360x640] 43 : webm [360x640] 5 : flv [240x400] 17 : mp4 [144x176] $ $ youtube-dl -f137 http://www.youtube.com/watch?v=OnoNITE-CLc [youtube] Setting language [youtube] OnoNITE-CLc: Downloading video webpage [youtube] OnoNITE-CLc: Downloading video info webpage [youtube] OnoNITE-CLc: Extracting video information ERROR: requested format not available $ youtube-dl -f37 http://www.youtube.com/watch?v=OnoNITE-CLc [youtube] Setting language [youtube] OnoNITE-CLc: Downloading video webpage [youtube] OnoNITE-CLc: Downloading video info webpage [youtube] OnoNITE-CLc: Extracting video information ERROR: requested format not available

The 1080p file is around 85 MB.

Some other Linux available programs have already tried to implement this feature. For exemple ClipGrab. However it is still a mess to remux Audio and Video with the currently available libraries. According to its developper I would have to remove entirely the ffmpeg binary and replace it by a symlink to the libav binary version available in daily build from GitHub in Launchpad albeit only from Raring 13.04 ... FYI ClipGrab in Windows is working fine.

To me it should be better to use libav from the sources instead of using ffmpeg if the DASH streams are not yet supported in ffmpeg.

I was just looking if some other program would fit the bill ... But it looks like you even arn't even aware of the latest rules implemented by YouTube ...

phihag commented 11 years ago

@pmo-19 We are well aware of DASH, and muxing works fine with ffmpeg. Are you sure you're using a recent version of youtube-dl? Add -v to see more details, and type sudo youtube-dl -U to update. I get

$ youtube-dl -vF http://www.youtube.com/watch?v=OnoNITE-CLc 
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-vF', 'http://www.youtube.com/watch?v=OnoNITE-CLc']
[debug] youtube-dl version 2013.11.11
[debug] Git HEAD: eb0a839
[debug] Python version 2.7.5+ - Linux-3.4-trunk-amd64-x86_64-with-debian-jessie-sid
[debug] Proxy map: {}
[youtube] Setting language
[youtube] OnoNITE-CLc: Downloading video webpage
[youtube] OnoNITE-CLc: Downloading video info webpage
[youtube] OnoNITE-CLc: Extracting video information
Available formats:
22      :       mp4     [720x1280]
18      :       mp4     [360x640]
43      :       webm    [360x640]
5       :       flv     [240x400]
36      :       3gp     [240x320]
17      :       3gp     [144x176]
84      :       mp4     [720p] (3D)
82      :       mp4     [360p] (3D)
137     :       mp4     [1080p] (DASH Video)
136     :       mp4     [720p] (DASH Video)
135     :       mp4     [480p] (DASH Video)
134     :       mp4     [360p] (DASH Video)
133     :       mp4     [240p] (DASH Video)
160     :       mp4     [192p] (DASH Video)
141     :       m4a     [256k] (DASH Audio)
172     :       webm    [256k] (DASH Audio)
140     :       m4a     [128k] (DASH Audio)
171     :       webm    [128k] (DASH Audio)
139     :       m4a     [48k] (DASH Audio)

pmo-19 commented 11 years ago

@phihag OK thank you for your answer. Sorry for the delay I took answering back.

However you were right my installed version was not up to date. It looks like the tool sees now all the available formats.

But after testing the latest version I can confirm that, as I told you before, the ffmpeg library (you seem to use) is not working for muxing back the sound and video files.

$ youtube-dl -F http://www.youtube.com/watch?v=OnoNITE-CLc

Using your tool I get a Video file with no sound at all ... With ClipGrab I got a somewaht garbled Audio (the file size match the Audio file size) and no Video ... Which as I said before was fixed according to its German developper by using a dirty symlink from ffmpeg to libav binary. However from the daily github packages available in Launchpad. These packages are supplied for 13.04 Raring and up. So at this moment as I am using 12.04 Precise I cannot test it properly.

Can you confirm this behaviour ? As there has been 1 or 2 updates since my last test. If you can't confirm I will perform a remake ...

pmo-19 commented 11 years ago

@phihag I forgot to state again that I intended to download the 1080p version of the above mentioned clip.

$ youtube-dl -f137 http://www.youtube.com/watch?v=OnoNITE-CLc

xanadu commented 11 years ago

It might be something to do with your version of ffmpeg

both ffmpeg and libav should be able to do this by different parameters, and I am using ffmpeg built from source. The combined video plays fine.

$ youtube-dl -f 137 -o v.mp4 http://www.youtube.com/watch?v=OnoNITE-CLc
[youtube] Setting language
[youtube] OnoNITE-CLc: Downloading video webpage
[youtube] OnoNITE-CLc: Downloading video info webpage
[youtube] OnoNITE-CLc: Extracting video information
[download] Destination: v.mp4
[download] 100% of 63.70MiB in 00:11
$ youtube-dl -f 141 -o a.mp4 http://www.youtube.com/watch?v=OnoNITE-CLc
[youtube] Setting language
[youtube] OnoNITE-CLc: Downloading video webpage
[youtube] OnoNITE-CLc: Downloading video info webpage
[youtube] OnoNITE-CLc: Extracting video information
[download] Destination: a.mp4
[download] 100% of 7.06MiB in 00:01
$ ffmpeg -i v.mp4 -i a.mp4 -map 0:v:0 -map 1:a:0 -strict experimental av.mp4
ffmpeg version 0.8.9-4:0.8.9-0ubuntu0.12.04.1, Copyright (c) 2000-2013 the Libav developers
  built on Nov  9 2013 19:08:00 with gcc 4.6.3
*** THIS PROGRAM IS DEPRECATED ***
This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'v.mp4':
  Metadata:
    major_brand     : dash
    minor_version   : 0
    compatible_brands: iso6avc1mp41
    creation_time   : 2013-11-26 19:10:58
  Duration: 00:07:44.46, start: 232.232000, bitrate: 1150 kb/s
    Stream #0.0(und): Video: h264 (High), yuv420p, 1920x1080, 29.97 tbr, 90k tbn, 59.94 tbc
    Metadata:
      creation_time   : 2013-11-26 19:10:58
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'a.mp4':
  Metadata:
    major_brand     : dash
    minor_version   : 0
    compatible_brands: iso6mp41
    creation_time   : 2013-11-26 19:10:23
  Duration: 00:07:44.63, start: 232.315646, bitrate: 127 kb/s
    Stream #1.0(und): Audio: aac, 44100 Hz, stereo, s16
    Metadata:
      creation_time   : 2013-11-26 19:10:23
File 'av.mp4' already exists. Overwrite ? [y/N] y
[buffer @ 0xc2c620] w:1920 h:1080 pixfmt:yuv420p
Output #0, mp4, to 'av.mp4':
  Metadata:
    major_brand     : dash
    minor_version   : 0
    compatible_brands: iso6avc1mp41
    creation_time   : 2013-11-26 19:10:58
    encoder         : Lavf53.21.1
    Stream #0.0(und): Video: mpeg4, yuv420p, 1920x1080, q=2-31, 200 kb/s, 30k tbn, 29.97 tbc
    Metadata:
      creation_time   : 2013-11-26 19:10:58
    Stream #0.1(und): Audio: aac, 44100 Hz, stereo, s16, 200 kb/s
    Metadata:
      creation_time   : 2013-11-26 19:10:23
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #1.0 -> #0.1 [sync #0.0]
Press ctrl-c to stop encoding
frame= 6960 fps= 58 q=24.8 Lsize=   71095kB time=232.23 bitrate=2507.9kbits/s    
video:65890kB audio:5069kB global headers:0kB muxing overhead 0.191712%

FernandoMiguel commented 10 years ago

FYI all URLs provided here, with todays build, none show 1080p available, except for dash :(

pmo-19 commented 10 years ago

@xanadu Thank you very much for your insight. Well I took almost 3 month to answer back. My version of ffmpeg was not the culprit as I have the latest version of this library installed through the following Launchpad ppa

https://launchpad.net/~jon-severinsson/+archive/ffmpeg

In fact I thought the launch of the ffmpeg cmd. lines were part of the youtube-dl program. Then I had never had to use these type of ffmpeg cmd. Your set of cmd. worked perfectly. Thank you again.

I tested it end of December. I did not retry with the latest build ... May be in the coming days.

Here is the result for the available formats:

$ youtube-dl -F http://www.youtube.com/watch?v=OnoNITE-CLc [youtube] Setting language [youtube] OnoNITE-CLc: Downloading webpage [youtube] OnoNITE-CLc: Downloading video info webpage [youtube] OnoNITE-CLc: Extracting video information [info] Available formats for OnoNITE-CLc: format code extension resolution note 171 webm audio only DASH webm audio , audio@ 48k (worst) 140 m4a audio only DASH audio , audio@128k 160 mp4 192p DASH video 133 mp4 240p DASH video 134 mp4 360p DASH video 135 mp4 480p DASH video 136 mp4 720p DASH video 137 mp4 1080p DASH video 82 mp4 360p 3D 84 mp4 720p 3D 17 3gp 176x144 36 3gp 320x240 5 flv 400x240 43 webm 640x360 18 mp4 640x360 22 mp4 1280x720 (best)

@FernandoMiguel Do you see something different ?

FernandoMiguel commented 10 years ago

@pmo-19 exactly. google killed all >720p streams and now we only have dash one availables, which require us to use ffmpeg to mix them

on a related note, it's amazing how much processing power youtube has on their hands, to be able to delete all files they dont want (that took a big chunk of time to process previously) and make new ones for ALL their archive!

pmo-19 commented 10 years ago

@FernandoMiguel DASH or Dynamic Adaptive Streaming over HTTP is an adaptative bitrate streaming technology. The specification provides specific guidance and formats for use with two types of containers: ISO base media file format (e.g. MP4 file format) or MPEG-2 Transport Stream. See Wikipedia http://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP

You can also refer to DASH Adaptive Streaming for HTML 5 Video https://developer.mozilla.org/en-US/docs/DASH_Adaptive_Streaming_for_HTML_5_Video

This is a more efficient way to transmit multimedia over HTTP and may be the reason why it is used by Google's YouTube.

If you don't like to use ffmpeg cmd. line to re-assemble the multimedia file you can use a GUI application like ClipGrab which works (now) well in Linux, Mac OS X and Windows ... to download the parts from the URL and re-assemble it later on.

http://clipgrab.org/

I have tested it successfuly although some previous versions (end of 2013) were compatible with ffmpeg only and now the latest version will only work with libav ...

goldensun87 commented 10 years ago

I just found a Firefox Add-On called "Complete Youtube Saver". It can automatically mux the dash and audio, but only if the user directs the add-on to the folder where they have their latest version of ffmpeg.exe saved.

@phihag I don't know if the developer of Complete Youtube Saver provides the source code for his program to everyone, but if he does, then maybe you can study the programming and incorporate the auto function into your program.

FernandoMiguel commented 10 years ago

@pmo-19 after i got ffmpeg working (required 3rd party repo), it's working well, other than checking if there's an higher version and pick that one. we need a flag to do all of this automaticly and fallback graciously

pmo-19 commented 10 years ago

@xanadu I used successfuly your ffmpeg cmd. lines to mux back Audio / Video. I recently build myself from source the libav 9.12. Can you tell me which cmd. line instructions you use to mux with libav ?

juancri commented 10 years ago

@FernandoMiguel just wanted to say that actually, YouTube doesn't need that much of processing power in order to change to DASH because they can just transmux it. It would be another story if they had to transcode all the catalog. Transmuxing is a very light operation and it can be done only the first time a user watches the video at 1080p. Regards.

ytdl-org / youtube-dl

Youtube-dl Not Detecting Some 1080p Streams #1605