Vevo 1080p & Other SMIL Streams Not Downloadable

[goldensun87]

Here is an example URL:

http://www.vevo.com/watch/owl-city/beautiful-times/USUV71400914

And, here is what comes out when I use --verbose:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['http://www.vevo.com/watch/owl-city/beautiful-times/USUV71400914', '--ve
rbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2014.09.01.1
[debug] Python version 2.7.8 - Windows-7-6.1.7601-SP1
[debug] Proxy map: {}
[Vevo] USUV71400914: Downloading JSON metadata
[Vevo] USUV71400914: Downloading SMIL info
WARNING: Cannot download SMIL information, falling back to JSON ..
[debug] Invoking downloader on 'http://h264-aws.vevo.com/v3/h264/2014/06/USUV71400914/usuv71400914_h
igh_1280x720_h264_2000_aac_128.mp4'
[download] Destination: Beautiful Times-USUV71400914.mp4

[beastbg8]

i have the same problem

[muhasturk]

I have the same problem. Some of videos are downloadable with no problem.

[mh@muhasturk indila]$ youtube-dl --proxy 62.245.223.227:80 -v http://www.vevo.com/watch/owl-city/beautiful-times/USUV71400914 -F
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--proxy', '62.245.223.227:80', '-v', 'http://www.vevo.com/watch/owl-city/beautiful-times/USUV71400914', '-F']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.09.06
[debug] Python version 3.4.0 - Linux-3.16.2-031602-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] Proxy map: {'http': '62.245.223.227:80', 'https': '62.245.223.227:80'}
[Vevo] USUV71400914: Downloading JSON metadata
[Vevo] USUV71400914: Downloading SMIL info
WARNING: Cannot download SMIL information, falling back to JSON ..
[info] Available formats for USUV71400914:
format code extension resolution  note 
Low         mp4       176x144     H264@  56k, AAC@ 24k  (worst)
Med         mp4       480x360     H264@ 500k, AAC@128k 
High        mp4       1280x720    H264@2000k, AAC@128k  (best)

[goldensun87]

Yeah, the SMIL codes seem to have changed either in the beginning of May 2014, or the end of April 2014. youtube-dl has been unable to download SMIL info of videos posted since May 2014. Videos uploaded prior to that are downloading fully.

[FabianoGK]

I created a script to download the highest quality available stream for a Vevo video. Just keep in mind that I haven't tested with too many videos and error handling is not great either...

Here it is in case anyone find it helpful--it would be nice to have it integrated on youtube-dl:

#!/bin/sh

# Video ID
video_id=ABCD12345678 # from http://www.vevo.com/watch/ABCD12345678

# Retrieve OAuth 2 access token
token=`curl --silent --data-urlencode '' http://www.vevo.com/auth | sed -n "s/^.*access_token\": \"\(.*\)\",/\1/p"`

# Retrieve list of available streams
m3u8=`curl --silent https://apiv2.vevo.com/video/$video_id/streams/hls?token=$token | sed -n "s/^.*url\": \"\(.*\)\"/\1/p" | head -2 | tail -1`
if [ "$?" -ne "0" ]; then
    echo "Failed to retrieve list of available streams"
    exit
fi
if [ -z "$m3u8" ]; then
    echo "Failed to retrieve list of available streams"
    exit
fi

# Save stream URL
stream_url=${m3u8%/*}/

# Vevo stores available formats in ascending order. Retrieve highest quality (last in the list).
format_hq=`curl --silent $m3u8 | tail -1`

# Store video list sub-folder
format_dir=${format_hq%/*}
if [ -n "$format_dir" ]; then
    format_dir=$format_dir/
fi

# Retrieve list of files for HQ format.
file_list=`curl --silent $stream_url$format_hq | grep -v "#"`

# Count number of segments, and prepare list of segments to merge.
count=0
segment_list=""
for file in $file_list
do
    segment_list="$segment_list $file"
    count=`expr $count + 1`
done

echo "Downloading $count segments..."
for file in $file_list
do
    echo -n "wget " $file
    echo -n "          "
    # For progress bar in ubuntu, awk needs `interactive` flag
    kernel=`uname -n`
    if [ "$kernel" = "ubuntu" ]; then
        wget --progress=dot $stream_url$format_dir$file 2>&1 | grep --line-buffered "%" | sed -u -e "s,\.,,g" | awk -W interactive '{printf("\b\b\b\b%4s", $2)}'
        echo -n "\b\b\b\b"
    else
        wget --progress=dot $stream_url$format_dir$file 2>&1 | grep --line-buffered "%" | sed -u -e "s,\.,,g" | awk '{printf("\b\b\b\b%4s", $2)}'
        echo -ne "\b\b\b\b"
    fi
    echo " DONE"
done

sleep 1

echo "Merging segments..."
merged=`echo $file_list | awk -F' ' '{print $1}'`
merged=`echo $merged | sed "s/_0.${merged##*.}/-merged.${merged##*.}/"`

cat $segment_list > $merged

# Remove segments
rm $segment_list

[goldensun87]

Thank you, but how do I use this?

[FabianoGK]

eited by @phihag - snip (see below)

[goldensun87]

Fabiano, you are, a genius :3 . But, for some reason, I am unable to remux into mp4 container. I could remux into mkv, but the video becomes a little glitchy when I do that. It seems the script is not able to capture all the vital metadata. In mp4box gui, the video fps shows as unknown, and the audio codec and profile info are also unknown. Perhaps this is why muxing programs are unable to mux the raw files into an mp4 container.

But again, thank you for the script. This is definitely a step in the right direction.

[phihag]

Thank you very much for your posts, @FabianoGK! In the interest in keeping on-topic, please restrict the discussion to youtube-dl. You are welcome to develop your own tools, but please move the discussion about these tool's issues to their own forum.

[phihag]

@goldensun87 Thank you for the report. youtube-dl 2014.09.24.1 and newer support the 1080p versions. Type youtube-dl -U to update.

[goldensun87]

Thank you @phihag. Upon using the new version of youtube-dl, I have found that each of the vevo videos posted after may 2014, have unique sets of format codes. Just a heads up, that users will have to use -F before attempting to download each video.

[phihag]

@goldensun87 Sorry, I don't follow and cannot reproduce this. -F does not change any state, and there is no need to run it before downloading. Just running youtube-dl http://www.vevo.com/watch/owl-city/beautiful-times/USUV71400914 is sufficient to get the highest quality.

[goldensun87]

Huh, I didn't know it was so easy to download the highest quality available. But, I will show you what I mean.

C:\Users\Naved\Desktop>youtube-dl -F http://www.vevo.com/watch/colbie-caillat/try/USUV71401303
[Vevo] Retrieving oauth token
[Vevo] USUV71401303: Downloading JSON metadata
[Vevo] USUV71401303: Downloading HLS formats
[Vevo] USUV71401303: Downloading webpage
[info] Available formats for USUV71401303:
format code extension resolution  note
m3u8-meta   mp4       multiple    Quality selection URL  (worst)
Low         mp4       176x144     H264@  56k, AAC@128k
m3u8-269    mp4       416x234      269k , avc1, mp4a
m3u8-413    mp4       416x234      413k , avc1, mp4a
m3u8-634    mp4       480x270      634k , avc1, mp4a
Med         mp4       480x360     H264@ 500k, AAC@128k
m3u8-751    mp4       640x360      751k , avc1, mp4a
m3u8-1074   mp4       640x360     1074k , avc1, mp4a
m3u8-1513   mp4       960x540     1513k , avc1, mp4a
m3u8-2810   mp4       960x540     2810k , avc1, mp4a
High        mp4       1280x720    H264@2000k, AAC@128k
m3u8-3681   mp4       1280x720    3681k , avc1, mp4a
m3u8-4810   mp4       1920x1080   4810k , avc1, mp4a
m3u8-5921   mp4       1920x1080   5921k , avc1, mp4a  (best)

C:\Users\Naved\Desktop>youtube-dl -F http://www.vevo.com/watch/shakira/la-la-la-spanish-version/USRV
81400208
[Vevo] Retrieving oauth token
[Vevo] USRV81400208: Downloading JSON metadata
[Vevo] USRV81400208: Downloading HLS formats
[Vevo] USRV81400208: Downloading webpage
[info] Available formats for USRV81400208:
format code extension resolution  note
m3u8-meta   mp4       multiple    Quality selection URL  (worst)
Low         mp4       176x144     H264@  56k, AAC@ 24k
m3u8-279    mp4       416x234      279k , avc1, mp4a
m3u8-428    mp4       416x234      428k , avc1, mp4a
m3u8-651    mp4       480x270      651k , avc1, mp4a
Med         mp4       480x360     H264@ 500k, AAC@128k
m3u8-780    mp4       640x360      780k , avc1, mp4a
m3u8-1113   mp4       640x360     1113k , avc1, mp4a
m3u8-1557   mp4       960x540     1557k , avc1, mp4a
m3u8-2894   mp4       960x540     2894k , avc1, mp4a
High        mp4       1280x720    H264@2000k, AAC@128k
m3u8-3792   mp4       1280x720    3792k , avc1, mp4a
m3u8-4860   mp4       1920x1080   4860k , avc1, mp4a
m3u8-5995   mp4       1920x1080   5995k , avc1, mp4a  (best)

See, different format codes for different videos. Anyway, I just thought it would be useful to know for anyone who still downloads their videos using the format codes.

[beastbg8]

The problem with videos from HLS streams is that they are all locked at 25 fps.

[goldensun87]

@beastbg8: After a certain date, all Vevo videos started being uploaded at the 25 fps framerate, before HLS extraction was implemented into youtube-dl. Anyway, that's why I keep a copy of the 2014.09.24.0 version of youtube-dl in my Desktop folder.

[phihag]

@goldensun87 What exactly does 2014.09.24.0 do better than the current version? We can simply add that back to the current version then. Can you give some context? I am under the impression that the best video format is correctly found by youtube-dl. Isn't that the case?

[goldensun87]

@phihag I'm using both that version and the current version. I kept that old version because, versions after 2014.09.24.0 cannot detect Vevo's old SMIL codes. More recent vevo videos do not use SMIL anymore of course, but the SMIL streams are still present for the older videos. The reason I want those old streams is because, the HLS streams contain an extra file with the audio and video, and it looks like this:

Menu ID : 4096 (0x1000) Menu ID : 1 (0x1) Duration : 4mn 12s List : 256 (0x100) (AVC) / 257 (0x101) (AAC) Service name : GBE431400013 Service provider : FFmpeg Service type : digital television

This extra file is usually 10-12 MB, and it makes the file unable to be processed in video editors like mp4box. I am able to input the video into mkvmerge, and when I do, this 'Menu' stream gets left behind. After that I am able to repack into MP4 container.

Anyway, if it's possible to add SMIL detection back in, you can do that if you want, but you don't really have to. As for the "best video format" thing, it's just my personal preference that I use the -F and -f options while downloading, nothing to worry about there xD .

[phihag]

It's fine to privately use an old version, but please do report regressions so that we can fix them. And of course, if something breaks in the old version, you're on your own.

Also bear in mind that (at least in our experience), as soon as you publicly suggest to use an old version you can bet that dozens if not hundred of users who do not comment here will stick to the old version and tell their friends to do the same, resulting in bad experiences and additional bug reports. We've seen this for -citw, which is completely insane and took years to (hopefully) rot out from most our invocations.

[phihag]

youtube-dl 2015.02.02.3 and newer correct SMIL support, so you don't need that old version any more. Please do report any issues that persist.

[goldensun87]

@phihag Thank you for the update. Heh, I had no idea that people here would stick to an old version based on one comment. I'll remember that.

[ds115]

Is SMIL broken again? Old videos still work, but I've been getting 404 - Not Found errors for newer videos.

youtube-dl.exe -F http://www.vevo.com/watch/adele/Hello/GBH481500074 -v

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-F', u'http://www.vevo.com/watch/adele/Hello/GBH48
1500074', u'-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2016.01.15
[debug] Python version 2.7.10 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-59396-g4156df5, ffprobe N-59396-g4156df5
[debug] Proxy map: {}
[Vevo] Retrieving oauth token
[Vevo] GBH481500074: Downloading JSON metadata
[Vevo] GBH481500074: Downloading HLS formats
[Vevo] GBH481500074: Downloading m3u8 information
[Vevo] GBH481500074: Downloading SMIL file
WARNING: Unable to download SMIL file: HTTP Error 404: Not Found
[info] Available formats for GBH481500074:
format code  extension  resolution note
meta         mp4        multiple   Quality selection URL
Low          mp4        176x144    H264@  56k, AAC@128k
Med          mp4        480x360    H264@ 500k, AAC@128k
High         mp4        1280x720   H264@2000k, AAC@128k
1002         mp4        640x360    1002k , avc1, mp4a
1434         mp4        960x540    1434k , avc1, mp4a
2738         mp4        960x540    2738k , avc1, mp4a
3585         mp4        1280x720   3585k , avc1, mp4a
4661         mp4        1920x1080  4661k , avc1, mp4a
5885         mp4        1920x1080  5885k , avc1, mp4a  (best)

[yan12125]

@ds115 Please open a new issue.

[nguoiaoden]

HI all ! I "I ran into a problem.

From this URL : https://www.vevo.com/watch/vic-chou/make-a-wish/TWA480406903

https://i.imgur.com/qqvYWIN.png

I can't download the video with option 1080p and m3u8. Help me !

[yan12125]

@nguoiaode: Please open a new issue and paste verbose logs as texts. No screenshots, please!