youtube-dl dumps wrong URL for YouTube videos

[hdclark]

Description

youtube-dl will properly download YouTube videos, but the [-g] option spits out URLs to different videos. Wget, cURL, and mplayer download the wrong video. Behaviour does not vary when changing user-agent in any four (or any combination of the four) programs.

Most youtube links will result in the same video being downloaded (https://www.youtube.com/watch?v=5mPDhmP6ztQ). Behaviour happens across several machines (32 and 64 bit) at different addresses.

Minimal test program

!/bin/sh

BAREURL="$( youtube-dl -g '$1')" wget "$BAREURL" exit

Machine and Version info

youtube-dl version: 2014.02.25 OS: Arch Linux uname: Linux XXX 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:46:30 CET 2014 i686 GNU/Linux glibc version: glibc Python version: 3.3.3 Country: Canada

[jaimeMF]

How are you calling the script?

[phihag]

The script works fine for me from the US.

Can you define wrong? Is it another video? Is it random data?

Does the problem occur when you use youtube-dl's built-in downloader, i.e. do not pass in -g? What output do you get if you pass in the -v option instead of -g?

Can you reproduce the problem with http://www.youtube.com/watch?v=BaW_jenozKc ?

[jaimeMF]

I have added set -x to the start of the script and this is the output

./script.sh
++ youtube-dl -g '$1'
[debug] System config: []
[debug] User config: ['--keep-video', '--write-info-json', '--write-thumbnail', '--verbose']
[debug] Command-line args: ['-g', '$1']
[debug] Encodings: locale 'UTF-8', fs 'utf-8', out None, pref: 'UTF-8'
[debug] youtube-dl version 2014.02.28
[debug] Python version 2.7.6 - Darwin-13.1.0-x86_64-i386-64bit
[debug] Proxy map: {}
+ BAREURL='blah'
+ wget 'blah'

it downloads https://www.youtube.com/watch?v=J0-rgeU1E4w

My /bin/sh version is:

/bin/sh --version
GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)
Copyright (C) 2007 Free Software Foundation, Inc.

This script works fine:

#!/bin/sh
BAREURL="$( youtube-dl -g "$1")"
wget "$BAREURL"
exit

@phihag Maybe we should disable the automic search by default and let it fail.

[hdclark]

Calling the aforementioned script 'test.sh' and performing

./test.sh 'https://www.youtube.com/watch?v=yrBrH5PZtcc'

will result in downloading the YouTube video https://www.youtube.com/watch?v=5mPDhmP6ztQ. The URL youtube-dl spits out is:

http://r2---sn-j5o5gpx-t8ge.googlevideo.com/videoplayback?ipbits=0&mv=m&source=youtube&gcr=ca&sparams=gcr%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&ip=WWW.XXX.YYY.ZZZ&expire=1393891572&fexp=935627%2C909708%2C944900%2C904316%2C916624%2C902529%2C937417%2C937416%2C913434%2C936910%2C936913%2C902907&ms=au&mt=1393868125&ratebypass=yes&itag=22&sver=3&id=274fab81e535138c&upn=H0rTMY6w6HQ&key=yt5&signature=F4AF73DC612B7DFAB91AC90ECDDEFDFFCFB50325.B479CBFC47AEE9CF0862A1B1A9897EEE2B3A3ACC

The same behaviour happens on the test file at http://www.youtube.com/watch?v=BaW_jenozKc : youtube-dl spits out a googlevideo URL that points to another video (the same damn "TrapDoor" video at https://www.youtube.com/watch?v=5mPDhmP6ztQ).

In comparison, performing

youtube-dl 'https://www.youtube.com/watch?v=yrBrH5PZtcc'

will download the correct video (https://www.youtube.com/watch?v=yrBrH5PZtcc - "Opeth- Demon of the Fall-yrBrH5PZtcc.mp4").

Motivated by an earlier youtube-dl issue, I've tried removing ~/.config/youtube-dl/. It did not change the behaviour. The strace is non-distinct...

[hdclark]

Forgot to mention that this is intermittent.

Here is the verbose info you requested. Nothing fishy happening when youtube-dl handles the download itself.

[me@mymachine ~]$ youtube-dl -v 'http://www.youtube.com/watch?v=BaW_jenozKc' [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['-v', 'http://www.youtube.com/watch?v=BaW_jenozKc'] [debug] Encodings: locale 'UTF-8', fs 'utf-8', out 'UTF-8', pref: 'UTF-8' [debug] youtube-dl version 2014.02.26 [debug] Python version 3.3.3 - Linux-3.12.7-1-ARCH-x86_64-with-arch-Arch-Linux [debug] Proxy map: {} [youtube] Setting language [youtube] BaW_jenozKc: Downloading webpage [youtube] BaW_jenozKc: Downloading video info webpage [youtube] BaW_jenozKc: Extracting video information [download] Destination: youtube-dl test video /SOME UNICODE CHARS HERE THAT GITHUB DISLIKES/ BaW_jenozKc.mp4 [download] 100% of 1.53MiB in 00:00

Hmmm.... maybe there is something strange with my machine. (Or maybe YouTube is playing with me?) Forgoing the script seems to work at the moment:

[me@mymachine ~]$ youtube-dl -g 'http://www.youtube.com/watch?v=BaW_jenozKc' https://r3---sn-j5o5gpx-t8ge.googlevideo.com/videoplayback?expire=1393889820&pcm2fr=yes&source=youtube&key=yt5&signature=AE9ED0C1E31CBE2BB691936C82F7B39C90C9B7BA.52CA633C6D3CADC71376BA55DE754396BB18D88A&ip=WWW.XXX.YYY.ZZZ&mv=m&ms=au&ratebypass=yes&id=05a5bf8de9e8cca7&sparams=id%2Cip%2Cipbits%2Citag%2Cpcm2fr%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&fexp=935637%2C919121%2C909207%2C916623%2C937417%2C937416%2C913434%2C936910%2C936913%2C902907&upn=yZIfgu23iJQ&mt=1393869116&itag=22&requiressl=yes&ipbits=0&sver=3 [me@mymachine ~]$ wget 'https://r3---sn-j5o5gpx-t8ge.googlevideo.com/videoplayback?expire=1393889820&pcm2fr=yes&source=youtube&key=yt5&signature=AE9ED0C1E31CBE2BB691936C82F7B39C90C9B7BA.52CA633C6D3CADC71376BA55DE754396BB18D88A&ip=WWW.XXX.YYY.ZZZ&mv=m&ms=au&ratebypass=yes&id=05a5bf8de9e8cca7&sparams=id%2Cip%2Cipbits%2Citag%2Cpcm2fr%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&fexp=935637%2C919121%2C909207%2C916623%2C937417%2C937416%2C913434%2C936910%2C936913%2C902907&upn=yZIfgu23iJQ&mt=1393869116&itag=22&requiressl=yes&ipbits=0&sver=3' -O file.mp4

seems to correctly download the test file. And the first example from my last post appears to work now too:

./test.sh 'https://www.youtube.com/watch?v=yrBrH5PZtcc'

[hdclark]

I'm not sure what the issue is, but I seem to be the only one having it. I'll close for now and open again when (if) I can get a better handle on it.