search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.68k stars 9.97k forks source link

Configuration problem with youtube-dl on Ubuntu or being blocked by youtube? #9477

Closed eriophora closed 7 years ago

eriophora commented 8 years ago

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=5qVK7GIZ394', u'--verbose', u'--skip-download']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.05.10
[debug] Python version 2.7.6 - Linux-3.13.0-74-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] 5qVK7GIZ394: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 503: Service Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 388, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1945, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
...
<end of log>

Description of your issue, suggested solution and other information

As you can see above, I'm getting a 503: Unavailable error for any youtube video I try to download on my Ubuntu machine. However, on my mac machine, these videos download fine. Further, when I attempt to curl the URL on ubuntu:

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://ipv4.google.com/sorry/IndexRedirect?continue=https://www.youtube.com/watch%3Fv%3D5qVK7GIZ394&amp;q=CGMSBDbEbi0Yr8TUuQUiGQDxp4NLT7DjCLl7j6VcIrcOZl-a3Qr67R8">here</A>.
</BODY></HTML>

(which works like normal on the mac)

On the Ubuntu machine, I can curl youtube just fine, however. Please help!

nsilberman commented 8 years ago

same for us

Hrxn commented 8 years ago

Anything suspicious with --dump-user-agent?

Did you try it with another setting for --user-agent UA ?

eriophora commented 8 years ago

I'll try that, thanks!

eriophora commented 8 years ago

The user agent was normal:

Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/44.0 (Chrome)

However, the problem is gone--though I'm using a new IP address. The next time I can get it to happen, I'll check the user agent. However, I'm going to be extra cautious from here on out to avoid it.

Is there no precedence in youtube-dl history for people being temporarily (or permanently?) blocked[/throttled/otherwise attenuated] by youtube? that's surprising!

Hrxn commented 8 years ago

Not that I'm aware of..

But it is very much possible that some IPs or rather IP ranges are blocked or throttled because of unusual traffic, although only temporarily.

yan12125 commented 8 years ago

Could you try:

youtube-dl -v -4 "https://www.youtube.com/watch?v=5qVK7GIZ394"

And:

youtube-dl -v -6 "https://www.youtube.com/watch?v=5qVK7GIZ394"
snarfed commented 8 years ago

looks like it probably is an IP block or throttle. we (huffduff-video) started seeing this a couple days ago too from 52.37.171.81 (AWS EC2 us-west-2), first with 2016.04.13 and now with 2016.05.10. however, i'm able to download that video ok from 67.169.95.105 (Comcast residential, San Francisco).

here are the transcripts. first EC2 us-west-2:

youtube-dl -v -4 "https://www.youtube.com/watch?v=5qVK7GIZ394"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'-4', u'https://www.youtube.com/watch?v=5qVK7GIZ394']
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2016.05.10
[debug] Python version 2.7.10 - Linux-4.4.8-20.46.amzn1.x86_64-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 2.5.4-, ffprobe 2.5.4-
[debug] Proxy map: {}
[youtube] 5qVK7GIZ394: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 503: Service Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 388, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1945, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 469, in error
    result = self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 656, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

$ youtube-dl -v -6 "https://www.youtube.com/watch?v=5qVK7GIZ394"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'-6', u'https://www.youtube.com/watch?v=5qVK7GIZ394']
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2016.05.10
[debug] Python version 2.7.10 - Linux-4.4.8-20.46.amzn1.x86_64-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 2.5.4-, ffprobe 2.5.4-
[debug] Proxy map: {}
[youtube] 5qVK7GIZ394: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 101] Network is unreachable> (caused by URLError(error(101, 'Network is unreachable'),))
  File "/usr/local/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 388, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1945, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/site-packages/youtube_dl/utils.py", line 932, in https_open
    req, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 1199, in do_open
    raise URLError(err)

and Comcast SF:

$ youtube-dl -v -4 "https://www.youtube.com/watch?v=5qVK7GIZ394"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'-4', u'https://www.youtube.com/watch?v=5qVK7GIZ394']
[debug] Encodings: locale US-ASCII, fs utf-8, out US-ASCII, pref US-ASCII
[debug] youtube-dl version 2016.05.10
[debug] Python version 2.7.10 - Darwin-15.4.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.8.6, ffprobe 2.8.6
[debug] Proxy map: {}
[youtube] 5qVK7GIZ394: Downloading webpage
[youtube] 5qVK7GIZ394: Downloading video info webpage
[youtube] 5qVK7GIZ394: Extracting video information
[youtube] 5qVK7GIZ394: Downloading MPD manifest
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r3---sn-n4v7sn76.googlevideo.com/videoplayback?id=e6a54aec6219dfde&itag=135&source=youtube&requiressl=yes&nh=IgpwcjAzLnNqYzA3KgkxMjcuMC4wLjE&mn=sn-n4v7sn76&mv=m&ms=au&pl=17&initcwndbps=2321250&mm=31&ratebypass=yes&mime=video/mp4&gir=yes&clen=18666655&lmt=1420363726995111&dur=285.666&fexp=9406984,9416126,9416891,9422596,9428398,9431012,9433096,9433223,9433946,9436447&key=dg_yt0&signature=2296FDC9FA417CE6130702B465B1B1447F023A17.2A384B0B37A915DBD0DFBF38E949E26E7F6BC92B&mt=1463262197&upn=tRQPu42Hm8A&sver=3&ip=67.169.95.105&ipbits=0&expire=1463283981&sparams=ip,ipbits,expire,id,itag,source,requiressl,nh,mn,mv,ms,pl,initcwndbps,mm,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: Wo y3 me taste-5qVK7GIZ394.f135.mp4
[download] 100% of 17.80MiB in 00:16
[debug] Invoking downloader on u'https://r3---sn-n4v7sn76.googlevideo.com/videoplayback?id=e6a54aec6219dfde&itag=251&source=youtube&requiressl=yes&nh=IgpwcjAzLnNqYzA3KgkxMjcuMC4wLjE&mn=sn-n4v7sn76&mv=m&ms=au&pl=17&initcwndbps=2321250&mm=31&ratebypass=yes&mime=audio/webm&gir=yes&clen=4832926&lmt=1413956629572797&dur=285.781&fexp=9406984,9416126,9416891,9422596,9428398,9431012,9433096,9433223,9433946,9436447&key=dg_yt0&signature=81480B23F888B73B85F94A9D7E8265F84AB97058.5B2B2A85F4833B11E74F66F038050581BEA131B1&mt=1463262197&upn=tRQPu42Hm8A&sver=3&ip=67.169.95.105&ipbits=0&expire=1463283981&sparams=ip,ipbits,expire,id,itag,source,requiressl,nh,mn,mv,ms,pl,initcwndbps,mm,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: Wo y3 me taste-5qVK7GIZ394.f251.webm
[download] 100% of 4.61MiB in 00:03
[ffmpeg] Merging formats into "Wo y3 me taste-5qVK7GIZ394.mkv"
[debug] ffmpeg command line: ffmpeg -y -i 'file:Wo y3 me taste-5qVK7GIZ394.f135.mp4' -i 'file:Wo y3 me taste-5qVK7GIZ394.f251.webm' -c copy -map 0:v:0 -map 1:a:0 'file:Wo y3 me taste-5qVK7GIZ394.temp.mkv'
Deleting original file Wo y3 me taste-5qVK7GIZ394.f135.mp4 (pass -k to keep)
Deleting original file Wo y3 me taste-5qVK7GIZ394.f251.webm (pass -k to keep)
yan12125 commented 8 years ago

@snarfed Is the output of curl -v https://www.youtube.com/watch?v=5qVK7GIZ394 on the AWS VM similar to the one in @eriophora's report? (A 302 page with a link https://ipv4.google.com/sorry/IndexRedirect?...)

nsilberman commented 8 years ago

for us, it didn't work within a docker launched on a centOS instance, but worked well directly from the instance. strange

snarfed commented 8 years ago

@yan12125 i believe so, yes...but not any more, youtube-dl is working happily on the AWS VM now. looks like the block wasn't permanent.

yan12125 commented 8 years ago

@nsilberman @eriophora Is the problem still?

junmuzi commented 7 years ago

Hi @snarfed @yan12125 @junmuzi , I also hit this issue as follow show. But I could watch this video via chrome browser normally. So I don't think this is due to block my IP from youtube website.

Or if so, could anyone provide how to add chrome browser into youtube-dl, so youtube-dl could download the youtube video as normal. In other words, youtube-dl could be faked as a chrome browser. Thank you very much.

# youtube-dl -v -6 "https://www.youtube.com/watch?v=5qVK7GIZ394" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '-6', 'https://www.youtube.com/watch?v=5qVK7GIZ394'] [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2017.08.06 [debug] Python version 3.5.2 - Linux-4.4.0-38-generic-x86_64-with-Ubuntu-16.04-xenial [debug] exe versions: ffmpeg 2.8.6-1ubuntu2, ffprobe 2.8.6-1ubuntu2, rtmpdump 2.4 [debug] Proxy map: {} [youtube] 5qVK7GIZ394: Downloading webpage ERROR: Unable to download webpage: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: 'Service Unavailable'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/usr/local/bin/youtube-dl_juli/youtube_dl/extractor/common.py", line 502, in _request_webpage return self._downloader.urlopen(url_or_request) File "/usr/local/bin/youtube-dl_juli/youtube_dl/YoutubeDL.py", line 2166, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 504, in error result = self._call_chain(args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(args) File "/usr/lib/python3.5/urllib/request.py", line 696, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/usr/lib/python3.5/urllib/request.py", line 472, in open response = meth(req, response) File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.5/urllib/request.py", line 510, in error return self._call_chain(args) File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain result = func(args) File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp)

# curl -v -6 "https://www.youtube.com/watch?v=5qVK7GIZ394" * Trying 2404:6800:4008:c07::8a... * Connected to www.youtube.com (2404:6800:4008:c07::8a) port 443 (#0) * found 173 certificates in /etc/ssl/certs/ca-certificates.crt * found 704 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: .google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google Inc,CN=.google.com * start date: Tue, 25 Jul 2017 08:46:44 GMT * expire date: Tue, 17 Oct 2017 08:28:00 GMT * issuer: C=US,O=Google Inc,CN=Google Internet Authority G2 * compression: NULL * ALPN, server accepted to use http/1.1

GET /watch?v=5qVK7GIZ394 HTTP/1.1 Host: www.youtube.com User-Agent: curl/7.47.0 Accept: /

< HTTP/1.1 302 Found < Location: https://ipv6.google.com/sorry/index?continue=https://www.youtube.com/watch%3Fv%3D5qVK7GIZ394&q=EhAgAQ2oAgMyAW4LhP_-ClGvGOCMq8wFIhkA8aeDS8TSS7FEdAblyeKVz9e05q69AG6hMgNyY24 < Date: Wed, 09 Aug 2017 08:22:56 GMT < Pragma: no-cache < Expires: Fri, 01 Jan 1990 00:00:00 GMT < Cache-Control: no-store, no-cache, must-revalidate < Content-Type: text/html; charset=UTF-8 < Server: HTTP server (unknown) < Content-Length: 371 < X-XSS-Protection: 1; mode=block < X-Frame-Options: SAMEORIGIN < Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,36,35" <

302 Moved \

302 Moved

The document has moved here. \
  • Connection #0 to host www.youtube.com left intact

# youtube-dl --version 2017.08.06

junmuzi commented 7 years ago

Seems can not work still when I change my IPv6 addr.

yan12125 commented 7 years ago

curl reports 302, which means it should jump to the next page and the final result is unknown yet. Could you run the following command and paste the result?

curl -v -L -6 "https://www.youtube.com/watch?v=5qVK7GIZ394"

In other words, youtube-dl could be faked as a chrome browser.

FWIW, --add-header "User-Agent: (user-agent for chrome)". Yet it's unlikely that YouTube blocks clients based on User-Agent string.

junmuzi commented 7 years ago

@yan12125

# curl -v -L -6 "https://www.youtube.com/watch?v=5qVK7GIZ394"

About this page

Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen?

IP address: 2001:da8:203:3201:6e0b:84ff:fe0a:51af
Time: 2017-08-09T10:34:12Z
URL: https://www.youtube.com/watch?v=5qVK7GIZ394

junmuzi commented 7 years ago

hi @yan12125 ,

Based on above info "Our systems have detected unusual traffic from your computer network. ", seems it is due to my large num of requests.

Seems it still can not work when I just change my IPv6 addr simply.

Could you give me some advise to solve this issue? Thank you very much.

yan12125 commented 7 years ago

This is apparently the same issue as #13864. Let's merge the older issue into the new one.

Could you give me some advise to solve this issue?

I'mt not sure what's the cause yet. Please move on to #13864 for further discussion, thanks!