HTTP Error 429: Too Many Requests (caused by HTTPError()); [youtube]

[bussiere]

My version is the last one :100: youtube-dl --version 2015.03.03.1

youtube-dl -f 22 https://www.youtube.com/watch?v=8MMnbeGDzdw [youtube] 8MMnbeGDzdw: Downloading webpage ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Thanks for the program and regards

[marburger93]

I have the same problem with youtube. HTTP requests without youtube-dl are possible

~# youtube-dl https://www.youtube.com/watch?v=FS2EZ18Fcv8 [youtube] FS2EZ18Fcv8: Downloading webpage ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. ~# youtube-dl -U youtube-dl is up-to-date (2015.03.03.1)

[TheBoroer]

Same issue here. Requests without youtube-dl are working fine.

URL used: http://www.youtube.com/watch?v=PKvg5PNPihY

Verbose Output:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '-s', '-g', 'http://www.youtube.com/watch?v=PKvg5PNPihY']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.03.03.1
[debug] Python version 2.6.6 - Linux-3.10.23-xxxx-std-ipv6-64-x86_64-with-centos-6.5-Final
[debug] exe versions: ffmpeg 0.6.5, ffprobe 0.6.5
[debug] Proxy map: {}
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "./youtube-link/youtube_dl/extractor/common.py", line 314, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "./youtube-link/youtube_dl/YoutubeDL.py", line 1674, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[sargue]

+1

My output:

$ youtube-dl https://www.youtube.com/watch?v=j3mhkYbznBk --verbose [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['https://www.youtube.com/watch?v=j3mhkYbznBk', '--verbose'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2015.03.03.1 [debug] Python version 2.7.3 - Linux-3.2.13-xxxx-std-ipv6-32-i686-with-debian-7.8 [debug] exe versions: avconv 0.8.16-6, avprobe 0.8.16-6, ffmpeg 0.8.16-6, ffprobe 0.8.16-6 [debug] Proxy map: {} [youtube] j3mhkYbznBk: Downloading webpage ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/home/sargue/bin/youtube-dl/youtube_dl/extractor/common.py", line 314, in _request_webpage return self._downloader.urlopen(url_or_request) File "/home/sargue/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1674, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python2.7/urllib2.py", line 407, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 520, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 445, in error return self._call_chain(_args) File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain result = func(_args) File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[phihag]

Everything is working fine for me from multiple machines all over the world, so I cannot diagnose this issue. Can anyone of you offer a proxy or VPN at an affected machine? If so, please contact me.

[Mazwak]

It seems to be a server side throttling with a recaptcha. I have no idea how it can be bypassed.

[bo3rnd]

It can only be bypassed with an other IP.

BTW: I have a server at OVH and it seems their whole IPv6 address space is blacklisted, so changing to an other IPv6 address does not work. Only IPv4 does work.

[Mazwak]

I do proxy through OVH also. Loading exported firefox cookies into youtube-dl, after entering the captcha in FF, works.

I tried without ovh proxying, and it is the same.

[Aiexis]

Did someone found a solution for OVH ? Tried 2 VPS, and same issue.

[TheBoroer]

I opened a ticket with OVH support and they escalated it. I think it might be an issue with youtube blocking all OVH ipv6 blocks (since ipv4 works fine).

[Aiexis]

Is there a way to user ipv4 instead of ipv6 ? (configuring debian maybe ?)

[TheBoroer]

@Aiexis youtube-dl has -4 and -6 parameters that you can use to tell youtube-dl to run it through ipv4 or ipv6. Check the readme/docs

[Aiexis]

Thank you ;) -4 parameter works on one VPS (but not the second)

[jefi009]

@BlackPropeller Did you get any reply from OVH so far? I've opened a ticket as well and nobody took care of it so far.

[TheBoroer]

@jefi009 Yup, they replied but they weren't really that helpful.

After investigating the issue with our network specialists and testing the command from multiple sources inside and outside of our data centres, we can see that the issue is on youtube's side. From what we understand, they are trying to prioritize the use of their API instead. For more information on the issue, I would suggest contacting them directly.

They pretty much said we're on our own...lol.

[TheBoroer]

@isez I had a feeling OVH couldnt help but i did find it odd that OVH's entire IPv6 block was blocked by youtube (even on servers/systems that have never made a single request to youtube). They had this sort of issue before with IPv4 and were able to sort the issue out with youtube. That's why i asked them to look into it again but for IPv6 this time.

The cookies option/solution works fine for now i guess.

[jefi009]

@BlackPropeller It sounds like they didn't even contact YT. I also noticed that their whole Range is blocked since I've also tried to use servers that never made a request to YT.

My guts tell me that it is more likely that someone started a DDOS from within the OVH network towards yt which triggered an automatic system at YT that blocked the OVH range. Their explanation would only make sense if their IPv4 ranges would be blocked as well.

[jefi009]

I did also notice that the 429's appeared one week before and lasted only a few hours, on March 5th the 429's also disappeared temporarily after a few hours only to reappear permanently for now.

This could indicate

A) A test carried out by YT. B) An automatic DOS protection that increases the block time after each block.

[bo3rnd]

Not only OVH is affected, it seems the whole IPv6 space from online.net (another popular french provider) is also blacklisted.

[Utopiah]

For those using YoutubeDL.py setting source_address with the IPv4 in use works (no IPv4 parameter seen in ydl_opts{}).

[renaudallard]

I can confirm what bo3rnd says. My IPv6 range at online.net is also blocked by this and I get a captcha. But as jefi009 said, this makes no sense as they should also blacklist ipv4 ranges if it was to block servers. In my case, I use my ipv6 range at online to get an ipv6 for my home machines, so this is really a hassle.

[pitrou]

Same here on a OVH server, and if I force IPv4 (using the -4 option) it works.

[sisihagen]

Yes with ipv4 goes well with ovh servers....

[celogeek]

The IPV4 lead to a video that explain you need to do something with the captcha. But still the same issue.

[luklau88]

Hi,

Please accept my appreciation in advance: Can someone help me to understand / resolve the following issue?

I was playing with an aws instance to run the following command through node.js. I have not ran the command more than 20 times, and start getting the following error code. Will youtube block my IP for only ~20 requests?

---

youtube-dl -g -f 140 RgKAFK5djSk -v

[debug] System config: [u'--prefer-free-formats'] [debug] User config: [] [debug] Command-line args: [u'-g', u'-f', u'140', u'RgKAFK5djSk', u'-v'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2015.06.15 [debug] Python version 2.7.9 - Linux-3.14.44-32.39.amzn1.x86_64-x86_64-with-glibc2.2.5 [debug] exe versions: none [debug] Proxy map: {} ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 312, in _request_webpage return self._downloader.urlopen(url_or_request) File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1730, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib64/python2.7/urllib2.py", line 437, in open response = meth(req, response) File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python2.7/urllib2.py", line 475, in error return self._call_chain(args) File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain result = func(args) File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[brubraz]

What proxy servers are good to use? I have problem with low download speed.

[ishupreet]

For me wget is also not working in an AWS Ubuntu and in its browser if i go to youtube.com, it asked for a captcha once, after which youtube started working in the browser(without the captcha). But, wget is still not working in the shell.

[0xcaff]

I can confirm it is not working on AWS also. This limitation is probably put in place by youtube to prevent bots and scrapers from crawling their site causing unnecessary performance hits. http://meta.stackexchange.com/questions/90021/why-is-amazon-aws-blocked The only way this issue can be fixed is by prompting the user to view and complete the capacha.

[brubraz]

@ishupreet and @caffinatedmonkey Are you using Elastic IP? I had the same problem, I fixed removing EIP from my EC2, just using default Public IP.

[JehandadK]

@burnaz coudnt fix using Public IP :S

[rubenbarreiro]

disabling EIP did work for me

[JehandadK]

Changing the region from ireland to virginia worked for me!

[slwst]

Enabling / disabling EIP had no effect, still being blocked from the AWS virginia region.

[evgnomon]

-4 worked for me!

[TheNeo2k]

easy fix for linux users. nano /etc/youtube-dl.conf

--force-ipv4

ctrl + x Y done.

[BaseMax]

i have found something for this. if your ip blocked and your have linux vps/machine then install squid3 proxy on same vps/machine then set your browser to connect through this proxy open any youtube video url you will be prompted to solve recaptcha while submitting recaptcha challenge watch the http header cookies youtube.com sets in your browser. example

Set-Cookie: goojf=8b98f300cba05a75890b5ed60058e400c3cAdVJoVQ==; path=/; domain=.youtube.com; expires=Fri, 20-May-2016 13:24:57 GMT

then save this cookie in youtube-dl cookie file

its working for me but not sure for how long it will last :smile:

[Bryannn]

hebrew878 is that cookie still working for you?

[BaseMax]

no bruh, it was an ipv6, worked for only few hours. i ended up adding -4 to my youtube-dl command and its working so far no issue.

On 27/05/2016, Bryannn notifications@github.com wrote:

hebrew878 is that cookie still working for you?

---

You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/rg3/youtube-dl/issues/5138#issuecomment-222075872

[Bryannn]

Yeh cookie works for like 12 hrs only. How many request you send on an average? I send about 100k, I need something permanent.

[ralyodio]

using -4 option on OVH works

[yarwelp]

Using a VPS provided by Vultr, requests are blocked on both IPv4 and IPv6. I think we should develop a ReCAPTCHA proxy tool to make it fast and easy to solve a captcha and use the cookie. That might be outside the scope of youtube-dl, though.

[itslukej]

"-4" and "-6" still seems to be very experimental... (it's not forcing it)

[danwdart]

No good from my DigitalOcean droplet either :/ Going to try proxying through the machine and entering the captcha myself... Although this is never going to be any good if this keeps happening.

[rudolphos]

@bo3rnd why did they blacklist OVH servers ? Crooked Google as always trying to prevent people from downloading videos..

[danwdart]

I think it's against the terms of service not to use their own viewer tbh. Bad but they don't want you to watch the content without their ads.

Which is why they only allow CC-BY (in order to relicence it so that you're not allowed to download it).

[yarwelp]

@dandart CC-BY is the most liberal CC license short of CC0. This license does not imply that you're not allowed to download it, quite the opposite. CC-BY allows others to take a work someone has made and to redistribute it with or without modification as long as you attribute the original author.

Meanwhile, YouTube may have some conditions about downloading in their ToS. I dunno, I haven't read the ToS. Usually conditions about downloading are about large-scale, automated downloading of resources (such as videos) and/or scraping of content like titles, descriptions and comments.

If anyone knows the official stand YouTube has, if any, I'd like to hear it.

[danwdart]

Well, CC-BY is the most liberal in that it is not copyleft: i.e. you can still relicence it to say that you're not allowed to download it! This is what I mean when I think YouTube is doing that.

If other CC-licenced videos were allowed on YouTube (such as any -SA) then YouTube wouldn't be able to stop people downloading, and as it is, they don't support anything that stops them blocking you from downloading or stops them making money off of it, obviously for ads.

Their ToS says that one is not allowed to download from them, just at all.

I've tweeted them but not yet got a response.

[danwdart]

I wonder - has anyone tried it with authentication? Maybe it'll work with an API key or a un/pw login cookie?

[i2mo]

same here on a DigitalOcean Host located in Frankfurt. IPv6 is blocked, v4 is working. youtube-dl -4 is working then.

[ghost]

It's unfortunate, but I can understand why youtube does this. Since providers give their customers large IPv6 block allocations, it's not feasible to block abusers by their IPv6 address - they can simply change or randomise it.

Oh, and the bug here is that youtube-dl tries using IPv6 even when youtube is unreachable over IPv6. (Enabling IPv6 over OpenVPN requires additional configuration).