search

blackjack4494 / youtube-dlc

Command-line program to download various media from YouTube.com and other sites
https://blackjack4494.github.io/youtube-dlc/
The Unlicense
1.22k stars 13 forks source link

[Question] How to bypass HTTPError 429: 'Too Many Requests' #195

Open mbnoimi opened 4 years ago

mbnoimi commented 4 years ago

Checklist

Question

I want to know how to bypass HTTPError 429: 'Too Many Requests' error message. This issue frequently appears. To solve it I restart my router whenever it occurs (for renewing the public IP). Is there any workaround to bypass it without needing to restart my router or waiting google for ending the restriction?

Log

$ youtube-dlc --newline -i --force-ipv4 --age-limit 30 -R 99 --hls-prefer-native -o "/home/laptop/Downloads/youtube-dl/downloads/Documentary/Nat Geo Abu Dhabi/%(title)s.%(ext)s" -f mp4 --merge-output-format mp4 --write-sub --verbose --continue https://www.youtube.com/user/natgeoabudhabime/videos
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--newline', '-i', '--force-ipv4', '--age-limit', '30', '-R', '99', '--hls-prefer-native', '-o', '/home/laptop/Downloads/youtube-dl/downloads/Documentary/Nat Geo Abu Dhabi/%(title)s.%(ext)s', '-f', 'mp4', '--merge-output-format', 'mp4', '--write-sub', '--verbose', '--continue', 'https://www.youtube.com/user/natgeoabudhabime/videos']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dlc version 2020.10.09
[debug] Python version 3.8.5 (CPython) - Linux-5.4.0-51-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Proxy map: {}
[youtube:user] natgeoabudhabime: Downloading channel page
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading webpage
[download] Downloading playlist: Uploads from Nat Geo Abu Dhabi
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #1
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #2
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #3
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #4
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #5
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #6
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #7
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #8
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #9
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #10
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #11
[youtube:playlist] playlist Uploads from Nat Geo Abu Dhabi: Downloading 1119 videos
[download] Downloading video 1 of 1119
[youtube] R9K9NC2MpDs: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dlc with the --verbose flag and include its complete output.
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/extractor/common.py", line 632, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/YoutubeDL.py", line 2308, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

[download] Downloading video 2 of 1119
[youtube] d4lwAIglTrE: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dlc with the --verbose flag and include its complete output.
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/extractor/common.py", line 632, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/YoutubeDL.py", line 2308, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
^C
ERROR: Interrupted by user
$
mbnoimi commented 4 years ago

NOTE: I tried to use --cookies but it didn't solve it.

blackjack4494 commented 4 years ago

There is --sleep-interval however that will only sleep between downloads of files and not between (api) calls like Downloading page.
When I was working on the youtube history feed fix (which is not released btw) image I had to rate limit calls with sleep (internal function).
6 Seconds as in the example is a bit generous. It seems 1-3 seconds (where you should be completely safe with 3 seconds) is actually enough.
Furthermore it also seems this problem is rather related to youtube only as I haven't noticed any other service with similar problems.

october262 commented 4 years ago

try using a proxy or a VPN - https://github.com/ytdl-org/youtube-dl/issues/22839

blackjack4494 commented 4 years ago

@october262 that won't always work out. It's the anti bot design by YouTube. So the tools (youtube-dlc in this case) needs to adjust to that. As I already mentioned all that is needed is some timeout between calls. Current behaviour is easy to identify as bots. But normal user behaviour is not able to process so many request at once so you have to mirror this behaviour.