search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.43k stars 10.04k forks source link

Crunchyroll (or CloudFlare ?) throwing a 503. #20027

Closed skid9000 closed 2 years ago

skid9000 commented 5 years ago

Please follow the guide below

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.02.18. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

PS E:\youtube-dl> E:\youtube-dl\youtube-dl.exe --cookies C:\ProgramData\Crunchy-DL\cookies.txt --write-sub --sub-lang frFR --sub-format ass https://www.crunchyroll.com/the-rising-of-the-shield-hero/episode-7-the-savior-of-the-heavenly-fowl-781154 --verbose --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'C:\\ProgramData\\Crunchy-DL\\cookies.txt', '--write-sub', '--sub-lang', 'frFR', '--sub-format', 'ass', 'https://www.crunchyroll.com/the-rising-of-the-shield-hero/episode-7-the-savior-of-the-heavenly-fowl-781154', '--verbose', '--user-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2019.02.18
[debug] Python version 3.4.4 (CPython) - Windows-2012Server-6.2.9200-SP0
[debug] exe versions: ffmpeg N-86848-g03a9e6f, ffprobe N-86848-g03a9e6f
[debug] Proxy map: {}
[crunchyroll] 781154: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()); 
please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. 
Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpw73a_6nx\build\youtube_dl\extractor\common.py", line 605, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpw73a_6nx\build\youtube_dl\YoutubeDL.py", line 2225, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

PS E:\youtube-dl>
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

Description of your issue, suggested solution and other information

Try to download anything from crunchyroll is throwing a 503 error. And before someone ask me, YES my cookie file is correct and functioning properly, i checked if it was the issue multiple times.

serhatcalik commented 5 years ago

Having the same problem and my cookie file is correct and functioning properly

PonyFlare commented 5 years ago

Um. me too? Config is a bit different than the above, but error is identical. Edit: I both updated youtube-dl & used a fresh cookie before generating the following.

Edit 2: Just to avoid the need to scroll down, updating the user agent string fixes it for some. I used this, which only changes the Chrome version number from that shown below

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36

If that doesn't work, you'll have to keep watching for a different fix, unfortunately.

C:\Users\PonyFlare\Videos\BlackClover>youtube-dl --verbose http://www.crunchyroll.com/black-clover [debug] System config: [] [debug] User config: ['--output', '%(title)s.%(ext)s', '--download-archive', 'archive.txt', '--cookies', 'C:/Users/PonyFlare/Videos/cookies.txt', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36', '--sub-lang', 'enUS', '--write-sub', '--format', 'best[height<=720]'] [debug] Custom config: [] [debug] Command-line args: ['--verbose', 'http://www.crunchyroll.com/black-clover'] [debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252 [debug] youtube-dl version 2019.02.18 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763 [debug] exe versions: ffmpeg N-92429-g7992c2867e, ffprobe N-92429-g7992c2867e [debug] Proxy map: {} [crunchyroll:playlist] black-clover: Downloading webpage ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpw73a_6nx\build\youtube_dl\extractor\common.py", line 605, in _request_webpage File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpw73a_6nx\build\youtube_dl\YoutubeDL.py", line 2225, in urlopen File "C:\Python\Python34\lib\urllib\request.py", line 470, in open File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response File "C:\Python\Python34\lib\urllib\request.py", line 508, in error File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

TanukiAI commented 5 years ago

change user agent

Kojirosan commented 5 years ago

Same problem

TanukiAI commented 5 years ago

Well changing user agent is working for some people but not for all people, needs to be fixed, thank you very much!

PonyFlare commented 5 years ago

Confirmed user agent was the issue for me.

Kojirosan commented 5 years ago

PonyFlare, which one do you use?

PonyFlare commented 5 years ago

The one for the same browser I got the cookie from. Got it here: https://www.whoishostingthis.com/tools/user-agent/ My replacement was Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36 - only the Chrome version number changed.

Edit: Pointing out that since the only difference is the version number, if this happens again, going to menu>help>about and pulling the current version is all that's needed for those for whom a user agent change works.

serhatcalik commented 5 years ago

User agent is not working on my case rip

TanukiAI commented 5 years ago

that's what I meant, hope for an fast fix of this

Kojirosan commented 5 years ago

Ty PonyFlame, but it's same, We just have to wait a fix now.

skid9000 commented 5 years ago

I tried with @PonyFlare 's user agent, not working here either...

awei78 commented 5 years ago

first, we must sign in. extractor\crunchyroll.py, line 118, modyfy function _download_webpage: headers add:

        request.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36')
        request.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8')

it is run.

muyklet commented 5 years ago

You have the problem in python 3 too?

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'--verbose', u'-F', u'https://www.crunchyroll.com/my-roommate-is-a-cat'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2019.02.18 [debug] Python version 2.7.12 (CPython) - Linux-4.15.0-45-generic-x86_64-with-LinuxMint [debug] exe versions: ffmpeg 2.8.15, ffprobe 2.8.15, rtmpdump 2.4 [debug] Proxy map: {} [crunchyroll] 781284: Downloading webpage ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 605, in _request_webpage return self._downloader.urlopen(url_or_request) File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2225, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(args) File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

RGFRv2 commented 5 years ago

first, we must sign in. extractor\crunchyroll.py, line 118, modyfy function _download_webpage: headers add:

        request.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36')
        request.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8')

it is run.

@awei78 Can you show the file with modifications?

skid9000 commented 5 years ago

@awei78 solution dosen't work on my side :/

skid9000 commented 5 years ago

I FOUND THE PROBLEM. You must set the same user agent as the one you used to login to crunchyroll to retrieve your cookie file!!!

You don't need to pass thoses lines in the codes however, it works directly with the --user-agent argument.

DynamicField commented 5 years ago

that was dumb

serhatcalik commented 5 years ago

I FOUND THE PROBLEM. You must set the same user agent as the one you used to login to crunchyroll to retrieve your cookie file!!!

You don't need to ass thoses lines in the codes however, it works directly with the --user-agent argument.

Still not working for me :/

awei78 commented 5 years ago

@RGFRv2 @skid9000 : SORRY! In our project, we get cookies by browsercookie and set them into request. from chrome, I get agent from request headers, ant it run. can not run in your project, I think the reason is the cookies.

try: https://www.crunchyroll.com/naruto-shippuden/episode-499-hidden-leaf-story-the-perfect-day-for-a-wedding-part-6-the-outcome-of-the-secret-mission-732517 using Express VPN, select USA area

remitamine commented 5 years ago

the minimum working configuration for free videos:

in addition premium only videos require also a valid c_userid and c_userkey cookies.

joshbrown50 commented 5 years ago

the minimum working configuration for free videos:

  • a valid cookie file with valid cf_clearance cookie supplied to youtube-dl using --cookies option.
  • the same user agent used in the browser to export the cookies supplied to youtube-dl using --user-agent option.
  • using the same ip address where the cookies has been extracted.

in addition premium only videos require also a valid c_userid and c_userkey cookies.

I am doing that and it's still giving me the 503 error. Cookie file with cf_clearance, c_userid and c_userkey freshly generated after a relogin to crunchyroll with no IP switching and putting in the same user agent for the browser I used to get the cookies. Still doesn't work. Not even for free videos.

Cigaes commented 5 years ago

Same for me. Also, I notice that the cookie appears as #HttpOnly_.crunchyroll.com rather than .crunchyroll.com or www.crunchyroll.com in the cookie file, it might be related.

remitamine commented 5 years ago

I notice that the cookie appears as #HttpOnly_.crunchyroll.com rather than .crunchyroll.com or www.crunchyroll.com in the cookie file

#HttpOnly_ should be removed from from the host for the mentioned cookies.

Cigaes commented 5 years ago

#HttpOnly_ should be removed from from the host for the mentioned cookies.

Thanks, it works indeed. I used https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/ / https://github.com/lennonhill/cookies-txt with Firefox to generate the initial cookies file, this is what it produced.

Can we reopen this issue as a double enhancement wish?

joshbrown50 commented 5 years ago

I notice that the cookie appears as #HttpOnly_.crunchyroll.com rather than .crunchyroll.com or www.crunchyroll.com in the cookie file

#HttpOnly_ should be removed from from the host for the mentioned cookies.

I don't have #HttpOnly in my cookies file. It's just .crunchyroll.com but it still doesn't work

joshbrown50 commented 5 years ago

I even tried getting the cookies file from firefox and changing the user agent in the script to match and I'm still getting the same error

TanukiAI commented 5 years ago

@remitamine @rg3 Could you please reopen this? Thanks

remitamine commented 5 years ago
  • Accept the lines with #HttpOnly_ from the cookies file.

this has been rejected upstream: https://bugs.python.org/issue2190

  • Have the crunchyroll extractor print a better message in case of 403: “you must provide a cookies file and the exact same user-agent string that was used to generate it”.

opening for that.

for those saying that it doesn't work even though they really followed every step correctly, there is no way to fix the problem if it still happening, without been able to reproduce the problem locally.

Cigaes commented 5 years ago

this has been rejected upstream: https://bugs.python.org/issue2190

youtube-dl could do a first pass on the file to remove the prefix, but I guess it is not worth the effort. It can also be done with a one-liner, for example with GNU sed:

sed -i 's/^#HttpOnly_//' cookies.txt

Somebody could translate it into python for those who do not have sed and put it on a wiki somewhere, with a link in the error message.

opening for that.

Thanks.

sakalol commented 5 years ago

Guys, I'm using the youtube-dl linux version and I'm with the same problem.

youtube-dl -U youtube-dl is up-to-date (2019.03.01)

Even simple commands like F (just list all available formats of requested videos) doesn't work with Crunchyroll:

youtube-dl -F https://www.crunchyroll.com/boogiepop-and-others/episode-14-overdrive-the-king-of-distortion-1-782413 [crunchyroll] 782413: Downloading webpage ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

And of course trying to download the episode or the subs with the cookies returns the same problem:

youtube-dl --skip-download --all-subs --sub-format "ass" https://www.crunchyroll.com/radiant/episode-18-the-light-that-follows-the-darkness-awakening-780886 --cookies ./cookies.txt [crunchyroll] 780886: Downloading webpage ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Last week this was working perfectly, but now I have this problem. Help?

remitamine commented 5 years ago

@sakalol at least you're not using the same user agent where you have exported the cookies.

unless some one has something to add, it would be better to stop posting the me too comments(and most of them did not even provide the minimum information to check if the configuration is right).

joshbrown50 commented 5 years ago

C:\Windows\system32>D:\Users\User\Downloads\youtube_dl\youtube-dl.exe https://www.crunchyroll.com/the-quintessential-quintuplets/episode-8-the-photo-that-started-it-all-781300 -f best --write-sub --sub-lang enUS --cookies D:\Users\User\Downloads\youtube_dl\cookies.txt --user-agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0 -o "D:\Users\User\Downloads\youtube_dl\Downloads\%(title)s.%(ext)s" --verbose [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['https://www.crunchyroll.com/the-quintessential-quintuplets/episode-8-the-photo-that-started-it-all-781300', '-f', 'best', '--write-sub', '--sub-lang', 'enUS', '--cookies', 'D:\Users\User\Downloads\youtube_dl\cookies.txt', '--user-agent', 'Mozilla/5.0', '(Windows', 'NT', '10.0;', 'Win64;', 'x64;', 'rv:63.0)', 'Gecko/20100101', 'Firefox/63.0', '-o', 'D:\Users\User\Downloads\youtube_dl\Downloads\%(title)s.%(ext)s', '--verbose'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2019.03.01 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763 [debug] exe versions: ffmpeg 3.2.2, ffprobe 3.2.2, rtmpdump 2.4 [debug] Proxy map: {} [crunchyroll] 781300: Downloading webpage ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmplyslo9xs\build\youtube_dl\extractor\common.py", line 605, in _request_webpage File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmplyslo9xs\build\youtube_dl\YoutubeDL.py", line 2225, in urlopen File "C:\Python\Python34\lib\urllib\request.py", line 470, in open File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response File "C:\Python\Python34\lib\urllib\request.py", line 508, in error File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

This is what I'm getting when I try. Probably doesn't add much but figured I'd give more information.

remitamine commented 5 years ago

@joshbrown50 you didn't quote the user-agent.

joshbrown50 commented 5 years ago

Putting it in double quotes (i.e. "" ) worked. Thank you. Also I've never had to include the user agent before. Why the sudden change?

sakalol commented 5 years ago

Oh thanks @remitamine and @joshbrown50. The user-agent of the same browser that we got the cookie from + the double quotes ("") worked perfectly!!

But yeah, I also wonder why the change.

xorima commented 5 years ago

Getting a similar issue, cannot figure it out:

youtube-dl --cookies "/tmp/cookies.txt" --user-agent "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0"  --skip-download https://www.crunchyroll.com/en-gb/the-rising-of-the-shield-hero --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', '/tmp/cookies.txt', '--user-agent', 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0', '--skip-download', 'https://www.crunchyroll.com/en-gb/the-rising-of-the-shield-hero', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.03.01
[debug] Python version 3.6.8 (CPython) - Linux-4.20.8-100.fc28.x86_64-x86_64-with-fedora-28-Twenty_Eight
[debug] exe versions: none
[debug] Proxy map: {}
[generic] the-rising-of-the-shield-hero: Requesting header
WARNING: Could not send HEAD request to https://www.crunchyroll.com/en-gb/the-rising-of-the-shield-hero: HTTP Error 503: Service Temporarily Unavailable
[generic] the-rising-of-the-shield-hero: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 503: Service Temporarily Unavailable (caused by <HTTPError 503: 'Service Temporarily Unavailable'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/jfield/dev/xorima/crunchyroll/.venv/lib64/python3.6/site-packages/youtube_dl/extractor/common.py", line 605, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/jfield/dev/xorima/crunchyroll/.venv/lib64/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

cookies extracted with firefox

TanukiAI commented 5 years ago

3 Things I wanna say:

-1st: I really hope that youtube-dl will find a way to automatically bypass this.

-2nd: What you actually have to do now:

  1. Get new cookies from Crunchyroll and the User Agent for your Browser
  2. Download it with "youtube-dl.exe --user-agent "YOUR USER AGENT" --cookies "YOUR PATH TO COOKIES" (and all the other arguments you use)"

(!)IF YOU USE A VPN, be sure to get the cookies while your VPN is active! I always use this chrome addon for getting the Cookies. In the Addon you have to click "click here" and you will get the right file!

-3rd: If you want to use it in your python script, do this:

import youtube_dl.utils
youtube_dl.utils.std_headers['User-Agent'] = YourUseragent

(Thanks to AKX)

Hope this helped somebody

Kojirosan commented 5 years ago

it's don't work for me

dirkf commented 2 years ago

So it is suggested that 503 is served when the login page is fetched and bot protection requires a captcha to be solved. Using cookies and not --username ... avoids the login page. Then any further blocking (403) is covered in #28398.