[Crunchyroll] all downloads timeout; works fine in browser

Ecliptix commented 7 years ago

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2017.07.09. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2017.07.09

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[ ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[x] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--user-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36', '--cookies', '/home/XXXX/cookies.txt', '-f', 'best', '--write-sub', '--sub-format', 'srt', '--embed-sub', '--sub-lang', 'enUS', '-v', 'http://www.crunchyroll.com/aho-girl/episode-2-aho-girl-multiplies-741345']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.07.09
[debug] Python version 3.6.1 - Linux-4.10.13-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.3, ffprobe 3.3, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] 741345: Downloading webpage
ERROR: Unable to download webpage: timed out (caused by timeout('timed out',))
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 502, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2137, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/bin/youtube-dl/youtube_dl/utils.py", line 974, in http_response
    content = resp.read()
  File "/usr/lib/python3.6/http/client.py", line 456, in read
    return self._readall_chunked()
  File "/usr/lib/python3.6/http/client.py", line 566, in _readall_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python3.6/http/client.py", line 612, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)

Description of your issue, suggested solution and other information

Started happening today -- yt-dl always times out when attempting to download from Crunchyroll, I also tried the embedded test cases with the same issue. I am able to access the site just fine in Chrome and Firefox. Seems like they made some change on their side, not sure how to troubleshoot it further.

dstftw commented 7 years ago

Post the output of strace -f youtube-dl -F http://www.crunchyroll.com/aho-girl/episode-2-aho-girl-multiplies-741345, curl -v http://www.crunchyroll.com/aho-girl/episode-2-aho-girl-multiplies-741345.

Ecliptix commented 7 years ago

Attached as requested:

curl_output.txt strace_output.txt

jonathancatt commented 7 years ago

I am seeing the same issue. Issue appears to be with login, removing my login from .netrc and accessing as "guest/free" users works fine.

Ecliptix commented 7 years ago

I have the problem even if I don't use cookies / login, so I think it's a different issue. The test cases that come with youtube-dl also fail with the same message:

$ python test/test_download.py TestDownload.test_Crunchyroll [Crunchyroll] 645513: Downloading webpage ERROR: Unable to download webpage: timed out (caused by timeout('timed out',)) File "/tmp/youtube-dl/youtube_dl/extractor/common.py", line 502, in _request_webpage return self._downloader.urlopen(url_or_request) File "/tmp/youtube-dl/youtube_dl/YoutubeDL.py", line 2137, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/tmp/youtube-dl/youtube_dl/utils.py", line 974, in http_response content = resp.read() File "/usr/lib/python3.6/http/client.py", line 456, in read return self._readall_chunked() File "/usr/lib/python3.6/http/client.py", line 563, in _readall_chunked chunk_left = self._get_chunk_left() File "/usr/lib/python3.6/http/client.py", line 546, in _get_chunk_left chunk_left = self._read_next_chunk_size() File "/usr/lib/python3.6/http/client.py", line 506, in _read_next_chunk_size line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python3.6/socket.py", line 586, in readinto return self._sock.recv_into(b)

Retrying: 1 failed tries

Ecliptix commented 7 years ago

It appears that it may be Cloudflare (Crunchyroll's CDN) throttling the connection and causing it to timeout, as it works from a different IP. Closing this issue.

ytdl-org / youtube-dl