ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.41k stars 10.04k forks source link

CrunchyRoll Download says preview of video #17116

Closed 619beave closed 2 years ago

619beave commented 6 years ago

Please follow the guide below


Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.07.29. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?


The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2018.07.29
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

Explanation of your issue in arbitrary form goes here. Please make sure the description is worded well enough to be understood. Provide as much context and examples as possible. If work on your issue requires account credentials please provide them or explain how one can obtain them.

619beave commented 6 years ago

E:\FuniNow-DL>bin\youtube-dl -v -u *** -p **** -f "[height <=? 1080]" --sub-lang enUS --sub-format "ass" --write-sub "http://www.crunchyroll.com/phantom-in-the-twilight/episode-4-important-friends-775039" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', '-f', '[height <=? 1080]', '--sub-lang', 'enUS', '--sub-format', 'ass', '--write-sub', 'http://www.crunchyroll.com/phantom-in-the-twilight/episode-4-important-friends-775039'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2018.04.16 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134 [debug] exe versions: ffmpeg N-83622-gdcd3418, ffprobe N-83622-gdcd3418, rtmpdump 2.3 [debug] Proxy map: {} [Crunchyroll] Logging in [Crunchyroll] 775039: Downloading webpage ERROR: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\build\youtube_dl\YoutubeDL.py", line 789, in extract_info File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\build\youtube_dl\extractor\common.py", line 440, in extract File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\build\youtube_dl\extractor\crunchyroll.py", line 426, in _real_extract youtube_dl.utils.ExtractorError: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I am able to log into CrunchyRoll and view the video on site. Just not able to download it.

bato3 commented 6 years ago

possible duplicate https://github.com/rg3/youtube-dl/issues/17115

(Probably crunchroll disable RCP - Login)

619beave commented 6 years ago

The weird thing is that it worked earlier this morning.

RGFRv2 commented 6 years ago

Same problem.

So there's a solution?

bato3 commented 6 years ago

I have different error, but I'm think it is similar cuse:

ERROR: Wrong login info: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Moved Temporarily (caused by HTTPError());

Original login procedure only make request to RCP-login-page, and don't check is login was OK

I can't get your error, but:

RGFRv2 commented 6 years ago

I have this by running the command with your fix:

python -m youtube_dl --verbose -u USER -p PASS --skip-download --write-sub --sub-format "ass" --sub-lang "frFR" http://www.crunchyroll.com/oneroom/episode-4-hanasaka-yui-is-near-me-774694

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'--skip-download', u'--write-sub', u'--sub-format', u'ass', u'--sub-lang', u'frFR', u'http://www.crunchyroll.com/oneroom/episode-4-hanasaka-yui-is-near-me-774694']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.21
[debug] Python version 2.7.9 (CPython) - Linux-3.16.0-4-amd64-x86_64-with-debian-8.11
[debug] exe versions: ffmpeg 2.6.9, ffprobe 2.6.9, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] Logging in
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/user/youtube-dl-master/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "youtube_dl/__init__.py", line 472, in main
    _real_main(argv)
  File "youtube_dl/__init__.py", line 462, in _real_main
    retcode = ydl.download(all_urls)
  File "youtube_dl/YoutubeDL.py", line 2001, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "youtube_dl/extractor/common.py", line 501, in extract
    self.initialize()
  File "youtube_dl/extractor/common.py", line 405, in initialize
    self._real_initialize()
  File "youtube_dl/extractor/crunchyroll.py", line 118, in _real_initialize
    self._login()
  File "youtube_dl/extractor/crunchyroll.py", line 75, in _login
    cf_page = self.cf_solve_and_download_webpage(login_page, self._LOGIN_URL)
AttributeError: 'CrunchyrollIE' object has no attribute 'cf_solve_and_download_webpage'
bato3 commented 6 years ago

@RGFRv2 it look you don't have this: https://github.com/bato3/youtube-dl/blob/crunchroll2/youtube_dl/extractor/common.py#L2821-L2877

I get fresh copy, and this work for me. (Sometimes on second run - I don't know why sometimes on get token crunchy returns second challegne)

Rikku1990 commented 6 years ago

@bato3, I have the same problem as @RGFRv2 , could you please post an example so that we / I understand how exactly this should work?

Sorry but my english is not the best.

RGFRv2 commented 6 years ago

@bato3 It's working!

This modification will be added in a new version of Youtube-DL? Because right now, I can't run my scripts normally :/

@Rikku1990

It is necessary to clone the branch "crunchroll2" of the git of bato3 https://github.com/bato3/youtube-dl/tree/crunchroll2

Here is the command :

git clone -b crunchroll2 --single-branch https://github.com/bato3/youtube-dl.git

[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.29
[debug] Git HEAD: 1659616
[debug] Python version 2.7.9 (CPython) - Linux-3.16.0-4-amd64-x86_64-with-debian-8.11
[debug] exe versions: ffmpeg 2.6.9, ffprobe 2.6.9, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] Logging in
[Crunchyroll] Solving Cloudflare challenge (5s)
[Crunchyroll] Sending Cloudflare challenge
[Crunchyroll] Get new CSRF Token
[Crunchyroll] Logging in
[Crunchyroll] 774694: Downloading webpage
[Crunchyroll] 774694: Downloading media info for 360p
[Crunchyroll] 774694: Downloading stream info for 360p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info for 480p
[Crunchyroll] 774694: Downloading stream info for 480p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info for 720p
[Crunchyroll] 774694: Downloading stream info for 720p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info for 1080p
[Crunchyroll] 774694: Downloading stream info for 1080p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info
[Crunchyroll] 774694: Downloading subtitles for English (US)
[Crunchyroll] 774694: Downloading subtitles for Español
[Crunchyroll] 774694: Downloading subtitles for Español (España)
[Crunchyroll] 774694: Downloading subtitles for Français (France)
[Crunchyroll] 774694: Downloading subtitles for Português (Brasil)
[Crunchyroll] 774694: Downloading subtitles for العربية
[Crunchyroll] 774694: Downloading subtitles for Italiano
[Crunchyroll] 774694: Downloading subtitles for Deutsch
[Crunchyroll] 774694: Downloading subtitles for Русский
[info] Writing video subtitles to: One Room (Saison 2) Épisode 4 – Yui Hanasaka est à mes côtés-774694.frFR.ass
...
bato3 commented 6 years ago

@Rikku1990 Just install python, and downoad repo: https://github.com/bato3/youtube-dl/archive/crunchroll2.zip (no git need)

@RGFRv2 IDK when, but code need some refactoring - so not so fast. Especially, I'm just learning python. So every hint is indicated

btw - for better experience add --cookie=cookie.txt. This allows you to avoid the cloudflare-challenge on the "second" run

RGFRv2 commented 6 years ago

@bato3

For now I'll go through an alias and we'll see later.

Thanks again !

bato3 commented 6 years ago

@dstftw

  1. It should be detected and solved at generic webpage download level not in extractor.

I don't know how to do, but I would like to see how should this be do. (My Python experience is about 2 weeks)

  1. It should not use any non public domain code.

on MIT License: https://github.com/Anorov/cloudflare-scrape/blob/master/LICENSE enough credits

  1. It should not use nodejs.

cloudflare-scrape perviously use PyExecJS but switched to node. You can still think about using phantom, but this one is also suspended.

619beave commented 6 years ago

@RGFRv2

When will the fix be applied to the new version?

Just wondering...

Thanks

bato3 commented 6 years ago

@619beave according to decision @dstftw it won't.

I hope someone else will do it better

dystopia2017 commented 6 years ago

@RGFRv2 Hello could you provide a windows binary, or explain how to compile or run your scripts on windows. I have python installed, but when i do "python setup.py" it just errors out with

setup.py:73: UserWarning: Skipping file youtube-dl.bash-completion since it is not present. Type make to build all automatically generated files. warnings.warn('Skipping file %s since it is not present. Type make to build all automatically generated files.' %fn)``

bato3 commented 6 years ago

@dystopia2017 this file is in ./devscripts/

they use https://github.com/rg3/youtube-dl/blob/master/devscripts/buildserver.py but I don't know details (only wget "http://$buildserver/build/rg3/youtube-dl/youtube-dl.exe?rev=$REV" -O youtube-dl.exe) or https://github.com/rg3/youtube-dl/blob/master/devscripts/wine-py2exe.sh

619beave commented 6 years ago

It's working again after CrunchyRolls Website restart this morning.

bato3 commented 6 years ago

I'm curious how long it will work.

leprasson commented 6 years ago

I updated to latest windows veriosn 2018.08.04 and now i get this message :

crunchyroll returned error : the CSRF token is invalid. please try to resubmit the form

besweeet commented 6 years ago

Same problem as @leprasson. That's when manually specifying the user agent and cookies.txt. Can still download with streamlink but no way to get subtitles that way. anime-dl seems to work fine, though.

federicorosso1993 commented 6 years ago

Same problem here.

PS. If you are in hurry Xonshiz anime-dl is working right now... Since it's open source too I'm sure youtube-dl will be fixed too soon.

619beave commented 6 years ago

Current error I'm getting now:

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', '-f', '[height <=? 1080]', '--sub-lang', 'enUS', '--sub-format', 'ass', '--write-sub', 'http://www.crunchyroll.com/attack-on-titan/episode-40-old-story-776025'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2018.08.04 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134 [debug] exe versions: ffmpeg N-83622-gdcd3418, ffprobe N-83622-gdcd3418, rtmpdump 2.3 [debug] Proxy map: {} [Crunchyroll] Logging in [Crunchyroll] 776025: Downloading webpage ERROR: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpckoq891b\build\youtube_dl\YoutubeDL.py", line 792, in extract_info File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpckoq891b\build\youtube_dl\extractor\common.py", line 502, in extract File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpckoq891b\build\youtube_dl\extractor\crunchyroll.py", line 429, in _real_extract youtube_dl.utils.ExtractorError: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

N0197exe commented 6 years ago

@kseniasolo yes, it works for me in Windows, but adding --cookies cookies.txt (previously downloaded)

federicorosso1993 commented 6 years ago

@N0197exe I know how to use cookies... but how do you convert that string "/usr/bin/python /home/private/youtube_dl/main.py " to windows?

Can someone from the development team implement kseniasolo fix (if it make sense)?

bato3 commented 6 years ago

For my: It gave me cloudflare anti boot test on login page.

federicorosso1993 commented 6 years ago

I tried with --cookies and user-agent and even -u PRIVATE -p PRIVATE: ./youtube-dl.exe -u PRIVATE -p PRIVATE --username myuser --password mypassword http://www.crunchyroll.com/gintama/episode-358-countless-kings-774987 --cookies cookies.txt --user-agent "USER-AGENT" I still get the ERROR: crunchyroll returned error: The CSRF token is invalid. Please try to resubmit the form.

N0197exe commented 6 years ago

@federicorosso1993. as @bato3 says:

Just install python, and downoad repo: https://github.com/bato3/youtube-dl/archive/crunchroll2.zip

then go to the download path, in my case it is: "C:\Users\Lenovo\Downloads\youtube-dl-crunchroll2\youtube_dl" there you will find "main.py" (will have underscores)

I go to the location with the cmd and do the following example:

main.py -u PRIVATE -p PRIVATE --verbose --cookies cookies.txt --write-sub --sub-format "ass" --sub-lang "enUS" http://www.crunchyroll.com/attack -on-titan / episode-40-old-story-776025

federicorosso1993 commented 6 years ago

@N0197exe Ah ok, thank you. Still it would be great to get a single .exe file... I created a small UI for my personal use (where I can get url, cookies and user-agent with one click), if I have to use main.py I need to change my code.

bato3 commented 6 years ago

@federicorosso1993 Today I'm thinking about:

  1. Request without login and new cookie file
  2. Request with login and cookie

And you can create bath file like: windows: (*.bat)

cd c:/youtube_dl && python -m youtube_dl %*

linux

cd ~/youtube_dl && python -m youtube_dl "$@"

The biggest difference is the need to use kill-tree instead simplekill.

619beave commented 6 years ago

Thanks for the help. The PYthon script is working as:

E:\youtube-dl-crunchroll2\youtube_dl__main__.py -v -u PRIVATE -p PRIVATE -f "[height <=? 1080]" --sub-lang enUS --sub-format "ass" --write-sub "%1"

This is a batch file that is run from another folder where the encodes are muxed together

rehtlaw commented 6 years ago

@bato3 I tried your fix but I got this output instead.

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--geo-bypass-country', 'US', '--username', 'PRIVATE', '--password', 'PRIVATE', '--all-subs', '--verbose', 'https://crunchyroll.com/darling-in-the-franxx']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.29
[debug] Python version 3.7.0 (CPython) - Linux-4.17.12-arch1-1-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg n4.0.2, ffprobe n4.0.2, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Using fake IP 3.230.14.11 (US) as X-Forwarded-For.
[crunchyroll:playlist] Logging in
ERROR: Unable to extract login form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/common.py", line 501, in extract
    self.initialize()
  File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/common.py", line 405, in initialize
    self._real_initialize()
  File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/crunchyroll.py", line 120, in _real_initialize
    self._login()
  File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/crunchyroll.py", line 87, in _login
    login_page, 'login form', group='form')
  File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/common.py", line 972, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract login form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
bato3 commented 6 years ago

@freddiaN strange. This error is raised after solving cloudflare challenge.

Please add --write-pages and attach saved requests (eg paste it in https://pastebin.com/)

And do you have installed node?

puxlit commented 6 years ago

I can reproduce the above error with Python 3.7 too. It seems that reading http.client.HTTPResponse (from fp of the urllib.error.HTTPError raised when making the initial request to https://www.crunchyroll.com/login?next=%2Facct%2Fmembership) yields nothing. Consequently, has_cf_challenge gets tested against an empty string (skipping the bulk of @bato3's contribution), and we finally bail when searching for the login token (again against that empty string).

bato3 commented 6 years ago

I can't reproduce. I always have some HTML in response.

@puxlit You talk about first login_page (is empty)? Can you try change fail_url to _LOGIN_URL.

And can you store cookies by --cookie and what cookies is in?. __cfduid + c_visitor - without CF challenge, cf_clearance - after CF challenge, c_d, c_userid, c_userkey - crunchroll cookies

puxlit commented 6 years ago

Yes, login_page appears to be empty. Changing fail_url doesn't seem to make a difference. Only __cfduid and c_visitor appear to be set.

Your patch works fine on Python 2(.7.15) though; not sure about other versions of Python 3.

puxlit commented 6 years ago

Okay, it would seem that HTTPError closes fp when the error's destroyed, thanks to bpo-15002 (introduced in Python 3.4.1). Given that's the case, I doubt anything that relies on _request_webpage's expected_status functionality will work.

FoxMcloud5655 commented 6 years ago

Not that I have anything to add to the discussion, but I also encounter this error with the most recent version of youtube-dl.

bato3 commented 6 years ago

@puxlit Can you fix it, will you report it as a new issue?

federicorosso1993 commented 6 years ago

Still nothing changed ERROR: crunchyroll returned error: The CSRF token is invalid. Please try to resubmit the form.

FoxMcloud5655 commented 6 years ago

As a workaround, I've logged in using my web browser, then saved the cookies into a text file. You can then use said cookies to "stay" logged in when you run your command on youtube-dl.

federicorosso1993 commented 6 years ago

Ok now I know what the main problem is. I was using --username and --password and --cookies and --user-agent. But unlike the first time right now if you use --username and --password it will ignore -cookies and --user-agent. I think it's better if I can use both and if one fail youtube-dl try to use the other system to log in. I say this because if the cookies are too old the download will fail too, so you cannot ignore --username and --password if you use --cookies and --user-agent.

Right now, for my personal UI, I'm using a double download line: ./youtube-dl.exe --username myuser --password mypass http://www.crunchyroll.com/gintama/episode-358-countless-kings-774987 --cookies cookies.txt --user-agent "myagent"; ./youtube-dl.exe http://www.crunchyroll.com/gintama/episode-358-countless-kings-774987 --cookies cookies.txt --user-agent "myagent"

Of course right now the first line always fail, in the future, if the log in will be fixed, it will download the episode two times (so it's not ideal at all)... That's why I ask you if you can fix and keep in mind that "I should be able to use both method at the same time to log in"