Closed 619beave closed 2 years ago
E:\FuniNow-DL>bin\youtube-dl -v -u *** -p **** -f "[height <=? 1080]" --sub-lang enUS --sub-format "ass" --write-sub "http://www.crunchyroll.com/phantom-in-the-twilight/episode-4-important-friends-775039" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', '-f', '[height <=? 1080]', '--sub-lang', 'enUS', '--sub-format', 'ass', '--write-sub', 'http://www.crunchyroll.com/phantom-in-the-twilight/episode-4-important-friends-775039'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2018.04.16 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134 [debug] exe versions: ffmpeg N-83622-gdcd3418, ffprobe N-83622-gdcd3418, rtmpdump 2.3 [debug] Proxy map: {} [Crunchyroll] Logging in [Crunchyroll] 775039: Downloading webpage ERROR: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\build\youtube_dl\YoutubeDL.py", line 789, in extract_info File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\build\youtube_dl\extractor\common.py", line 440, in extract File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp9r95p1_v\build\youtube_dl\extractor\crunchyroll.py", line 426, in _real_extract youtube_dl.utils.ExtractorError: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
I am able to log into CrunchyRoll and view the video on site. Just not able to download it.
possible duplicate https://github.com/rg3/youtube-dl/issues/17115
(Probably crunchroll disable RCP - Login)
The weird thing is that it worked earlier this morning.
Same problem.
So there's a solution?
I have different error, but I'm think it is similar cuse:
ERROR: Wrong login info: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Moved Temporarily (caused by HTTPError());
Original login procedure only make request to RCP-login-page, and don't check is login was OK
I can't get your error, but:
youtube-dl>python -m youtube_dl
I have this by running the command with your fix:
python -m youtube_dl --verbose -u USER -p PASS --skip-download --write-sub --sub-format "ass" --sub-lang "frFR" http://www.crunchyroll.com/oneroom/episode-4-hanasaka-yui-is-near-me-774694
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'--skip-download', u'--write-sub', u'--sub-format', u'ass', u'--sub-lang', u'frFR', u'http://www.crunchyroll.com/oneroom/episode-4-hanasaka-yui-is-near-me-774694']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.21
[debug] Python version 2.7.9 (CPython) - Linux-3.16.0-4-amd64-x86_64-with-debian-8.11
[debug] exe versions: ffmpeg 2.6.9, ffprobe 2.6.9, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] Logging in
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/user/youtube-dl-master/youtube_dl/__main__.py", line 19, in <module>
youtube_dl.main()
File "youtube_dl/__init__.py", line 472, in main
_real_main(argv)
File "youtube_dl/__init__.py", line 462, in _real_main
retcode = ydl.download(all_urls)
File "youtube_dl/YoutubeDL.py", line 2001, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "youtube_dl/YoutubeDL.py", line 792, in extract_info
ie_result = ie.extract(url)
File "youtube_dl/extractor/common.py", line 501, in extract
self.initialize()
File "youtube_dl/extractor/common.py", line 405, in initialize
self._real_initialize()
File "youtube_dl/extractor/crunchyroll.py", line 118, in _real_initialize
self._login()
File "youtube_dl/extractor/crunchyroll.py", line 75, in _login
cf_page = self.cf_solve_and_download_webpage(login_page, self._LOGIN_URL)
AttributeError: 'CrunchyrollIE' object has no attribute 'cf_solve_and_download_webpage'
@RGFRv2 it look you don't have this: https://github.com/bato3/youtube-dl/blob/crunchroll2/youtube_dl/extractor/common.py#L2821-L2877
I get fresh copy, and this work for me. (Sometimes on second run - I don't know why sometimes on get token crunchy returns second challegne)
@bato3, I have the same problem as @RGFRv2 , could you please post an example so that we / I understand how exactly this should work?
Sorry but my english is not the best.
@bato3 It's working!
This modification will be added in a new version of Youtube-DL? Because right now, I can't run my scripts normally :/
@Rikku1990
It is necessary to clone the branch "crunchroll2" of the git of bato3 https://github.com/bato3/youtube-dl/tree/crunchroll2
Here is the command :
git clone -b crunchroll2 --single-branch https://github.com/bato3/youtube-dl.git
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.29
[debug] Git HEAD: 1659616
[debug] Python version 2.7.9 (CPython) - Linux-3.16.0-4-amd64-x86_64-with-debian-8.11
[debug] exe versions: ffmpeg 2.6.9, ffprobe 2.6.9, rtmpdump 2.4
[debug] Proxy map: {}
[Crunchyroll] Logging in
[Crunchyroll] Solving Cloudflare challenge (5s)
[Crunchyroll] Sending Cloudflare challenge
[Crunchyroll] Get new CSRF Token
[Crunchyroll] Logging in
[Crunchyroll] 774694: Downloading webpage
[Crunchyroll] 774694: Downloading media info for 360p
[Crunchyroll] 774694: Downloading stream info for 360p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info for 480p
[Crunchyroll] 774694: Downloading stream info for 480p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info for 720p
[Crunchyroll] 774694: Downloading stream info for 720p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info for 1080p
[Crunchyroll] 774694: Downloading stream info for 1080p
[Crunchyroll] 774694: Downloading m3u8 information
[Crunchyroll] 774694: Downloading media info
[Crunchyroll] 774694: Downloading subtitles for English (US)
[Crunchyroll] 774694: Downloading subtitles for Español
[Crunchyroll] 774694: Downloading subtitles for Español (España)
[Crunchyroll] 774694: Downloading subtitles for Français (France)
[Crunchyroll] 774694: Downloading subtitles for Português (Brasil)
[Crunchyroll] 774694: Downloading subtitles for العربية
[Crunchyroll] 774694: Downloading subtitles for Italiano
[Crunchyroll] 774694: Downloading subtitles for Deutsch
[Crunchyroll] 774694: Downloading subtitles for Русский
[info] Writing video subtitles to: One Room (Saison 2) Épisode 4 – Yui Hanasaka est à mes côtés-774694.frFR.ass
...
@Rikku1990 Just install python, and downoad repo: https://github.com/bato3/youtube-dl/archive/crunchroll2.zip (no git
need)
@RGFRv2 IDK when, but code need some refactoring - so not so fast. Especially, I'm just learning python. So every hint is indicated
btw - for better experience add --cookie=cookie.txt
. This allows you to avoid the cloudflare-challenge on the "second" run
@bato3
For now I'll go through an alias and we'll see later.
Thanks again !
@dstftw
- It should be detected and solved at generic webpage download level not in extractor.
I don't know how to do, but I would like to see how should this be do. (My Python experience is about 2 weeks)
- It should not use any non public domain code.
on MIT License: https://github.com/Anorov/cloudflare-scrape/blob/master/LICENSE enough credits
- It should not use nodejs.
cloudflare-scrape
perviously use PyExecJS
but switched to node
. You can still think about using phantom
, but this one is also suspended.
@RGFRv2
When will the fix be applied to the new version?
Just wondering...
Thanks
@619beave according to decision @dstftw it won't.
I hope someone else will do it better
@RGFRv2 Hello could you provide a windows binary, or explain how to compile or run your scripts on windows. I have python installed, but when i do "python setup.py" it just errors out with
setup.py:73: UserWarning: Skipping file youtube-dl.bash-completion since it is not present. Type make to build all automatically generated files. warnings.warn('Skipping file %s since it is not present. Type make to build all automatically generated files.' %
fn)``
@dystopia2017 this file is in ./devscripts/
they use https://github.com/rg3/youtube-dl/blob/master/devscripts/buildserver.py but I don't know details (only wget "http://$buildserver/build/rg3/youtube-dl/youtube-dl.exe?rev=$REV" -O youtube-dl.exe
) or https://github.com/rg3/youtube-dl/blob/master/devscripts/wine-py2exe.sh
It's working again after CrunchyRolls Website restart this morning.
I'm curious how long it will work.
I updated to latest windows veriosn 2018.08.04 and now i get this message :
crunchyroll returned error : the CSRF token is invalid. please try to resubmit the form
Same problem as @leprasson. That's when manually specifying the user agent and cookies.txt. Can still download with streamlink but no way to get subtitles that way. anime-dl seems to work fine, though.
Same problem here.
PS. If you are in hurry Xonshiz anime-dl is working right now... Since it's open source too I'm sure youtube-dl will be fixed too soon.
Current error I'm getting now:
[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '-u', 'PRIVATE', '-p', 'PRIVATE', '-f', '[height <=? 1080]', '--sub-lang', 'enUS', '--sub-format', 'ass', '--write-sub', 'http://www.crunchyroll.com/attack-on-titan/episode-40-old-story-776025'] [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 [debug] youtube-dl version 2018.08.04 [debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134 [debug] exe versions: ffmpeg N-83622-gdcd3418, ffprobe N-83622-gdcd3418, rtmpdump 2.3 [debug] Proxy map: {} [Crunchyroll] Logging in [Crunchyroll] 776025: Downloading webpage ERROR: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpckoq891b\build\youtube_dl\YoutubeDL.py", line 792, in extract_info File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpckoq891b\build\youtube_dl\extractor\common.py", line 502, in extract File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpckoq891b\build\youtube_dl\extractor\crunchyroll.py", line 429, in _real_extract youtube_dl.utils.ExtractorError: This is a sample clip. To watch the full video, start a free trial today!; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
@kseniasolo yes, it works for me in Windows, but adding --cookies cookies.txt (previously downloaded)
@N0197exe I know how to use cookies... but how do you convert that string "/usr/bin/python /home/private/youtube_dl/main.py " to windows?
Can someone from the development team implement kseniasolo fix (if it make sense)?
For my: It gave me cloudflare anti boot test on login page.
I tried with --cookies and user-agent and even -u PRIVATE -p PRIVATE: ./youtube-dl.exe -u PRIVATE -p PRIVATE --username myuser --password mypassword http://www.crunchyroll.com/gintama/episode-358-countless-kings-774987 --cookies cookies.txt --user-agent "USER-AGENT" I still get the ERROR: crunchyroll returned error: The CSRF token is invalid. Please try to resubmit the form.
@federicorosso1993. as @bato3 says:
Just install python, and downoad repo: https://github.com/bato3/youtube-dl/archive/crunchroll2.zip
then go to the download path, in my case it is: "C:\Users\Lenovo\Downloads\youtube-dl-crunchroll2\youtube_dl" there you will find "main.py" (will have underscores)
I go to the location with the cmd and do the following example:
main.py -u PRIVATE -p PRIVATE --verbose --cookies cookies.txt --write-sub --sub-format "ass" --sub-lang "enUS" http://www.crunchyroll.com/attack -on-titan / episode-40-old-story-776025
@N0197exe Ah ok, thank you. Still it would be great to get a single .exe file... I created a small UI for my personal use (where I can get url, cookies and user-agent with one click), if I have to use main.py I need to change my code.
@federicorosso1993 Today I'm thinking about:
And you can create bath file like: windows: (*.bat)
cd c:/youtube_dl && python -m youtube_dl %*
linux
cd ~/youtube_dl && python -m youtube_dl "$@"
The biggest difference is the need to use kill-tree
instead simplekill
.
Thanks for the help. The PYthon script is working as:
E:\youtube-dl-crunchroll2\youtube_dl__main__.py -v -u PRIVATE -p PRIVATE -f "[height <=? 1080]" --sub-lang enUS --sub-format "ass" --write-sub "%1"
This is a batch file that is run from another folder where the encodes are muxed together
@bato3 I tried your fix but I got this output instead.
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--geo-bypass-country', 'US', '--username', 'PRIVATE', '--password', 'PRIVATE', '--all-subs', '--verbose', 'https://crunchyroll.com/darling-in-the-franxx']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.07.29
[debug] Python version 3.7.0 (CPython) - Linux-4.17.12-arch1-1-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg n4.0.2, ffprobe n4.0.2, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Using fake IP 3.230.14.11 (US) as X-Forwarded-For.
[crunchyroll:playlist] Logging in
ERROR: Unable to extract login form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/YoutubeDL.py", line 792, in extract_info
ie_result = ie.extract(url)
File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/common.py", line 501, in extract
self.initialize()
File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/common.py", line 405, in initialize
self._real_initialize()
File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/crunchyroll.py", line 120, in _real_initialize
self._login()
File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/crunchyroll.py", line 87, in _login
login_page, 'login form', group='form')
File "/home/jonas/Downloads/youtube-dl-crunchroll2/youtube_dl/extractor/common.py", line 972, in _search_regex
raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract login form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
@freddiaN strange. This error is raised after solving cloudflare challenge.
Please add --write-pages
and attach saved requests (eg paste it in https://pastebin.com/)
And do you have installed node
?
I can reproduce the above error with Python 3.7 too. It seems that reading http.client.HTTPResponse
(from fp
of the urllib.error.HTTPError
raised when making the initial request to https://www.crunchyroll.com/login?next=%2Facct%2Fmembership
) yields nothing. Consequently, has_cf_challenge
gets tested against an empty string (skipping the bulk of @bato3's contribution), and we finally bail when searching for the login token (again against that empty string).
I can't reproduce. I always have some HTML in response.
@puxlit You talk about first login_page
(is empty)? Can you try change fail_url
to _LOGIN_URL
.
And can you store cookies by --cookie
and what cookies is in?. __cfduid
+ c_visitor
- without CF challenge, cf_clearance
- after CF challenge, c_d
, c_userid
, c_userkey
- crunchroll cookies
Yes, login_page
appears to be empty. Changing fail_url
doesn't seem to make a difference. Only __cfduid
and c_visitor
appear to be set.
Your patch works fine on Python 2(.7.15) though; not sure about other versions of Python 3.
Okay, it would seem that HTTPError
closes fp
when the error's destroyed, thanks to bpo-15002 (introduced in Python 3.4.1). Given that's the case, I doubt anything that relies on _request_webpage
's expected_status
functionality will work.
Not that I have anything to add to the discussion, but I also encounter this error with the most recent version of youtube-dl.
@puxlit Can you fix it, will you report it as a new issue?
Still nothing changed ERROR: crunchyroll returned error: The CSRF token is invalid. Please try to resubmit the form.
As a workaround, I've logged in using my web browser, then saved the cookies into a text file. You can then use said cookies to "stay" logged in when you run your command on youtube-dl.
Ok now I know what the main problem is. I was using --username and --password and --cookies and --user-agent. But unlike the first time right now if you use --username and --password it will ignore -cookies and --user-agent. I think it's better if I can use both and if one fail youtube-dl try to use the other system to log in. I say this because if the cookies are too old the download will fail too, so you cannot ignore --username and --password if you use --cookies and --user-agent.
Right now, for my personal UI, I'm using a double download line: ./youtube-dl.exe --username myuser --password mypass http://www.crunchyroll.com/gintama/episode-358-countless-kings-774987 --cookies cookies.txt --user-agent "myagent"; ./youtube-dl.exe http://www.crunchyroll.com/gintama/episode-358-countless-kings-774987 --cookies cookies.txt --user-agent "myagent"
Of course right now the first line always fail, in the future, if the log in will be fixed, it will download the episode two times (so it's not ideal at all)... That's why I ask you if you can fix and keep in mind that "I should be able to use both method at the same time to log in"
Please follow the guide below
x
into all the boxes [ ] relevant to your issue (like this:[x]
)Make sure you are using the latest version: run
youtube-dl --version
and ensure your version is 2018.07.29. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.Before submitting an issue make sure you have:
What is the purpose of your issue?
The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue
If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add the
-v
flag to your command line you run youtube-dl with (youtube-dl -v <your command line>
), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):
Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
Description of your issue, suggested solution and other information
Explanation of your issue in arbitrary form goes here. Please make sure the description is worded well enough to be understood. Provide as much context and examples as possible. If work on your issue requires account credentials please provide them or explain how one can obtain them.