search

jaw20 / Crunchyroll-XML-Decoder

GNU General Public License v2.0
37 stars 14 forks source link

Problems starting the download #62

Open halvella opened 5 years ago

halvella commented 5 years ago 
python version=2.7.9 64 bit
OS Version=Windows 8 6.2.9200
System Type=AMD64
Crypto installed
lxml installed
Cfscrape installed
User Name=XXXXXXXXX
Membership Type=Premium Member
Options:
0.- Exit
1.- Download Anime
2.- Download Subtitle only
3.- Login
4.- Login As Guest
5.- Download an entire Anime(Autocatch links)
6.- Run Queue
7.- Settings

> 3
Username: XXXXXXXXX
Password(don't worry the password are typing but hidden:
Login as Premium Member successfully.
Options:
0.- Exit
1.- Download Anime
2.- Download Subtitle only
3.- Login
4.- Login As Guest
5.- Download an entire Anime(Autocatch links)
6.- Run Queue
7.- Settings

> 1
Please enter Crunchyroll video URL:
https://www.crunchyroll.com/pt-br/the-rising-of-the-shield-hero/episode-8-curse-shield-781155

--------------------------
---- Start New Export ----
--------------------------

CrunchyRoll Downloader Toolkit DX v0.98b

Crunchyroll hasn't changed anything.

If you don't have a premium account, go and sign up for one now. It's well worth it, and supports the animators.

----------
Booting up...

Traceback (most recent call last):
  File "C:\Users\XXXXXXXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 369, in <module>
    makechoise()
  File "C:\Users\XXXXXXXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 309, in makechoise
    makechoise()
  File "C:\Users\XXXXXXXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 302, in makechoise
    ultimate.ultimate(raw_input('Please enter Crunchyroll video URL:\n'), '', '')
  File "crunchy-xml-decoder\ultimate.py", line 258, in ultimate
    vid_id = xmlconfig.find('media_id').string
AttributeError: 'NoneType' object has no attribute 'string'
Pressione qualquer tecla para continuar. . .
darkhelmet2016 commented 5 years ago

Not sure if it is related but they have changed their site header again and the program might not have been updated to reflect the new info that needs to be filtered out while pulling the url since it is set up to remove the parts of the page like watching anime on crunchyroll. The Rising of the Shield Hero Episode 8 - Watch on Crunchyroll is what it would be looking for in the data that does not exist since it is using the old info. But might be wrong on all of this.

halvella commented 5 years ago

Not sure if it is related but they have changed their site header again and the program might not have been updated to reflect the new info that needs to be filtered out while pulling the url since it is set up to remove the parts of the page like watching anime on crunchyroll. The Rising of the Shield Hero Episode 8 - Watch on Crunchyroll is what it would be looking for in the data that does not exist since it is using the old info. But might be wrong on all of this.

How could I solve the problem?

darkhelmet2016 commented 5 years ago

Can not guarantee this might help but try checking this line in the ultimate.py title = re.findall('(.+?)', html)[0].replace to see what it has. Try editing it to what they currently have and see if that helps. If not then something was changed in the data that it is looking for that either got moved or may not exist anymore, but what it is looking for would have to do some checking.

halvella commented 5 years ago

Can not guarantee this might help but try checking this line in the ultimate.py title = re.findall('(.+?)', html)[0].replace to see what it has. Try editing it to what they currently have and see if that helps. If not then something was changed in the data that it is looking for that either got moved or may not exist anymore, but what it is looking for would have to do some checking.

I do not know if I did it correctly. But it did not work... Thank you anyway!

darkhelmet2016 commented 5 years ago

Also found this in regards to the one error. https://python-forum.io/Thread-AttributeError-NoneType-object-has-no-attribute-error

darkhelmet2016 commented 5 years ago

Looking at your error I have an idea what is wrong but someone needs to update the login for premium accounts, when was your most recent log in to the account? The line being referred to in your error is for logging in. If you have not logged back in for a while it could be using the old authentication. If you clear the cookie data and it gives you an error about trying to log in then it is an authentication error.

halvella commented 5 years ago

Looking at your error I have an idea what is wrong but someone needs to update the login for premium accounts, when was your most recent log in to the account? The line being referred to in your error is for logging in. If you have not logged back in for a while it could be using the old authentication. If you clear the cookie data and it gives you an error about trying to log in then it is an authentication error.

I deleted the cookie file. After trying to log it keeps crashing. I already tried changing the password on the site and nothing works!

python version=2.7.1 64 bit
OS Version=Windows 10 10.0.17134
System Type=AMD64
Crypto installed
lxml installed
Cfscrape installed
Do you have an account [Y/N]?y
Username: XXXXXXX
Password(don't worry the password are typing but hidden:
Login failed.
Pressione qualquer tecla para continuar. . .
darkhelmet2016 commented 5 years ago

Yea, they have changed the log in credentials which means no more premium downloads until it gets fixed but they have also changed internal urls also since they merged with vrv which means all the old data strings it was searching for are hosed.

halvella commented 5 years ago

Yea, they have changed the log in credentials which means no more premium downloads until it gets fixed but they have also changed internal urls also since they merged with vrv which means all the old data strings it was searching for are hosed.

I do not quite understand what you mean (I'm using Google translator). So will I have to wait for an update of Crunchyroll-XML-Decoder or look for another program or code?

darkhelmet2016 commented 5 years ago

You can wait but it has been months, might be time to look for a new one or just get them from torrents.

Elegond commented 5 years ago

i think the problem is the http header that is used to query the xml data. if i print xmlconfig directly after creating in ultimate.py i get the cloudflare check page instead of the expected xml data. i think if we replace the header in altfuncs.py in line 110 with a new one that is not checked by cloudflare the problem could be fixed.

darkhelmet2016 commented 5 years ago

It is not just a cloud flare check, the authentication string has changed which is what the make choice lines are referring to, line line 369, in makechoise()is referring to the crunchy-xml-decoder.pyi file for the log in. It will not create a new cookie if it can no find the correct log in which is why it is making the error log in failed. Line 110 in altfuncs is for the flash video verification to make sure it is pulling the current version of flash player.

AhmadKahlid commented 5 years ago

The problem because cloudflare check, just replace the following file to fix it. altfuncs.zip

halvella commented 5 years ago

The problem because cloudflare check, just replace the following file to fix it. altfuncs.zip

The download of single episodes has returned to normal. The seasons download still does not work. And for some reason the .mkv files are named "Just a moment....mkv"

nigoki2 commented 5 years ago

@AhmadKahlid Now I have the "CSRF token not found" error mesage.

darkhelmet2016 commented 5 years ago

Told you was not a cloud flare issue. It is not generating the proper log in. It needs to create the proper cookies and token. It is missing header data as it has changed.

def login(username, password): headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0', 'Connection': 'keep-alive'} session = cfscrape.create_scraper() res_get = session.get('https://www.crunchyroll.com/login', headers=headers)

s = re.search('name="login_form\\[_token\\]" value="([^"]*)"', res_get.text)
if s is None:
   print 'CSRF token not found'
   sys.exit()
token = s.group(1)

payload = {'login_form[redirect_url]': '/',
           'login_form[name]': username,
           'login_form[password]': password,
           'login_form[_token]': token}

res_post = session.post('https://www.crunchyroll.com/login', data=payload, headers=headers, allow_redirects = False)
if not (res_post.status_code == 302 or (res_post.status_code == 200 and username == '')):
  print 'Login failed'
  sys.exit()
darkhelmet2016 commented 5 years ago

Not sure if this will help but you can find the code page for the log in here. view-source:https://www.crunchyroll.com/login

darkhelmet2016 commented 5 years ago

You need to verify you are automatized do download anything beyond 480p as those are for premium users unless it stays that a show or episode will be unlocked at a higher res for non paying customers. The token is basically your key to that encrypted data unless you make a script to bypass it.

Elegond commented 5 years ago

for me the login works again if i set the c_locale cookie in login.py and the naming problem is solved by overwriting the title with the title from the xml answer. I also need the fix form AhmadKahlid in altfunktions.py to make everything work crunchy-xml-decoder.zip

halvella commented 5 years ago

for me the login works again if i set the c_locale cookie in login.py and the naming problem is solved by overwriting the title with the title from the xml answer. I also need the fix form AhmadKahlid in altfunktions.py to make everything work crunchy-xml-decoder.zip

The seasons download still does not work. It seems that this problem is general with all projects

Elegond commented 5 years ago

i have briefly rewritten the autocatch method to crunchy-xml-decoder.py and put all the changes into this zip

crunchy-xml-decoder2.zip

def autocatch():
    print 'indicate the url : '
    url=raw_input()
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0',
               'Connection': 'keep-alive'}
    session = cfscrape.create_scraper()
    mykey = session.get(url, headers=headers)
    take = open("queue_.txt", "w")
    asdasd= BeautifulSoup(mykey.content, 'html.parser')

    for text in asdasd.find_all('a'):
        if "episode" in text.get('href'):
            print text.get('href')
            print >> take, 'http://www.crunchyroll.com/'+text.get('href')

    take.close()

    with open('queue_.txt') as f,  open('queue.txt', 'w') as fout:
        fout.writelines(reversed(f.readlines()))
    os.remove('queue_.txt')
ErickAguilera commented 5 years ago

I'm getting a different error:

> python version=2.7.9 32 bit
OS Version=Windows 8 6.2.9200
System Type=AMD64
Crypto installed
lxml installed
Cfscrape installed
User Name=Guest
Membership Type=Guest
Options:
0.- Exit
1.- Download Anime
2.- Download Subtitle only
3.- Login
4.- Login As Guest
5.- Download an entire Anime(Autocatch links)
6.- Run Queue
7.- Settings
> 3
Username: XXXXXXX
Password(don't worry the password are typing but hidden:
Traceback (most recent call last):
  File "C:\Users\xxx\xxy\xxz\Crunchyroll-XML-Decoder-master (old1)\crunchy-xml-decoder.py", line 369, in <module>
    makechoise()
  File "C:\Users\xxx\xxy\xxz\Crunchyroll-XML-Decoder-master (old1)\crunchy-xml-decoder.py", line 308, in makechoise
    login.login(username, password)
  File "crunchy-xml-decoder\login.py", line 37, in login
    res_get = session.get('https://www.crunchyroll.com/login', headers=headers)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "build\bdist.win32\egg\cfscrape\__init__.py", line 66, in request
  File "build\bdist.win32\egg\cfscrape\__init__.py", line 97, in solve_cf_challenge
  File "build\bdist.win32\egg\cfscrape\__init__.py", line 127, in solve_challenge
ValueError: Error parsing Cloudflare IUAM Javascript challenge. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."

Tried downloading everything again and replacing with the .zips chared above.

Elegond commented 5 years ago

you should try to rebuild and reinstall your version of cloudflare-scrape https://github.com/Anorov/cloudflare-scrape/archive/master.zip and run: setup.py build setup.py install

Virusf commented 5 years ago

here is a change for Subtitle only based on the latest changes decode.zip

nigoki2 commented 5 years ago

@Elegond & @Virusf Thank you! It works again.

FacelessFresh commented 5 years ago

Yeah, I'm not sure, apparently I messed something up so I may just re-download the whole crunchyroll xml decoder and try to edit a fresh install and see which way it goes from there. I tried downloading an older video to see if I could get it to do anything with no luck still. the error it gives is different as well.

python version=2.7.9 32 bit OS Version=Windows 8 6.2.9200 System Type=AMD64 Crypto installed lxml installed Cfscrape installed User Name=Guest Membership Type=Guest Options: 0.- Exit 1.- Download Anime 2.- Download Subtitle only 3.- Login 4.- Login As Guest 5.- Download an entire Anime(Autocatch links) 6.- Run Queue 7.- Settings

1 Please enter Crunchyroll video URL: https://www.crunchyroll.com/wagnaria/episode-1-welcome-to-wagnaria-takanashi-goes-to-work-585582

---- Start New Export ----

CrunchyRoll Downloader Toolkit DX v0.98d

Crunchyroll hasn't changed anything.

If you don't have a premium account, go and sign up for one now. It's well worth it, and supports the animators.

Booting up...

Traceback (most recent call last): File "Q:\A\crunchy-xml-decoder.py", line 377, in makechoise() File "Q:\A\crunchy-xml-decoder.py", line 310, in makechoise ultimate.ultimate(raw_input('Please enter Crunchyroll video URL:\n'), '', '') File "crunchy-xml-decoder\ultimate.py", line 249, in ultimate xmlconfig = BeautifulSoup(altfuncs.getxml('RpcApiVideoPlayer_GetStandardConfig', media_id), 'xml') File "crunchy-xml-decoder\altfuncs.py", line 101, in getxml res_get = session.get('https://www.crunchyroll.com/login', headers=headers) File "C:\Python27\lib\site-packages\requests\sessions.py", line 546, in get return self.request('GET', url, kwargs) File "Q:\A\cfscrape__init__.py", line 68, in request resp = self.solve_cf_challenge(resp, kwargs) File "Q:\A\cfscrape__init.py", line 96, in solve_cf_challenge params["jschl_answer"] = self.solve_challenge(body, domain) File "Q:\A\cfscrape\init__.py", line 140, in solve_challenge raise EnvironmentError("Missing Node.js runtime. Node is required and must be in the PATH (check with node -v). Your Node binary may be called nodejs rather than node, in which case you may need to run apt-get install nodejs-legacy on some Debian-based systems. (Please read the cfscrape" EnvironmentError: Missing Node.js runtime. Node is required and must be in the PATH (check with node -v). Your Node binary may be called nodejs rather than node, in which case you may need to run apt-get install nodejs-legacy on some Debian-based systems. (Please read the cfscrape README's Dependencies section: https://github.com/Anorov/cloudflare-scrape#dependencies. Press any key to continue . . .

Elegond commented 5 years ago

install https://nodejs.org/dist/v10.15.2/node-v10.15.2-x64.msi and reboot

FacelessFresh commented 5 years ago

Thanks, It's working now. I much appreciate the assistance everyone of all those that know how to fix these things gives to the ones of us that don't really understand it.

reikitsune commented 5 years ago

I upgrade to your modded build @Elegond and it worked fine. However, I used it yesterday, and it started giving me this error.

python version=2.7.9 64 bit

---- Start New Export ----

CrunchyRoll Downloader Toolkit DX v0.98b

Crunchyroll hasn't changed anything.

If you don't have a premium account, go and sign up for one now. It's well worth it, and supports the animators.

Booting up...

Traceback (most recent call last): File "C:\Users\Public\Crunchyroll-XML-Decoder_link\crunchy-xml-decoder.py", line 369, in makechoise() File "C:\Users\Public\Crunchyroll-XML-Decoder_link\crunchy-xml-decoder.py", line 302, in makechoise ultimate.ultimate(raw_input('Please enter Crunchyroll video URL:\n'), '', '') File "crunchy-xml-decoder\ultimate.py", line 258, in ultimate vid_id = xmlconfig.find('media_id').string AttributeError: 'NoneType' object has no attribute 'string' Press any key to continue . . .

By chance you might know what could have caused it?

darkhelmet2016 commented 5 years ago

Is this program even updated to be compatible with that version of python? Some versions will give errors especially the later builds from what this was complied off of.

reikitsune commented 5 years ago

Is this program even updated to be compatible with that version of python? Some versions will give errors especially the later builds from what this was complied off of.

It is. At least I've used for over a 2 months without any problems. I didn't have any problems til the recent changes to Crunchyroll's site itself. I downloaded a clean version of the Decoder, then copied and pasted Elegond's changes into it. Worked til yesterday. I could give downgrading Python a try.

halvella commented 5 years ago

Stopped running for two days

python version=2.7.1 64 bit
OS Version=Windows 10 10.0.17134
System Type=AMD64
Crypto installed
lxml installed
Cfscrape installed
User Name=XXXXX
Membership Type=Premium Member
Options:
0.- Exit
1.- Download Anime
2.- Download Subtitle only
3.- Login
4.- Login As Guest
5.- Download an entire Anime(Autocatch links)
6.- Run Queue
7.- Settings

> 1
Please enter Crunchyroll video URL:
https://www.crunchyroll.com/pt-br/black-clover/episode-74-flower-of-resolution-780902

--------------------------
---- Start New Export ----
--------------------------

CrunchyRoll Downloader Toolkit DX v0.98d

Crunchyroll hasn't changed anything.

If you don't have a premium account, go and sign up for one now. It's well worth it, and supports the animators.

----------
Booting up...

Traceback (most recent call last):
  File "C:\Users\XXXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 377, in <module>
    makechoise()
  File "C:\Users\XXXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 310, in makechoise
    ultimate.ultimate(raw_input('Please enter Crunchyroll video URL:\n'), '', '')
  File "crunchy-xml-decoder\ultimate.py", line 269, in ultimate
    print xmlconfig.find('exclusive').string.replace('Stream','Download') + ' in ' +  xmlconfig.find('height').string + 'p'
TypeError: coercing to Unicode: need string or buffer, NoneType found
Pressione qualquer tecla para continuar. . .
Elegond commented 5 years ago

as i wrote in problem #63 crunchyroll stopped sending the resolution in xml yesterday

remove + ' in ' + xmlconfig.find('height').string + 'p' in ultimate.py line 269 and downloads in 1080p will work again

crunchy-xml-decoder.zip

halvella commented 5 years ago

Has this error started now, crunchyroll changed anything again?

python version=2.7.1 64 bit
OS Version=Windows 10 10.0.17134
System Type=AMD64
Crypto installed
lxml installed
Cfscrape installed
User Name=XXXX
Membership Type=Premium Member
Options:
0.- Exit
1.- Download Anime
2.- Download Subtitle only
3.- Login
4.- Login As Guest
5.- Download an entire Anime(Autocatch links)
6.- Run Queue
7.- Settings

> 1
Please enter Crunchyroll video URL:
https://www.crunchyroll.com/pt-br/that-time-i-got-reincarnated-as-a-slime/episode-245-veldoras-journal-780742

--------------------------
---- Start New Export ----
--------------------------

CrunchyRoll Downloader Toolkit DX v0.98d

Crunchyroll hasn't changed anything.

If you don't have a premium account, go and sign up for one now. It's well worth it, and supports the animators.

----------
Booting up...

Traceback (most recent call last):
  File "C:\Users\XXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 377, in <module>
    makechoise()
  File "C:\Users\XXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 310, in makechoise
    ultimate.ultimate(raw_input('Please enter Crunchyroll video URL:\n'), '', '')
  File "crunchy-xml-decoder\ultimate.py", line 265, in ultimate
    if int(xmlconfig.find('episode_number').string) < 10:
ValueError: invalid literal for int() with base 10: '24.5'
Pressione qualquer tecla para continuar. . .
Elegond commented 5 years ago

https://github.com/Elegond/Crunchyroll-XML-Decoder/blob/master/crunchy-xml-decoder/ultimate.py Change line 265if int(xmlconfig.find('episode_number').string) < 10: to if float(xmlconfig.find('episode_number').string) < 10: in ultimate.py

nigoki2 commented 5 years ago

I still have the same error after editing ultimate.py.

Elegond commented 5 years ago

if you edit ultimate.py like this, it is impossible to get the same error. Replace your ultimate.py with https://github.com/Elegond/Crunchyroll-XML-Decoder/blob/master/crunchy-xml-decoder/ultimate.py or post your file and error tracking.

nigoki2 commented 5 years ago

Indeed it works after replacing the file with yours. The last I copied and past. Sorry for my mistake nand thank you for your help.

halvella commented 5 years ago

New error

python version=2.7.1 64 bit
OS Version=Windows 10 10.0.17134
System Type=AMD64
Crypto installed
lxml installed
Cfscrape installed
User Name=XXXX
Membership Type=Premium Member
Options:
0.- Exit
1.- Download Anime
2.- Download Subtitle only
3.- Login
4.- Login As Guest
5.- Download an entire Anime(Autocatch links)
6.- Run Queue
7.- Settings

> 1
Please enter Crunchyroll video URL:
https://www.crunchyroll.com/pt-br/my-sister-my-writer/my-sister-my-writer-ova-eu-minha-irm-e-o-ciberespao-parte-2-777917

--------------------------
---- Start New Export ----
--------------------------

CrunchyRoll Downloader Toolkit DX v0.98d

Crunchyroll hasn't changed anything.

If you don't have a premium account, go and sign up for one now. It's well worth it, and supports the animators.

----------
Booting up...

Traceback (most recent call last):
  File "C:\Users\XXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 377, in <module>
    makechoise()
  File "C:\Users\XXXX\Desktop\Crunchyroll-XML-Decoder-master\crunchy-xml-decoder.py", line 310, in makechoise
    ultimate.ultimate(raw_input('Please enter Crunchyroll video URL:\n'), '', '')
  File "crunchy-xml-decoder\ultimate.py", line 265, in ultimate
    if float(xmlconfig.find('episode_number').string) < 10:
ValueError: invalid literal for float(): 11b
Pressione qualquer tecla para continuar. . .
Elegond commented 5 years ago

ok in this case delete lines 265 to 267. i didn't expect any episodes with letters. https://github.com/Elegond/Crunchyroll-XML-Decoder/blob/master/crunchy-xml-decoder/ultimate.py

AlcorSalvador commented 5 years ago

I believe they are somehow tracking this now, because I can use the exact same files with a different account and download just 1 episode before it throws XML errors.

Update: for some reason if you audit the site, you can download again

Elegond commented 5 years ago

can you show me your error traceback?

alzamer2 commented 5 years ago

can you try this code https://github.com/alzamer2/Crunchyroll-XML-Decoder-py3 i re-write it to use python3.3

Jroddmqz commented 5 years ago

Error found. `CrunchyRoll Downloader Toolkit DX v0.98b

Crunchyroll hasn't changed anything.

If you don't have a premium account, go and sign up for one now. It's well worth it, and supports the animators.

Booting up...

Traceback (most recent call last): File "C:\Users\Public\Crunchyroll-XML-Decoder_link\crunchy-xml-decoder.py", line 369, in makechoise() File "C:\Users\Public\Crunchyroll-XML-Decoder_link\crunchy-xml-decoder.py", line 302, in makechoise ultimate.ultimate(raw_input('Please enter Crunchyroll video URL:\n'), '', '') File "crunchy-xml-decoder\ultimate.py", line 215, in ultimate page_url = altfuncs.vidurl(page_url, False, False) File "crunchy-xml-decoder\altfuncs.py", line 119, in vidurl res = gethtml(url) File "crunchy-xml-decoder\altfuncs.py", line 64, in gethtml if not forceusa and localizecookies: NameError: global name 'forceusa' is not defined`

alzamer2 commented 5 years ago

@Jroddmqz can you try this code https://github.com/alzamer2/Crunchyroll-XML-Decoder-py3 i re-write it to use python3.3