search

crunchy-labs / crunchy-cli

👇 Command-line downloader for Crunchyroll
MIT License
606 stars 62 forks source link

Cloudfront 403 Error #355

Closed Cazka closed 8 months ago

Cazka commented 8 months ago

When I try to download an episode, I get the following error:

./crunchy-cli download https://www.crunchyroll.com/de/watch/GJWU294XK/attack-on-titan-final-season-the-final-chapters-special-1
:: √ Logging in
:: √ Parsed url 1
:: √ Loaded series information for url 1
:: Attack on Titan Season 4 (Attack on Titan Final Season (German Dub))
::      1. Erdrumoren » S04ESP1
:: An error occurred: parse error HEAD: Expecting </META> found </HEAD> (https://pl.crunchyroll.com/evs3/25c7dccaa8c6970503b348dc97d006b0/assets/604ffb646f0fc34e3a0f00307d396093_,5265820.mp4,5265821.mp4,5265819.mp4,5265817.mp4,5265818.mp4,.urlset/manifest.mpd?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cCo6Ly9wbC5jcnVuY2h5cm9sbC5jb20vZXZzMy8yNWM3ZGNjYWE4YzY5NzA1MDNiMzQ4ZGM5N2QwMDZiMC9hc3NldHMvNjA0ZmZiNjQ2ZjBmYzM0ZTNhMGYwMDMwN2QzOTYwOTNfLDUyNjU4MjAubXA0LDUyNjU4MjEubXA0LDUyNjU4MTkubXA0LDUyNjU4MTcubXA0LDUyNjU4MTgubXA0LC51cmxzZXQvbWFuaWZlc3QubXBkIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNzEwNTkzNzE5fX19XX0_&Signature=OOreZO8OpWmQi3UfX2zV9yK1bwK~OGQilnAyx18~7BNbDuEjmd0Xm5nmUr6flFwh40VXFGIRqjUwd1W2l-vyEm9DL9coo3pKAz10hDsSui1U4rWxNXctLF5AhI7CVKN9KKnwlgKcs1wx50h2kr6eLIq5s9sSm2tquz6e3bIDTt1OdaSlK9y4mJoNiueiK9V8DPG8bRZdKDDRaoO6~fiWVTC9qGaEbVRWTuOtc37kCpJ0~SPjlXJZP3yIGYFxIL7vC9klfMNJhw0TmGnEF3ZHdkrgyR9ykJbYfBPhKMC2nYKp~QEkyq1U6pH1VoEJiXld2j3t3nv6uZJYB4P6Okz9wA__&Key-Pair-Id=APKAJMWSQ5S7ZB3MF5VA): <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: zHX99stDva3I9Mq3Ila7Icvuy26WYKAZUnmWhHlE9aIkl0HTBa4B1g==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

It was working fine a few days ago.

Client:

satyakiran29 commented 8 months ago

same

CrushedAsian255 commented 8 months ago

I am also getting this error.

CrushedAsian255 commented 8 months ago

I think I worked out why, Crunchyroll updated their server protocol i think, the missing file says something about manifest.mpd, with a format of

https://pl.crunchyroll.com/evs3/[something]/assets/[something]_,[fileset],.urlset/manifest.mpd?Policy=[base64_data]&Signature=[base64_data]&Key-Pair-Id=[something]

now it's in a format of 

https://cr-play-service.prd.crunchyrollsvc.com/v1/manifest/[video_id]/evs3/[something]/assets/p/[something],[fileset],.urlset/manifest.mpd?playbackGuid=[guid]&accountid=[guid]

with a token set in Authorization: Bearer: [base64_data]

Hope this helps!

CrushedAsian255 commented 8 months ago

i also see something about Widevine DRM so that might cause further problems (drm would make it effectively impossible to download from unless there was some exploit/key leak)

samchop commented 8 months ago

yt-dlp tracking this same issue https://github.com/yt-dlp/yt-dlp/issues/9439 Someone in that thread posted that DRM doesn't seem to be an issue based on different script they wrote, so hopefully that's not the case.

Also important to note that they are currently having server issues so there's a chance this won't be resolved until that clears up: https://status.crunchyroll.com/

CrushedAsian255 commented 8 months ago

Wait sorry I’m kinda new to GitHubs what’s the difference between an issue and a pull request?

CrushedAsian255 commented 8 months ago

yt-dlp tracking this same issue yt-dlp/yt-dlp#9439

There seems to be a workaround that was made on that issue, it might help with this project

garouthe commented 8 months ago

same error

Spoonanator commented 8 months ago

es broke for me too.

kerkmann commented 8 months ago

@TheAznCoderPro You don't need to "exploit" or "have a key leak", you could easily sniff the key with a MITM on a rooted android. ^^"

@samchop The server status went well again, but the problem still exists. I don't think it's server related. :)

samchop commented 8 months ago

Wait sorry I’m kinda new to GitHubs what’s the difference between an issue and a pull request?

@TheAznCoderPro An issue is generally when someone is having an "issue" with the code, haha. Like they've found a bug/error, or sometimes they're used to ask the devs to add new features. A pull request is when someone has written new code or changes they would like to contribute to the project. A ticket is opened and the code is reviewed before deciding if it should be merged with the project.

@kerkmann dang, I was sure hoping this would fix itself haha

CrushedAsian255 commented 8 months ago

@kerkmann dang, I was sure hoping this would fix itself haha

I think the downloading logic might have to be rewritten slightly to use a different download algorithm, might have to wait for devs unless someone can 'pull request' something.

CrushedAsian255 commented 8 months ago

I think it could be the end See https://github.com/yt-dlp/yt-dlp/issues/9453

hazy-kun commented 8 months ago

You plan to add support to WV stream using self owned l3 file?

CrushedAsian255 commented 8 months ago

You plan to add support to WV stream using self owned l3 file?

what's an i3 file? decryption key?

bytedream commented 8 months ago

I've tested all endpoints I know of and none has working drm-free streams anymore. If Crunchyroll keeps this change, downloading/decrypting with an L3 file/decryption key as @hazy-kun suggested will be the only way to go

bytedream commented 8 months ago

You plan to add support to WV stream using self owned l3 file?

what's an i3 file? decryption key?

@TheAznCoderPro yes it is

KingGeorgeDCM commented 8 months ago

now what are the assumable ways we can do to download

bytedream commented 8 months ago

now what are the assumable ways we can do to download

Currently nothing. I will evaluate the impact of the issue and see if I can add something to bypass it later today

CrushedAsian255 commented 8 months ago

You plan to add support to WV stream using self owned l3 file?

what's an i3 file? decryption key?

@TheAznCoderPro yes it is

how do you get i3 file?

KingGeorgeDCM commented 8 months ago

now what are the assumable ways we can do to download

Currently nothing. I will evaluate the impact of the issue and see if I can add something to bypass it later today

Please do it, we will be grateful to you (thats all I can say hehe)

hazy-kun commented 8 months ago

I've tested all endpoints I know of and none has working drm-free streams anymore. If Crunchyroll keeps this change, downloading/decrypting with an L3 file/decryption key as @hazy-kun suggested will be the only way to go

The best will be switch to Android endpoint, MPD and self-provided L3 (from user not you) as a lot of others teams does but it's just a hint, I'm just here to help

Burve commented 8 months ago

And some hints about getting L3 from yourself would be appriciated.

garouthe commented 8 months ago

[Crunchyroll-Downloader-v3.0] is fixed for now

https://github.com/hama3254/Crunchyroll-Downloader-v3.0/issues/911 check if fix works for you

tamodolo commented 8 months ago

[Crunchyroll-Downloader-v3.0] is fixed for now

hama3254/Crunchyroll-Downloader-v3.0#911 check if fix works for you

It's broken again.

KingGeorgeDCM commented 8 months ago

I've tested all endpoints I know of and none has working drm-free streams anymore. If Crunchyroll keeps this change, downloading/decrypting with an L3 file/decryption key as @hazy-kun suggested will be the only way to go

TPD has made a mark on this Crunchy DL and it is working fine as per users, I guess you need to have a look at implementation of his version and that would definitely crunchy-cli take its mark again!

napgiver commented 8 months ago

[Crunchyroll-Downloader-v3.0] is fixed for now hama3254/Crunchyroll-Downloader-v3.0#911 check if fix works for you

It's broken again.

The END of the Crunchyroll-Downloader-v3.0!

DRM

tamodolo commented 8 months ago

[Crunchyroll-Downloader-v3.0] is fixed for now hama3254/Crunchyroll-Downloader-v3.0#911 check if fix works for you

It's broken again.

The END of the Crunchyroll-Downloader-v3.0!

DRM

there's 2 paid softwares that still can download cr. streamfab and cleverget. If you're clever you'll find the means for free.

W3bP3n commented 8 months ago

[Crunchyroll-Downloader-v3.0] is fixed for now hama3254/Crunchyroll-Downloader-v3.0#911 check if fix works for you

It's broken again.

The END of the Crunchyroll-Downloader-v3.0! DRM

there's 2 paid softwares that still can download cr. streamfab and cleverget. If you're clever you'll find the means for free.

may i know which softwares you meationed?

tamodolo commented 8 months ago

[Crunchyroll-Downloader-v3.0] is fixed for now hama3254/Crunchyroll-Downloader-v3.0#911 check if fix works for you

It's broken again.

The END of the Crunchyroll-Downloader-v3.0! DRM

there's 2 paid softwares that still can download cr. streamfab and cleverget. If you're clever you'll find the means for free.

may i know which softwares you meationed?

oh... well... this is weird as I already mentioned them...

Morgul commented 8 months ago

Other downloaders are working on it:

https://github.com/anidl/multi-downloader-nx/issues/588

(I've got the mpd branch working locally for me.) It seems like this will be the only path forward, which is fine, but has the downside of not being able to be a 'batteries included' solution. Linking to information about how to get the required keys dumped might also be an issue, so it because a bit of a burden on the user. That said, it wasn't even that hard with the tutorial I found.

tamodolo commented 8 months ago

Other downloaders are working on it:

anidl/multi-downloader-nx#588

(I've got the mpd branch working locally for me.) It seems like this will be the only path forward, which is fine, but has the downside of not being able to be a 'batteries included' solution. Linking to information about how to get the required keys dumped might also be an issue, so it because a bit of a burden on the user. That said, it wasn't even that hard with the tutorial I found.

nah, you just need to tell what to get and where to put (folder). How to get will depend of research skills of the user. It's 5 minutes until a good tutorial shows on google. If the user can't do that then it's for the better (both for devs and user)

CrushedAsian255 commented 8 months ago

It's 5 minutes until a good tutorial shows on google.

I found a bunch of tutorials for Android and Windows users, but couldn't find anything for MacOS and iPhone, should i just spin up a windows vm and dump keys from that?

daviddanielng commented 8 months ago

It's 5 minutes until a good tutorial shows on google.

I found a bunch of tutorials for Android and Windows users, but couldn't find anything for MacOS and iPhone, should i just spin up a windows vm and dump keys from that?

There is tutorial for Android and Windows only for a reason. I don't think it would be possible to extract CDM from any apple device. Instead download android studio and follow this tutorial

CrushedAsian255 commented 8 months ago

It's 5 minutes until a good tutorial shows on google.

I found a bunch of tutorials for Android and Windows users, but couldn't find anything for MacOS and iPhone, should i just spin up a windows vm and dump keys from that?

There is tutorial for Android and Windows only for a reason. I don't think it would be possible to extract CDM from any apple device. Instead download android studio and follow this tutorial

Thanks for the tutorial link!

CrushedAsian255 commented 8 months ago

It's 5 minutes until a good tutorial shows on google.

I found a bunch of tutorials for Android and Windows users, but couldn't find anything for MacOS and iPhone, should i just spin up a windows vm and dump keys from that?

There is tutorial for Android and Windows only for a reason. I don't think it would be possible to extract CDM from any apple device. Instead download android studio and follow this tutorial

got my keys, just wondering how long do they last?

daviddanielng commented 8 months ago

It's 5 minutes until a good tutorial shows on google.

I found a bunch of tutorials for Android and Windows users, but couldn't find anything for MacOS and iPhone, should i just spin up a windows vm and dump keys from that?

There is tutorial for Android and Windows only for a reason. I don't think it would be possible to extract CDM from any apple device. Instead download android studio and follow this tutorial

got my keys, just wondering how long do they last

They are unique to device, they should last for some time or utill Google revokes it.

CrushedAsian255 commented 8 months ago

utill Google revokes it.

if i don't share the key is the chance of that pretty low? do they have some kind of key rate limit?

daviddanielng commented 8 months ago

if i don't share the key is the chance of that pretty low?

Yes , the fewer people that knows about the key been used illegally, the lower the chances of Google revolving it.

do they have some kind of key rate limit?

Knowing Google, there will be a rate limit or some AI watching the activities of each key.

tamodolo commented 8 months ago

utill Google revokes it.

if i don't share the key is the chance of that pretty low? do they have some kind of key rate limit?

Do not share your keys. Do you share the keys to your house? Do not do that. You'll lose control over it therefore the key itself.

Burve commented 8 months ago

Quick question about the keys - I tried Pixel 6 with Pie, but CR there said that Chrome is too old. What Device/OS combination worked?

KoffeinKaio commented 8 months ago

Quick question about the keys - I tried Pixel 6 with Pie, but CR there said that Chrome is too old. What Device/OS combination worked?

at which step? browsing to a drm site to dump the keys?

use https://bitmovin.com/demos/drm

Burve commented 8 months ago

Hmm, oh, keys are per device and not site ...

CrushedAsian255 commented 8 months ago

The blob thing is the .bin right? And the private key is the json? Am I doing that right?

KoffeinKaio commented 8 months ago

The blob thing is the .bin right? And the private key is the json? Am I doing that right?

private_key.pem -> device_private_key

client_id.bin -> device_client_id_blob

CrushedAsian255 commented 8 months ago

The blob thing is the .bin right? And the private key is the json? Am I doing that right?

private_key.pem -> device_private_key

client_id.bin -> device_client_id_blob

my .pem is a json is that a problem?

samchop commented 8 months ago

multi-downloader-nx is working for me on both mac and linux, but have to say I still miss crunchy-cli. I'm not sure if this was something effected by the recent changes but downloading with CC was much faster than MDN (even when I don't include the time it takes to decrypt). it would be amazing if there's a way to use it with our own cdm.

CrushedAsian255 commented 8 months ago

multi-downloader-nx is working for me on both mac and linux, but have to say I still miss crunchy-cli. I'm not sure if this was something effected by the recent changes but downloading with CC was much faster than MDN (even when I don't include the time it takes to decrypt). it would be amazing if there's a way to use it with our own cdm.

i tried to use it but it gave me a weird error, what install did u use?

samchop commented 8 months ago

On mac I got weird issues at first using the GUI so now im using the latest mac os cli on the release page Make sure you do the following:

Carloswms10 commented 8 months ago

It's a shame, it seems there's still no solution for this DRM issue. I even used an automated script to make the download of new episodes automatic haha. I hope the developers can find a workaround for this problem. I would even pay a reasonable amount (within my budget) monthly to keep this tool functional. I use StreamFab, but it doesn't compare to our beloved crunchy-cli. The real charm is in automation, and crunchy-cli is a thousand times more powerful. I'm eagerly awaiting an update that can bypass this issue.