r0oth3x49 / udemy-dl

A cross-platform python based utility to download courses from udemy for personal offline use.
MIT License
4.83k stars 1.19k forks source link

DRM protected video streams are being skipped (udemy-dl not downloading videos) #627

Open nChauhan91 opened 3 years ago

nChauhan91 commented 3 years ago

When i'm trying to download any course any videos that have their URLs starting in blob:https://URL_UDEMY are being skipped.

the lectures are recognized but only assets in that chapter are downloaded. I have all requirements including ffmpeg installed & working in command line.

[i] : Downloading chapter : (2) [i] : Chapter (02 JavaScript Refresher) [i] : Found (13) lecture(s).

[i] : Downloading asset(s) [i] : Downloading (022 next-gen-js-summary.pdf) [+] : Asset : '022 next-gen-js-summary.pdf' (already downloaded)

Lectures were found but not downloaded

Xd43431T commented 3 years ago

Hey Christopher sierra,

How about Udemy business? already used business account and udemy-dl to download courses

Vào Th 4, 12 thg 5, 2021 vào lúc 00:32 christopher sierra < @.***> đã viết:

When someone uses the udemy-dl script and tries to see if the refund is available the following message appears:

Based on your download history, this course is not eligible for a refund. In accordance with our refund policy, we are unable to grant you a refund request for the course because you downloaded it in a way that violates our terms of use. If breaches occur in the future, we may take action on your account.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r0oth3x49/udemy-dl/issues/627#issuecomment-838877411, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATYHO7SUXTJ4QXUQ3QJJNWTTNFS2NANCNFSM423F5HFQ .

Xd43431T commented 3 years ago

image

Hey Christopher sierra,

How about Udemy business? already used business account and udemy-dl to download courses

Ex-Communicad0 commented 3 years ago

@r0oth3x49 check this repo's hope this will help

https://github.com/parnexcodes/widevine-L3-WEB-DL-Script https://github.com/WHTJEON/narrowvine

iampikus commented 3 years ago

@r0oth3x49 check this repo's hope this will help

https://github.com/parnexcodes/widevine-L3-WEB-DL-Script https://github.com/WHTJEON/narrowvine

Thank you for sharing! I'll try use that. =)

UPDATE Just tested narrowvine script and it is not working. yt-dlp is not optimized for udemy. I get error:

[udemy:course] assets: Downloading webpage ERROR: Unable to extract course id; please report this issue on https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see https://github.com/yt-dlp/yt-dlp on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

I think widevine-L3-WEB-DL-Script won't work either, cause it uses yt-dlp

r0oth3x49 commented 3 years ago

Hello! @CHITTIMOTU I couldn’t adapt this script to the udemi mpd file. I don't have enough python knowledge. I have manually downloaded several files (index_video_19_0_init.mp4 + index_video_19_0_1.mp4), concatenated them and successfully decrypted with mp4decrypt. It would be nice if r0oth3x49 could implement an MPD parser to download all segments and combine them into one encrypted file. We will decrypt the resulting files ourselves using mp4decrypt

Thanks @iampikus the solution seems implementable to me, i will test it locally, it will be great help if you can email me a proper steps/solution, i was thinking about using the Bento4 tools, but don't have enough time to research on how i could automate the process that's why asking you send me full solution how you manage to use it manually so that i can try implement it in udemy-dl.

Once again thank you for your great help and research.

email: nasir251292@gmail.com

Puyodead1 commented 3 years ago

Hello! @CHITTIMOTU I couldn’t adapt this script to the udemi mpd file. I don't have enough python knowledge. I have manually downloaded several files (index_video_19_0_init.mp4 + index_video_19_0_1.mp4), concatenated them and successfully decrypted with mp4decrypt. It would be nice if r0oth3x49 could implement an MPD parser to download all segments and combine them into one encrypted file. We will decrypt the resulting files ourselves using mp4decrypt

Thanks @iampikus the solution seems implementable to me, i will test it locally, it will be great help if you can email me a proper steps/solution, i was thinking about using the Bento4 tools, but don't have enough time to research on how i could automate the process that's why asking you send me full solution how you manage to use it manually so that i can try implement it in udemy-dl.

Once again thank you for your great help and research.

email: nasir251292@gmail.com

take a look at this, would this work?
https://github.com/Jayapraveen/Drm-Dash-stream-downloader/blob/master/dashdownloader_multisegment.py#L125

joaobrasil65 commented 3 years ago

Great work guys! Look forward to a solution.

Kaatne-Wala-Kutta commented 3 years ago

@r0oth3x49 bro you got the account? I've one business account and I'm ready to share. Let me know if you need. And btw I'm too getting error. It automatically skips lectures.

Dudeonyx commented 3 years ago

I've paid for well over 20 course on udemy, but with this current policy of preventing downloads which affects genuine customers far more than it affects pirates I'm honestly disappointed.

For now I've given up on the download ever working in a convenient manner so I'll just search for torrents of courses I've bought and download those.

I'll look for a more convenient site to give my money to in the future or failing that, stick with free courses but no piracy.

Puyodead1 commented 3 years ago

This probably won't ever get fixed, it's possible to download DRM encrypted media and decrypt it but it would get a DMCA takedown pretty quickly. Even then, there's still some problems I've noticed since it doesn't look like udemy puts the decryption key IDs in the dash manifests so automation would be slightly more difficult.

joaobrasil65 commented 3 years ago

So we can't download videos anymore?

On Mon, May 17, 2021 at 7:04 PM Puyodead1 @.***> wrote:

This probably won't ever get fixed, it's possible to download DRM encrypted media and decrypt it but it would get a DMCA takedown pretty quickly. Even then, there's still some problems I've noticed since it doesn't look like udemy puts the decryption key IDs in the dash manifests so automation would be more difficult.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r0oth3x49/udemy-dl/issues/627#issuecomment-842697723, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ3XKPIJLP45T2GEXQH673TOGOH7ANCNFSM423F5HFQ .

Puyodead1 commented 3 years ago

Just a quick little bit of information i seem to have found, all the lectures in a course use the same decryption key (per user ofc). So if you get the key from something like a browser extension, you could download all the DRM lectures and decrypt them pretty easy compared to having to get different keys for each video. Probably not all the helpful to most people but felt like its worth sharing.

One possible way to get around legal issues, would be to not provide something that can get the keys, if the program asked the user to get the key themself and provide it to the program, then it should be fine i think, idk.

joaobrasil65 commented 3 years ago

how would i do that exactly? get the key? Could we automate that process if they can figure out how to get the video pieces and put them together?

On Tue, May 18, 2021 at 2:45 PM Puyodead1 @.***> wrote:

Just a quick little bit of information i seem to have found, all the lectures in a course use the same decryption key (per user ofc). So if you get the key from something like a browser extension, you could download all the DRM lectures and decrypt them pretty easy compared to having to get different keys for each video. Probably not all the helpful to most people but felt like its worth sharing.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r0oth3x49/udemy-dl/issues/627#issuecomment-843442632, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ3XKOTSR7DOWVZZWUYQJ3TOKYUBANCNFSM423F5HFQ .

Puyodead1 commented 3 years ago

how would i do that exactly? get the key? Could we automate that process if they can figure out how to get the video pieces and put them together? On Tue, May 18, 2021 at 2:45 PM Puyodead1 @.***> wrote: Just a quick little bit of information i seem to have found, all the lectures in a course use the same decryption key (per user ofc). So if you get the key from something like a browser extension, you could download all the DRM lectures and decrypt them pretty easy compared to having to get different keys for each video. Probably not all the helpful to most people but felt like its worth sharing. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ3XKOTSR7DOWVZZWUYQJ3TOKYUBANCNFSM423F5HFQ .

you would need to use something like a chrome extension that can decrypt the license request (such things have been mentioned in this issues comments). everything could be automated other than getting the keys due to legal issues.

Currently I'm using a modified version of https://github.com/Jayapraveen/Drm-Dash-stream-downloader, with modified code from https://github.com/alastairmccormack/pywvpssh and https://github.com/alastairmccormack/pymp4parse/ for semi-automated downloading of videos.

mahdisky commented 3 years ago

@Puyodead1 can you make a video for do solution step to step ? then upload it in youtube or private.

@r0oth3x49 can you help us ?

Puyodead1 commented 3 years ago

@Puyodead1 can you make a video for do solution step to step ? then upload it in youtube or private.

@r0oth3x49 can you help us ?

I don't think i can/should since it involves bypassing DRM.

atakurt034 commented 3 years ago

@Puyodead1 will you be able to share your current setup on using the https://github.com/Jayapraveen/Drm-Dash-stream-downloader and https://github.com/alastairmccormack/pywvpssh / https://github.com/alastairmccormack/pymp4parse/ ? No need for getting the keys. By the way if you guys still are trying to bypass the DRM using the keys I regret to inform you that by the end of the month it's already going to be updated and won't support the current work around.

Puyodead1 commented 3 years ago

@Puyodead1 will you be able to share your current setup on using the https://github.com/Jayapraveen/Drm-Dash-stream-downloader and https://github.com/alastairmccormack/pywvpssh / https://github.com/alastairmccormack/pymp4parse/ ? No need for getting the keys. By the way if you guys still are trying to bypass the DRM using the keys I regret to inform you that by the end of the month it's already going to be updated and won't support the current work around.

Yeah, the repository is on my GitHub. currently it only supports downloading a single course by the mpd url but im going to add downloading an entire course (drm and non-drm videos).

atakurt034 commented 3 years ago

@Puyodead1 will you be able to share your current setup on using the https://github.com/Jayapraveen/Drm-Dash-stream-downloader and https://github.com/alastairmccormack/pywvpssh / https://github.com/alastairmccormack/pymp4parse/ ? No need for getting the keys. By the way if you guys still are trying to bypass the DRM using the keys I regret to inform you that by the end of the month it's already going to be updated and won't support the current work around.

Yeah, the repository is on my GitHub. currently it only supports downloading a single course by the mpd url but im going to add downloading an entire course (drm and non-drm videos).

Nice. Thanks a lot.

CHITTIMOTU commented 3 years ago

@Puyodead1 will you be able to share your current setup on using the https://github.com/Jayapraveen/Drm-Dash-stream-downloader and https://github.com/alastairmccormack/pywvpssh / https://github.com/alastairmccormack/pymp4parse/ ? No need for getting the keys. By the way if you guys still are trying to bypass the DRM using the keys I regret to inform you that by the end of the month it's already going to be updated and won't support the current work around.

Yeah, the repository is on my GitHub. currently it only supports downloading a single course by the mpd url but im going to add downloading an entire course (drm and non-drm videos).

Nice. Thanks a lot.

I can get the key, but don't know the steps to download, Can you please help us with instructions , how to use your repo https://github.com/Puyodead1/udemy-downloader image

Puyodead1 commented 3 years ago

@Puyodead1 will you be able to share your current setup on using the https://github.com/Jayapraveen/Drm-Dash-stream-downloader and https://github.com/alastairmccormack/pywvpssh / https://github.com/alastairmccormack/pymp4parse/ ? No need for getting the keys. By the way if you guys still are trying to bypass the DRM using the keys I regret to inform you that by the end of the month it's already going to be updated and won't support the current work around.

Yeah, the repository is on my GitHub. currently it only supports downloading a single course by the mpd url but im going to add downloading an entire course (drm and non-drm videos).

Nice. Thanks a lot.

I can get the key, but don't know the steps to download, Can you please help us with instructions , how to use your repo https://github.com/Puyodead1/udemy-downloader image

Make an issue on the repo and i can help you there, i dont want to get too offtopic here.

saurabh9813 commented 3 years ago

Hello guys Just wondering whether we are going to get a workaround to this issue or not? I am also facing the same issue. Videos being skipped while downloading. In few situations, entire course is getting skipped. Ex: https://wipro.udemy.com/course/progressive-web-app-pwa-the-complete-guide/ Plz lets find a fix to this issue at the earliest, or at least a workaround

Thankyou

Puyodead1 commented 3 years ago

@r0oth3x49 I think maybe you have to add "access_token" and "dj_session_id" both to solve the problem. have a try.

that wont magically download and decrypt encrypted videos. Udemy added Widevine DRM to newer videos.

MdTalhaZubayer commented 3 years ago

@r0oth3x49 do you know what is "skd://32 character hex text" is it the AES key? maybe we need to convert it to decimal or binary bit if I convert the 32char hex I'm getting a 128 character binary.

Puyodead1 commented 3 years ago

@r0oth3x49 do you know what is "skd://32 character hex text" is it the AES key? maybe we need to convert it to decimal or binary bit if I convert the 32char hex I'm getting a 128 character binary.

That's related to Apple FairPlay DRM, the skd:// is probably the asset_id but afaik there no way to get keys for fairplay content.

a8uh35h4m commented 3 years ago

I think Udemy has for sure seen this code and decided to implement the DRM so that no one should possibly be able to download anything from their site. But still not losing hope ! I have my subscription till September.

joaobrasil65 commented 3 years ago

Probably. They make it so hard to use offline. Their mobile apps are horrible

On Thu, May 20, 2021, 11:01 PM a8uh35h4m @.***> wrote:

I think Udemy has for sure seen this code and decided to implement the DRM so that no one should possibly be able to download anything from their site. But still not losing hope ! I have my subscription till September.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r0oth3x49/udemy-dl/issues/627#issuecomment-845616733, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ3XKKNYJYNR3GYZC5NSR3TOXEGRANCNFSM423F5HFQ .

Farrukh2019 commented 3 years ago

image Seemingly its not single file , there is a list of files which video player plays. Audio and video are in separated files so i think 2 lists are working here one is list of videos other is list of audio. You have to come up with method to download all videos and audios in list then merge them in single file.

So this blob file contains lists which video player plays probably.

Puyodead1 commented 3 years ago

Yes, It's adaptive streaming, the files are split into segments. There's 2 types: HLS and MPEG-DASH. It's still encrypted with DRM. I already made a program that can download the segments, join them and decrypt them but you need to know the key still.

On Sat, May 22, 2021 at 10:29 AM Farrukh2019 @.***> wrote:

[image: image] https://user-images.githubusercontent.com/55839110/119229984-b89daf00-bb33-11eb-8d47-f54dc15310ce.png Seemingly its not single file , there is a list of files which video player plays. Audio and video are in separated files so i think 2 lists are working here one is list of videos other is list of audio. You have to come up with method to download all videos and audios in list then merge them in single file.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/r0oth3x49/udemy-dl/issues/627#issuecomment-846416029, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADREJXX5BSPRPHPVYSQG2QLTO65UFANCNFSM423F5HFQ .

Farrukh2019 commented 3 years ago

Yes, It's adaptive streaming, the files are split into segments. There's 2 types: HLS and MPEG-DASH. It's still encrypted with DRM. I already made a program that can download the segments, join them and decrypt them but you need to know the key still. On Sat, May 22, 2021 at 10:29 AM Farrukh2019 @.***> wrote: [image: image] https://user-images.githubusercontent.com/55839110/119229984-b89daf00-bb33-11eb-8d47-f54dc15310ce.png Seemingly its not single file , there is a list of files which video player plays. Audio and video are in separated files so i think 2 lists are working here one is list of videos other is list of audio. You have to come up with method to download all videos and audios in list then merge them in single file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADREJXX5BSPRPHPVYSQG2QLTO65UFANCNFSM423F5HFQ .

Key must be some where on the page javascript is open system , since browser is playing it it must be identifyable.

Puyodead1 commented 3 years ago

Yes, It's adaptive streaming, the files are split into segments. There's 2 types: HLS and MPEG-DASH. It's still encrypted with DRM. I already made a program that can download the segments, join them and decrypt them but you need to know the key still. On Sat, May 22, 2021 at 10:29 AM Farrukh2019 @.***> wrote: [image: image] https://user-images.githubusercontent.com/55839110/119229984-b89daf00-bb33-11eb-8d47-f54dc15310ce.png Seemingly its not single file , there is a list of files which video player plays. Audio and video are in separated files so i think 2 lists are working here one is list of videos other is list of audio. You have to come up with method to download all videos and audios in list then merge them in single file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADREJXX5BSPRPHPVYSQG2QLTO65UFANCNFSM423F5HFQ .

Key must be some where on the page javascript is open system , since browser is playing it it must be identifyable.

lol no, thats not how it works. its Widevine DRM, they dont just leave a key laying around. the point of DRM is to prevent people from downloading the videos offline. If you're interested on learning how it works, read into Encrypted Media Extensions.

widevine

Farrukh2019 commented 3 years ago

Yes, It's adaptive streaming, the files are split into segments. There's 2 types: HLS and MPEG-DASH. It's still encrypted with DRM. I already made a program that can download the segments, join them and decrypt them but you need to know the key still. On Sat, May 22, 2021 at 10:29 AM Farrukh2019 @.***> wrote: [image: image] https://user-images.githubusercontent.com/55839110/119229984-b89daf00-bb33-11eb-8d47-f54dc15310ce.png Seemingly its not single file , there is a list of files which video player plays. Audio and video are in separated files so i think 2 lists are working here one is list of videos other is list of audio. You have to come up with method to download all videos and audios in list then merge them in single file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADREJXX5BSPRPHPVYSQG2QLTO65UFANCNFSM423F5HFQ .

Key must be some where on the page javascript is open system , since browser is playing it it must be identifyable.

lol no, thats not how it works. its Widevine DRM, they dont just leave a key laying around. the point of DRM is to prevent people from downloading the videos offline. If you're interested on learning how it works, read into Encrypted Media Extensions

Lol i was just trying to download a course i made a tampermonkey extension which was doing the job but some video were not coming so i decided to try udemy-dl it has same issue.

Puyodead1 commented 3 years ago

Yes, It's adaptive streaming, the files are split into segments. There's 2 types: HLS and MPEG-DASH. It's still encrypted with DRM. I already made a program that can download the segments, join them and decrypt them but you need to know the key still. On Sat, May 22, 2021 at 10:29 AM Farrukh2019 @.***> wrote: [image: image] https://user-images.githubusercontent.com/55839110/119229984-b89daf00-bb33-11eb-8d47-f54dc15310ce.png Seemingly its not single file , there is a list of files which video player plays. Audio and video are in separated files so i think 2 lists are working here one is list of videos other is list of audio. You have to come up with method to download all videos and audios in list then merge them in single file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADREJXX5BSPRPHPVYSQG2QLTO65UFANCNFSM423F5HFQ .

Key must be some where on the page javascript is open system , since browser is playing it it must be identifyable.

lol no, thats not how it works. its Widevine DRM, they dont just leave a key laying around. the point of DRM is to prevent people from downloading the videos offline. If you're interested on learning how it works, read into Encrypted Media Extensions

Lol i was just trying to download a course i made a tampermonkey extension which was doing the job but some video were not coming so i decided to try udemy-dl it has same issue.

Yeah? whats your point exactly? Udemy-DL is broken....

Farrukh2019 commented 3 years ago

My point is that there must be some way to download them and see them with ease , that's what i am finding.

Puyodead1 commented 3 years ago

My point is that there must be some way to download them and see them with ease , that's what i am finding.

the reason some videos work and other dont is that newer videos have DRM, older ones dont. good luck cracking the widevine CDM pal. if it was so easy, everyone would be ripping content from netflix, spotify, hulu, hbo max, Disney+, apple tv and tons of other streaming sites that use Widevine DRM

Puyodead1 commented 3 years ago

You can download the videos but you cant view them because they're encrypted.

Farrukh2019 commented 3 years ago

so how user finds key ?

rijenkii commented 3 years ago

Considering that Widevine L3 decryptors were DMCAd from GitHub by Google, IMHO this issue can be marked as unresolvable.

Puyodead1 commented 3 years ago

so how user finds key ?

by knowing about the extension that intercepts the license requests :) im not giving instructions on it because its considered piracy, do a little research and you might find it

Puyodead1 commented 3 years ago

Considering that Widevine L3 decryptors were DMCAd from GitHub by Google and L2 and L1 decryptors do not exist, IMHO this issue can be marked as unresolvable.

Yeah, i dont see this ever getting a "fix" or a "workaround".

Farrukh2019 commented 3 years ago

i better use screen recording system to save time .... i will check later if issue is solved or not.

Puyodead1 commented 3 years ago

i better use screen recording system to save time .... i will check later if issue is solved or not.

that might work, but iirc the screen blacks out to prevent screen recording

Farrukh2019 commented 3 years ago

i have checked it does not black out.

Farrukh2019 commented 3 years ago

record in HD see many times save bandwidth load.

atakurt034 commented 3 years ago

Is it true that per account you get a specific set of keys or they would vary from each course?

Puyodead1 commented 3 years ago

Is it true that per account you get a specific set of keys or they would vary from each course?

well usually each video should have its own unique key that isnt shared, but udemy seems to share the same key throughout an entire course

MdTalhaZubayer commented 3 years ago

@r0oth3x49 there is master m3u8 that is not encrypted but I contain the encrypted m3u8 files but I don't have any knowledge about the MPD file is it also encrypted?

Puyodead1 commented 3 years ago

@r0oth3x49 there is master m3u8 that is not encrypted but I contain the encrypted m3u8 files but I don't have any knowledge about the MPD file is it also encrypted?

the master m3u8 is just links to other m3u8 playlists of different qualities, which all have encryption as well as the dash mpd files.

joaobrasil65 commented 3 years ago

How do they do it in the Android app? You can download to your device. It sucks but is there something different there?

On Sat, May 22, 2021 at 3:44 PM Puyodead1 @.***> wrote:

@r0oth3x49 https://github.com/r0oth3x49 there is master m3u8 that is not encrypted but I contain the encrypted m3u8 files but I don't have any knowledge about the MPD file is it also encrypted?

The m3u8 playlist files are also encrypted, you most likely downloaded the m3u8 playlist from a video without DRM

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r0oth3x49/udemy-dl/issues/627#issuecomment-846454979, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ3XKIWMPNU7QATEOCO7FDTPACSNANCNFSM423F5HFQ .

Puyodead1 commented 3 years ago

How do they do it in the Android app? You can download to your device. It sucks but is there something different there? On Sat, May 22, 2021 at 3:44 PM Puyodead1 @.***> wrote: @r0oth3x49 https://github.com/r0oth3x49 there is master m3u8 that is not encrypted but I contain the encrypted m3u8 files but I don't have any knowledge about the MPD file is it also encrypted? The m3u8 playlist files are also encrypted, you most likely downloaded the m3u8 playlist from a video without DRM — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ3XKIWMPNU7QATEOCO7FDTPACSNANCNFSM423F5HFQ .

It downloads the encrypted video and requests a license key when you want to watch it, its the same thing as how the netflix app allows you to download videos for offline viewing