search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
129.77k stars 9.79k forks source link

Unable to download webpage: HTTP Error 403: Forbidden - Udemy (video page) #29705

Open qrrabbit opened 2 years ago

qrrabbit commented 2 years ago

Checklist

Verbose log

name@iMac-Pro % youtube-dl --verbose https://gale.udemy.com/course/tableau-for-beginners/learn/lecture/24794144 --cookies ~/udemy.com_cookies.txt

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://gale.udemy.com/course/tableau-for-beginners/learn/lecture/24794144', '--cookies', '/Users/me/udemy.com_cookies.txt']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.9.6 (CPython) - macOS-11.2.3-x86_64-i386-64bit
[debug] exe versions: ffmpeg present, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[udemy:course] course: Downloading webpage
[udemy:course] 2209946: Downloading course curriculum
[download] Downloading playlist: 2209946
[udemy:course] playlist 2209946: Collected 195 video ids (downloading 195 of them)
[download] Downloading video 1 of 195
[udemy] 24874884: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 555, in error
    result = self._call_chain(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 555, in error
    result = self._call_chain(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

I logged out of udemy and re-logged back on, extracted cookies with cookiestxt extension. I'm able to watch the udemy video course using chrome, but running youtube-dl does not work --> ERROR: Unable to download webpage: HTTP Error 403 Please check for the complete verbose dump.

Don't know if this is relevant, or helpful, but I'm able to download using udemy-dl.py, but the quality is limited to 720, and if I manually set quality to 1080, the application freezes indefinitely. Now I'm trying with youtube-dl, hopefully I manage to grab the videos with the same resolution as I'm able to watch them with the browser.

qrrabbit commented 2 years ago

added more info to the description.

jack-perter-333 commented 2 years ago

403 is an access error, maybe you need an account and password or cookies

qrrabbit commented 2 years ago

Hello Jack, thanks for looking into my case. Well, I do use cookies as you can see from my command-line parameters. The login would not work for me since I'm logging in from the library account, so with cookies I can login from the private browser session and continue watching the video, but for youtube-dl something is not working. Please note, I specifically am looking for 1080, otherwise I could use udemy-dl.py, which, also breaks on 1080.

jack-perter-333 commented 2 years ago

how do you get cookies file?

jack-perter-333 commented 2 years ago

you may have to review the configuration file coockies and source code. You are denied because you do not have enough permissions to access this page.

qrrabbit commented 2 years ago

I use Get cookies.txt, as per the youtube-dl specification.

jack-perter-333 commented 2 years ago

this video is free ? i saw somewhere that you have to pay before downloading in udemy

qrrabbit commented 2 years ago

Hello Jack, No this video is not free, this is the subscription based, and I have full access and full rights to watch it. Just so, I would like to download it so I could use it inside VLC, and not through Udemy annoying interface. But, no, this specific tutorial is not a fee one.

qrrabbit commented 2 years ago

By the way, I did specify in my description that I also tried udemy-dl.py, and it works up to 720p but not for 1080. It does seem like it is about to start downloading, but then it just spins doing nothing. 720p goes right into download really fast, but 720 is too low resolution for this type of instruction.

qrrabbit commented 2 years ago

Another option is, if you would be ready to debug it, I could share my cookies file with you, see if you could step through the code and see where it stops. I don't mind doing that.

jack-perter-333 commented 2 years ago

i will try it.contact me

qrrabbit commented 2 years ago

Jack, the embarrassing thing is, I don't know how to contact you. If you use whatsapp, I can put it here, then take it out once you copied it.

qrrabbit commented 2 years ago

One thing I noticed is that cookies for udemy are very short-lived, and I must re-authenticate myself even when I step out for lunch.

jack-perter-333 commented 2 years ago

that's normal. but i'm wondering if you got the right link?because maybe udemydl supports downloading from articles or courses but youtubedl doesn't. maybe it just downloads from video link. have you tried to see if the video has the right link?

qrrabbit commented 2 years ago

Let me get you the fresh cookies file, see if you can get it to work. About the different links for udemy-dl.py and youtube-dl, I don't know, ubemy-dl shouldn't be any smarter than youtube-dl at identifying the proper link to an actual video file.

jack-perter-333 commented 2 years ago

I sent you a message

iadcialim commented 2 years ago

hi @jack-perter-333 @qrrabbit can u also help me? I have the same prob here

youtube-dl --cookies udemy.com_cookies.txt --verbose  https://www.udemy.com/course/learn-flutter-dart-to-build-ios-android-apps/learn/lecture/10459792
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--cookies', u'udemy.com_cookies.txt', u'--verbose', u'https://www.udemy.com/course/learn-flutter-dart-to-build-ios-android-apps/learn/lecture/10459792']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 2.7.16 (CPython) - Darwin-20.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[udemy:course] course: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
jack-perter-333 commented 2 years ago

contact me bro

niklzz commented 2 years ago

anybody can help?

G:\Udemy>youtube-dl.exe --cookies cookie.txt https://www.udemy.com/course/plsql-beginner-to-advanced-become-a-perfect-plsql-developer --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'cookie.txt', 'https://www.udemy.com/course/plsql-beginner-to-advanced-become-a-perfect-plsql-developer', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: none
[debug] Proxy map: {}
[udemy:course] course: Downloading webpage
[udemy:course] 1312466: Downloading course curriculum
[download] Downloading playlist: 1312466
[udemy:course] playlist 1312466: Collected 118 video ids (downloading 118 of them)
[download] Downloading video 1 of 118
[udemy] 8511332: Downloading webpage
[udemy] 8511332: Downloading lecture JSON
[udemy] 8511332: Downloading lecture JSON
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\common.py", line 634, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\YoutubeDL.py", line 2288, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
dirkf commented 2 years ago

anybody can help?

See #22306, closed as solved, but, no.

dirkf commented 2 years ago

Also see why a dedicated tool for Udemy was abandoned.

deepaksood619 commented 5 months ago

Any solution to this problem? I am facing the same error.

Cookies extracted from extenstion - https://chrome.google.com/webstore/detail/cclelndahbckbenkjhflpdbgdldlbecc in Netscape format. Tried clear and relogin too. Able to access all videos from web directly with same cookie.

youtube-dl --cookies cookies.txt https://www.udemy.com/aws-certified-developer-associate-dva-c01/ --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'cookies.txt', 'https://www.udemy.com/aws-certified-developer-associate-dva-c01/', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: cfac516ee0
[debug] Python version 3.12.1 (CPython) - macOS-14.2.1-arm64-arm-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[udemy:course] aws-certified-developer-associate-dva-c01: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/opt/homebrew/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.12/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.12/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 521, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 630, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 559, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 639, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
dirkf commented 5 months ago

Your log is the same as mine for unauthenticated access. But in the browser I get a redirect to the URL with `.../course/...'. The 403 comes with a Cloudflare challenge page that is bypassed by even quite an unusual real browser. Presumably the site is bouncing out non-browser clients by TLS fingerprinting or similar applied by the CF CDN server. If cookies aren't helping to bypass this, there's nothing yt-dl can do at the moment, though far downstream is https://github.com/yt-dlp/yt-dlp/pull/7595.