ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.89k stars 10k forks source link

Cannot download from Udemy, through SSO, Username/Password, Cookies.txt #24138

Closed marcvisconte closed 4 years ago

marcvisconte commented 4 years ago

Checklist

Verbose log

This is for using the --username / --password options, discarding other (output formatting, audio, etc.) options: [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--verbose', '--dump-json', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://gdit.udemy.com/complete-linux-training-course-to-get-your-dream-it-job/'] [debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252 [debug] youtube-dl version 2020.02.16 [debug] Python version 3.6.4 (CPython) - Windows-7-6.1.7601-SP1 [debug] exe versions: none [debug] Proxy map: {} ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\common.py", line 627, in _request_webpage return self._downloader.urlopen(url_or_request) File "c:\program files\python36\lib\site-packages\youtube_dl\YoutubeDL.py", line 2237, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "c:\program files\python36\lib\urllib\request.py", line 532, in open response = meth(req, response) File "c:\program files\python36\lib\urllib\request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "c:\program files\python36\lib\urllib\request.py", line 570, in error return self._call_chain(args) File "c:\program files\python36\lib\urllib\request.py", line 504, in _call_chain result = func(args) File "c:\program files\python36\lib\urllib\request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

I get to access Udemy through a work account.
I can enter the site from the company portal, which is Okta-hosted, using SSO.
Udemy appears as company-branded Udemy URL: company_name.udemy.com I can locate and play the course.

From the same computer/browser, while NOT using the work portal, I can log in to the site, and play the content, probably due to saved cookies.

Udemy allows business-account users to create a linked individual account (i.e., using work email) and set a password. I have done this. I can access Udemy from a separate browser -- NOT using the Okta SSO portal, using the email address/ password.

From the NON-portal-using browser, I entered the company-branded Udemy URL and tried the option to continue to Udemy courses through SSO. Attempting to enter the company-branded URL and selecting the SSO option forwards the browser to another site: https://sso.connect.pingidentity.com/sso/sp/initsso/?idpid={companyID}&saasid={36-character-alphanumeric-hash}

I have results from multiple attempts, using different combinations of commands and parameters (explicitly pathing the executables, etc).
The LAST attempt, using the working email address/password, resulted in the VERBOSE log above.

Attempting to d/l from this company-branded URL with these settings results in various HTTP Error 403: FORBIDDEN messages.

Normal, using cookies.txt "C:\Program Files\Python36\Scripts\youtube-dl.exe" --verbose --restrict-filenames --cookie ".\cookies_2020_02_18_upd.txt" -f "best" -o "./Dload_p3/%(playlist)s/%(chapter_number)s - %(chapter)s/%(playlist_index)s - %(title)s.%(ext)s" --add-metadata --ffmpeg-location "F:\xfers-progs\ffmpeg\bin\ffmpeg.exe" --postprocessor-args "-id3v2_version 3" https://gdit.udemy.com/complete-linux-training-course-to-get-your-dream-it-job/ [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--verbose', '--restrict-filenames', '--cookie', '.\cookies_2020_02_18_upd.txt', '-f', 'best', '-o', './Dload_p3/%(playlist)s/%(chapter_number)s - %(chapter)s/%(playlist_index)s - %(title)s.%(ext)s', '--add-metadata', '--ffmpeg-location', 'F:\xfers-progs\ffmpeg\bin\ffmpeg.exe', '--postprocessor-args', '-id3v2_version 3', 'https://gdit.udemy.com/complete-linux-training-course-to-get-your-dream-it-job/'] [debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252 [debug] youtube-dl version 2020.02.16 [debug] Python version 3.6.4 (CPython) - Windows-7-6.1.7601-SP1 [debug] exe versions: ffmpeg git-2020-01-26-5e62100, ffprobe git-2020-01-26-5e62100 [debug] Proxy map: {} [udemy:course] complete-linux-training-course-to-get-your-dream-it-job: Downloading webpage [udemy:course] 1523066: Downloading course curriculum ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\common.py", line 627, in _request_webpage return self._downloader.urlopen(url_or_request) File "c:\program files\python36\lib\site-packages\youtube_dl\YoutubeDL.py", line 2237, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "c:\program files\python36\lib\urllib\request.py", line 532, in open response = meth(req, response) File "c:\program files\python36\lib\urllib\request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "c:\program files\python36\lib\urllib\request.py", line 570, in error return self._call_chain(args) File "c:\program files\python36\lib\urllib\request.py", line 504, in _call_chain result = func(args) File "c:\program files\python36\lib\urllib\request.py", line 650, in http_error_default

Not knowing if I needed to keep cookies.txt while using --username / --password, I attempted this: [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--cookies', 'cookies.txt', '--verbose', '--dump-json', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://gdit.udemy.com/complete-linux-training-course-to-get-your-dream-it-job/'] [debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252 [debug] youtube-dl version 2020.02.16 [debug] Python version 3.6.4 (CPython) - Windows-7-6.1.7601-SP1 [debug] exe versions: none [debug] Proxy map: {} ERROR: Unable to extract login-form form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "c:\program files\python36\lib\site-packages\youtube_dl\YoutubeDL.py", line 796, in extract_info ie_result = ie.extract(url) File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\common.py", line 529, in extract self.initialize() File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\common.py", line 433, in initialize self._real_initialize() File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\udemy.py", line 172, in _real_initialize self._login() File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\udemy.py", line 191, in _login login_form = self._form_hidden_inputs('login-form', login_popup) File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\common.py", line 1322, in _form_hidden_inputs html, '%s form' % form_id, group='form') File "c:\program files\python36\lib\site-packages\youtube_dl\extractor\common.py", line 1005, in _search_regex raise RegexNotFoundError('Unable to extract %s' % _name) youtube_dl.utils.RegexNotFoundError: Unable to extract login-form form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I also attempted using the SSO.CONNECT.PINGIDENTITY.COM url as a referrer URL. This also failed. Meh... it was worth a try.

I'm not compulsive. Not one bit.

dstftw commented 4 years ago

You must provide account credentials for any work on this.