derekcentrico commented 5 years ago

Checklist

[X ] I'm reporting a broken site support
[X ] I've verified that I'm running youtube-dl version 2019.07.02
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones

Verbose log

youtube-dl -v --cookies=/home/files/cookiediscovery.txt -f bestvideo[ext!=webm]+bestaudio[ext!=webm]/best[ext!=webm] https://www.discovery.com/tv-shows/naked-and-afraid/full-episodes/stalked-on-the-savannah? 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--cookies=/home/files/cookiediscovery.txt', u'-f', u'bestvideo[ext!=webm]+bestaudio[ext!=webm]/best[ext!=webm]', u'https://www.discovery.com/tv-shows/naked-and-afraid/full-episodes/stalked-on-the-savannah?']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.07.02
[debug] Python version 2.7.15+ (CPython) - Linux-4.15.0-52-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6
[debug] Proxy map: {}
[Discovery] stalked-on-the-savannah: Downloading webpage
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 474, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2008, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/discovery.py", line 75, in _real_extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 2819, in _get_cookies
  File "/usr/lib/python2.7/Cookie.py", line 583, in __init__
    if input: self.load(input)
  File "/usr/lib/python2.7/Cookie.py", line 643, in load
    for k, v in rawdata.items():
AttributeError: 'unicode' object has no attribute 'items'

Description

I am attempting to download videos that require a subscription. They play fine in the browser. I created the cookies and confirmed they are Netscape compatible. The error above occurs solely with subscriptions, but does not occur when downloading free content from Discovery. I am a Comcast Xfinity user. I've searched all over Google and read several bug tracker notes that came up for Discovery over the years, but none solve my issue.

cat cookiediscovery.txt

Netscape HTTP Cookie File

http://curl.haxx.se/rfc/cookie_spec.html

This is a generated file! Do not edit.

remitamine commented 5 years ago

https://github.com/ytdl-org/youtube-dl#how-do-i-pass-cookies-to-youtube-dl

derekcentrico commented 5 years ago

You closed and put this as invalid. However my cookie file and usage exactly matches your citation which I read well before posting. What are you asserting I'm failing at here?

derekcentrico commented 5 years ago

The only thing I can think of your assertion is to the use of = in the flag. I tried both with and without based on other examples I've seen in my searching. Here it is without... same error.

stalked-on-the-savannah?

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-v', u'--cookies', u'/home/files/cookiediscovery.txt', u'-f', u'bestvideo[ext!=webm]+bestaudio[ext!=webm]/best[ext!=webm]', u'https://www.discovery.com/tv-shows/naked-and-afraid/full-episodes/stalked-on-the-savannah?'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2019.07.02 [debug] Python version 2.7.15+ (CPython) - Linux-4.15.0-52-generic-x86_64-with-Ubuntu-18.04-bionic [debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6 [debug] Proxy map: {} [Discovery] stalked-on-the-savannah: Downloading webpage Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/local/bin/youtube-dl/main.py", line 19, in File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 474, in main File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 464, in _real_main File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2008, in download File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract File "/usr/local/bin/youtube-dl/youtube_dl/extractor/discovery.py", line 75, in _real_extract File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 2819, in _get_cookies File "/usr/lib/python2.7/Cookie.py", line 583, in init if input: self.load(input) File "/usr/lib/python2.7/Cookie.py", line 643, in load for k, v in rawdata.items(): AttributeError: 'unicode' object has no attribute 'items'

remitamine commented 5 years ago

obviously, the cookie file you have posted is empty, when you export the cookie file from the browser when you're logged, the file should contain cookies for the domain you're trying to download from(eosAf cookie for discovery domains).

remitamine commented 5 years ago

and for error you have posted, it's a python 2 specific bug, there is already an issue opened for it(#20673).

derekcentrico commented 5 years ago

Apologies. I pasted the top portion of said cookie file to show Netscape format and not Chrome variety. I didn't realize I should post the whole thing. Additionally, thank you for letting me know it is a known python error. I will try changing my shell to python3 and hope for a patch so my automation stuff will function using cookies.

derekcentrico commented 5 years ago

For what it's worth I built from latest source code and it won't work. Additionally, I made a well-discussed file change to adobepass.py for Comcast OAUTH which did not work either. (see https://github.com/bhamboss/youtube-dl/blob/patch-1/youtube_dl/extractor/adobepass.py) I passed just --cookie and --cookie with --ap-mso related flags as well.

403 forbidden.

I don't necessarily want to be a pain in the rear by posting a new bug report to have an open status for assistance so I'm posting here for now.

With --cookies and --ap-mso:

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '--ap-mso', 'Comcast_SSO', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', '--cookies', '/home/derek/Desktop/cookies.txt', '-f', 'bestvideo[ext!=webm]+bestaudio[ext!=webm]/best[ext!=webm]', 'https://www.discovery.com/tv-shows/naked-and-afraid/full-episodes/stalked-on-the-savannah?'] [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2019.07.14 [debug] Python version 3.6.8 (CPython) - Linux-4.18.0-15-generic-x86_64-with-Ubuntu-18.04-bionic [debug] exe versions: none [debug] Proxy map: {} [Discovery] stalked-on-the-savannah: Downloading webpage [Discovery] stalked-on-the-savannah: Downloading JSON metadata ERROR: This video is only available via cable service provider subscription that is not currently supported. You may want to use --cookies. Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage return self._downloader.urlopen(url_or_request) File "./youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.6/urllib/request.py", line 570, in error return self._call_chain(args) File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(args) File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/discovery.py", line 102, in _real_extract display_id, headers=headers) File "./youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage raise ExtractorError(errmsg, sys.exc_info()[2], cause=err) youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage return self._downloader.urlopen(url_or_request) File "./youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.6/urllib/request.py", line 570, in error return self._call_chain(args) File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(args) File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/discovery.py", line 102, in _real_extract display_id, headers=headers) File "./youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage raise ExtractorError(errmsg, sys.exc_info()[2], cause=err) youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info ie_result = ie.extract(url) File "./youtube-dl/youtube_dl/extractor/common.py", line 530, in extract ie_result = self._real_extract(url) File "./youtube-dl/youtube_dl/extractor/discovery.py", line 112, in _real_extract ' is not currently supported. You may want to use --cookies.', expected=True) youtube_dl.utils.ExtractorError: This video is only available via cable service provider subscription that is not currently supported. You may want to use --cookies.

With just --cookies:

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '--cookies', '/home/derek/Desktop/cookies.txt', '-f', 'bestvideo[ext!=webm]+bestaudio[ext!=webm]/best[ext!=webm]', 'https://www.discovery.com/tv-shows/naked-and-afraid/full-episodes/stalked-on-the-savannah?'] [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2019.07.14 [debug] Python version 3.6.8 (CPython) - Linux-4.18.0-15-generic-x86_64-with-Ubuntu-18.04-bionic [debug] exe versions: none [debug] Proxy map: {} [Discovery] stalked-on-the-savannah: Downloading webpage [Discovery] stalked-on-the-savannah: Downloading JSON metadata ERROR: This video is only available via cable service provider subscription that is not currently supported. You may want to use --cookies. Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage return self._downloader.urlopen(url_or_request) File "./youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.6/urllib/request.py", line 570, in error return self._call_chain(args) File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(args) File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/discovery.py", line 102, in _real_extract display_id, headers=headers) File "./youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage raise ExtractorError(errmsg, sys.exc_info()[2], cause=err) youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage return self._downloader.urlopen(url_or_request) File "./youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python3.6/urllib/request.py", line 532, in open response = meth(req, response) File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python3.6/urllib/request.py", line 570, in error return self._call_chain(args) File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(args) File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./youtube-dl/youtube_dl/extractor/discovery.py", line 102, in _real_extract display_id, headers=headers) File "./youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status) File "./youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage raise ExtractorError(errmsg, sys.exc_info()[2], cause=err) youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info ie_result = ie.extract(url) File "./youtube-dl/youtube_dl/extractor/common.py", line 530, in extract ie_result = self._real_extract(url) File "./youtube-dl/youtube_dl/extractor/discovery.py", line 112, in _real_extract ' is not currently supported. You may want to use --cookies.', expected=True) youtube_dl.utils.ExtractorError: This video is only available via cable service provider subscription that is not currently supported. You may want to use --cookies.

Diegus83 commented 5 years ago

There has been a commit a couple days ago that added the new "go.discovery.com" URLs and it is working with the proper cookies extracted from the browser.

You can run the latest source without compiling by doing python -m youtube_dl

If you have multiple versions of python like it is the case of Mac users with Brew installed, I would type instead python3 -m youtube_dl

Since otherwise you may end up running version 2.7 of Python which won't work.

ytdl-org / youtube-dl

Discovery subscriber required pages not working #21679

Checklist

Verbose log

Description

Netscape HTTP Cookie File

http://curl.haxx.se/rfc/cookie_spec.html

This is a generated file! Do not edit.

youtube-dl -v --cookies /home/files/cookiediscovery.txt -f bestvideo[ext!=webm]+bestaudio[ext!=webm]/best[ext!=webm] https://www.discovery.com/tv-shows/naked-and-afraid/full-episodes/stalked-on-the-savannah?