Resources parsed from MPD files dont contain signature for request resulting in a HTTP 403 (DiscoveryPlus)

Fuzion24 commented 1 year ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

➜  2021.12.17 git:(stable) ./bin/youtube-dl -Uv --no-geo-bypass --cookies discoveryplus.com_cookies.txt https://www.discoveryplus.com/video/homestead-rescue/hardcore-homesteading
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-Uv', '--no-geo-bypass', '--cookies', 'discoveryplus.com_cookies.txt', 'https://www.discoveryplus.com/video/homestead-rescue/hardcore-homesteading']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 3748bed37
[debug] Proxy map: {}
It looks like you installed youtube-dl with a package manager, pip, setup.py or a tarball. Please use that to update.
yoooo im here using common downloader
[DiscoveryPlus] homestead-rescue/hardcore-homesteading: Downloading JSON metadata
[DiscoveryPlus] 846793: Downloading JSON metadata
hello there: ext: mpd format_id: dash
[DiscoveryPlus] homestead-rescue/hardcore-homesteading: Downloading MPD manifest
urlh.geturl:https://dplus-northamerica.media-edge.prod-vod.h264.io/82a6a9c8-8581-42f8-b83d-f7aec5daccf5/dash_clear_fmp4/edge-cache-token=Expires=9999999999&KeyName=media-edge-key-prod-northamerica-200d417&Signature=[REDACTED]/master.mpd
Traceback (most recent call last):
  File "/youtube-dl/2021.12.17/./bin/youtube-dl", line 33, in <module>
    sys.exit(load_entry_point('youtube-dl==2021.12.17', 'console_scripts', 'youtube-dl')())
  File "__init__.py", line 475, in main
    _real_main(argv)
  File "__init__.py", line 465, in _real_main
    retcode = ydl.download(all_urls)
  File "YoutubeDL.py", line 2070, in download
    res = self.extract_info(
  File "YoutubeDL.py", line 810, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "YoutubeDL.py", line 817, in wrapper
    return func(self, *args, **kwargs)
  File "YoutubeDL.py", line 838, in __extract_info

This bug is related to #4147 but that fix didnt quite resolve the issue. The signature and expiration from the .mpd file is getting dropped when parsing the xml and appending the resources to the base_url. I'm not sure what the proper fix is, but I was able to fix it with this kludge of a hack:

    def _extract_mpd_formats(self, mpd_url, video_id, mpd_id=None, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):

[...]
        mpd_base_url = base_url(urlh.geturl())

        mpd_base_url = urlh.geturl().replace("master.mpd", "")

Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

➜  2021.12.17 git:(stable) ./bin/youtube-dl -Uv --no-geo-bypass --cookies discoveryplus.com_cookies.txt https://www.discoveryplus.com/video/homestead-rescue/hardcore-homesteading
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-Uv', '--no-geo-bypass', '--cookies', 'discoveryplus.com_cookies.txt', 'https://www.discoveryplus.com/video/homestead-rescue/hardcore-homesteading']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 3748bed37
[debug] Proxy map: {}
It looks like you installed youtube-dl with a package manager, pip, setup.py or a tarball. Please use that to update.
yoooo im here using common downloader
[DiscoveryPlus] homestead-rescue/hardcore-homesteading: Downloading JSON metadata
[DiscoveryPlus] 846793: Downloading JSON metadata
hello there: ext: mpd format_id: dash
[DiscoveryPlus] homestead-rescue/hardcore-homesteading: Downloading MPD manifest
urlh.geturl:https://dplus-northamerica.media-edge.prod-vod.h264.io/82a6a9c8-8581-42f8-b83d-f7aec5daccf5/dash_clear_fmp4/edge-cache-token=Expires=9999999999&KeyName=media-edge-key-prod-northamerica-200d417&Signature=[REDACTED]/master.mpd
Traceback (most recent call last):
  File "/youtube-dl/2021.12.17/./bin/youtube-dl", line 33, in <module>
    sys.exit(load_entry_point('youtube-dl==2021.12.17', 'console_scripts', 'youtube-dl')())
  File "__init__.py", line 475, in main
    _real_main(argv)
  File "__init__.py", line 465, in _real_main
    retcode = ydl.download(all_urls)
  File "YoutubeDL.py", line 2070, in download
    res = self.extract_info(
  File "YoutubeDL.py", line 810, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "YoutubeDL.py", line 817, in wrapper
    return func(self, *args, **kwargs)
  File "YoutubeDL.py", line 838, in __extract_info

dirkf commented 1 year ago

Is your problem is with yt-dl or yt-dlp? You seem to have pasted your issue https://github.com/yt-dlp/yt-dlp/issues/5663 without regard to the issue template here.

Also...

How is issue 4147 (from either repo) related?
Since the site serves the manifest with the signature, why "redact" it?
For yt-dl, just use -v to get the verbose log.

Fuzion24 commented 1 year ago

Sorry for the sloppy issue. Here's how youtube-dl (from homebrew) fails currently:

youtube-dl -Uv --no-geo-bypass --cookies cookies.txt https://www.discoveryplus.com/video/homestead-rescue/hardcore-homesteading
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-Uv', '--no-geo-bypass', '--cookies', 'cookies.txt', 'https://www.discoveryplus.com/video/homestead-rescue/hardcore-homesteading']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 3748bed37
[debug] exe versions: ffmpeg 5.1, ffprobe 5.1, rtmpdump 2.4
[debug] Proxy map: {}
It looks like you installed youtube-dl with a package manager, pip, setup.py or a tarball. Please use that to update.
[DiscoveryPlus] homestead-rescue/hardcore-homesteading: Downloading JSON metadata
[DiscoveryPlus] 846793: Downloading JSON metadata
[DiscoveryPlus] homestead-rescue/hardcore-homesteading: Downloading MPD manifest
WARNING: [DiscoveryPlus] Unknown MIME type image/jpeg in DASH manifest
WARNING: [DiscoveryPlus] Unknown MIME type application/mp4 in DASH manifest
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://dplus-northamerica-cloudfront-gcs.prod-vod.h264.io/82a6a9c8-8581-42f8-b83d-f7aec5daccf5/dash_clear_fmp4/x-discovery-token=Expires=[99999]&KeyName=primary&Signature=[REDACTED]/master.mpd'
[dashsegments] Total fragments: 630
[download] Destination: Hardcore Homesteading-846793.fdash-video=9861000.mp4
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 1 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 2 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 3 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 4 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 5 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 6 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 7 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 8 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 9 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 10 of 10)...
ERROR: giving up after 10 fragment retries
  File "/opt/homebrew/bin/youtube-dl", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/__init__.py", line 475, in main
    _real_main(argv)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/__init__.py", line 465, in _real_main
    retcode = ydl.download(all_urls)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2068, in download
    res = self.extract_info(
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 808, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 847, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 881, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1692, in process_video_result
    self.process_info(new_info)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1970, in process_info
    partial_success = dl(fname, new_info)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1915, in dl
    return fd.download(name, info)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/downloader/dash.py", line 75, in real_download
    self.report_error('giving up after %s fragment retries' % fragment_retries)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 628, in report_error
    self.trouble(error_message, tb)
  File "/opt/homebrew/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 590, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

My confusion with cross posting the issue came from searching the stack trace and finding a related issue on yt-dlp: https://github.com/yt-dlp/yt-dlp/issues/4187 It was fixed with this commit: https://github.com/metayan/yt-dlp/commit/65dd00c13c2eece13e1e73a556400111608befb5

Backporting this commit fixed my issue. I opened a pull request for the backport #31490

dirkf commented 1 year ago

That's all much clearer, thanks.

ytdl-org / youtube-dl