yt-dlp / yt-dlp

A feature-rich command-line audio/video downloader
https://discord.gg/H5MNcFW63r
The Unlicense
86.08k stars 6.71k forks source link

[ARDBetaMediathek] HTTP Error 503: Service Unavailable ARD Mediathek #8731

Closed w4grfw closed 10 months ago

w4grfw commented 10 months ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

Checklist

Region

Germany

Provide a description that is worded well enough to be understood

The error shown below occurs:

Provide verbose output that clearly demonstrates the problem

Complete Verbose Output

$ python -m yt_dlp --verbose --ignore-config https://www.ardmediathek.de/video/Y3JpZDovL3RhZ2Vzc2NoYXUuZGUvNDc3M2U0ZjUtOGNkMi00OGQyLWE3NWUtMWMyNmVkZjFkY2ViLVNFTkRVTkdTVklERU8
[debug] Command-line config: ['--verbose', '--ignore-config', 'https://www.ardmediathek.de/video/Y3JpZDovL3RhZ2Vzc2NoYXUuZGUvNDc3M2U0ZjUtOGNkMi00OGQyLWE3NWUtMWMyNmVkZjFkY2ViLVNFTkRVTkdTVklERU8']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.11.16 from yt-dlp/yt-dlp [24f827875] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 0b6f829b1
[debug] Python 3.11.6 (CPython x86_64 64bit) - Linux-6.5.0-14-generic-x86_64-with-glibc2.38 (OpenSSL 3.0.10 1 Aug 2023, glibc 2.38)
[debug] exe versions: ffmpeg 6.0 (setts), ffprobe 6.0, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, pyxattr-0.8.1, requests-2.31.0, secretstorage-3.3.3, sqlite3-3.42.0, urllib3-1.26.16, websockets-10.4
[debug] Proxy map: {}
[debug] Request Handlers: urllib
[debug] Loaded 1792 extractors
[debug] Using fake IP 53.140.36.43 (DE) as X-Forwarded-For
[ARDBetaMediathek] Extracting URL: https://www.ardmediathek.de/video/Y3JpZDovL3RhZ2Vzc2NoYXUuZGUvNDc3M2U0ZjUtOGNkMi00OGQyLWE3NWUtMWMyNmVkZjFkY2ViLVNFTkRVTkdTVklERU8
[ARDBetaMediathek] Y3JpZDovL3RhZ2Vzc2NoYXUuZGUvNDc3M2U0ZjUtOGNkMi00OGQyLWE3NWUtMWMyNmVkZjFkY2ViLVNFTkRVTkdTVklERU8: Downloading JSON metadata
ERROR: [ARDBetaMediathek] Y3JpZDovL3RhZ2Vzc2NoYXUuZGUvNDc3M2U0ZjUtOGNkMi00OGQyLWE3NWUtMWMyNmVkZjFkY2ViLVNFTkRVTkdTVklERU8: Unable to download JSON metadata: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: Service Unavailable>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/tmp/yt-dlp/yt_dlp/extractor/common.py", line 717, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/extractor/ard.py", line 595, in _real_extract
    player_page = self._download_json(
                  ^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/extractor/common.py", line 1071, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/extractor/common.py", line 1035, in download_handle
    res = self._download_webpage_handle(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/extractor/common.py", line 905, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/extractor/common.py", line 862, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/tmp/yt-dlp/yt_dlp/networking/_urllib.py", line 396, in _send
    res = opener.open(urllib_req, timeout=float(request.extensions.get('timeout') or self.timeout))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Service Unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/yt-dlp/yt_dlp/YoutubeDL.py", line 4069, in urlopen
    return self._request_director.send(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/networking/common.py", line 114, in send
    response = handler.send(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/networking/_helper.py", line 204, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/networking/common.py", line 325, in send
    return self._send(request)
           ^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/networking/_urllib.py", line 401, in _send
    raise HTTPError(UrllibResponseAdapter(e.fp), redirect_loop='redirect error' in str(e)) from e
yt_dlp.networking.exceptions.HTTPError: HTTP Error 503: Service Unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/yt-dlp/yt_dlp/extractor/common.py", line 849, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/yt-dlp/yt_dlp/YoutubeDL.py", line 4101, in urlopen
    raise _CompatHTTPError(e) from e
yt_dlp.networking.exceptions._CompatHTTPError: HTTP Error 503: Service Unavailable
Reiner1210 commented 10 months ago

Same here dvbtv@DVBTV:~/data/video/tvrec/Dahoam> yt-dlp --verbose https://www.ardmediathek.de/video/dahoam-is-dahoam/was-weg-is-is-weg-11-12-23-oder-folge-3275/br-fernsehen/Y3JpZDovL2JyLmRlL3ZpZGVvLzgzOGYyMzZiLTM1ZTAtNGZjYS05NTdiLTBiMTVhMWM5MTQyMA [debug] Command-line config: ['--verbose', 'https://www.ardmediathek.de/video/dahoam-is-dahoam/was-weg-is-is-weg-11-12-23-oder-folge-3275/br-fernsehen/Y3JpZDovL2JyLmRlL3ZpZGVvLzgzOGYyMzZiLTM1ZTAtNGZjYS05NTdiLTBiMTVhMWM5MTQyMA'] [debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version stable@2023.11.16 from yt-dlp/yt-dlp [24f827875] (zip) [debug] Python 3.8.6 (CPython x86_64 64bit) - Linux-5.8.15-1-default-x86_64-with-glibc2.2.5 (OpenSSL 1.1.1h 22 Sep 2020, glibc 2.33) [debug] exe versions: ffmpeg 4.3.1 (fdk), ffprobe 4.3.1 [debug] Optional libraries: brotli-None, certifi-2020.12.05, requests-2.26.0, sqlite3-3.34.0, urllib3-1.26.2 [debug] Proxy map: {} [debug] Request Handlers: urllib [debug] Loaded 1901 extractors [debug] Using fake IP 53.159.33.191 (DE) as X-Forwarded-For [ARDBetaMediathek] Extracting URL: https://www.ardmediathek.de/video/dahoam-is-dahoam/was-weg-is-is-weg-11-12-23-oder-folge-3275/br-fernsehen/Y3JpZDovL2JyLmRlL3ZpZGVvLzgzOGYyMzZiLTM1ZTAtNGZjYS05NTdiLTBiMTVhMWM5MTQyMA [ARDBetaMediathek] dahoam-is-dahoam/was-weg-is-is-weg-11-12-23-oder-folge-3275/br-fernsehen: Downloading JSON metadata ERROR: [ARDBetaMediathek] Y3JpZDovL2JyLmRlL3ZpZGVvLzgzOGYyMzZiLTM1ZTAtNGZjYS05NTdiLTBiMTVhMWM5MTQyMA: Unable to download JSON metadata: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: Service Unavailable>); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 715, in extract ie_result = self._real_extract(url) File "/usr/local/bin/yt-dlp/yt_dlp/extractor/ard.py", line 595, in _real_extract player_page = self._download_json( File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 1069, in download_content res = getattr(self, download_handle.name)(url_or_request, video_id, **kwargs) File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 1033, in download_handle res = self._download_webpage_handle( File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 903, in _download_webpage_handle urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status) File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 860, in _request_webpage raise ExtractorError(errmsg, cause=err)

File "/usr/local/bin/yt-dlp/yt_dlp/networking/_urllib.py", line 396, in _send res = opener.open(urllib_req, timeout=float(request.extensions.get('timeout') or self.timeout)) File "/usr/lib64/python3.8/urllib/request.py", line 531, in open response = meth(req, response) File "/usr/lib64/python3.8/urllib/request.py", line 640, in http_response response = self.parent.error( File "/usr/lib64/python3.8/urllib/request.py", line 569, in error return self._call_chain(args) File "/usr/lib64/python3.8/urllib/request.py", line 502, in _call_chain result = func(args) File "/usr/lib64/python3.8/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 4052, in urlopen return self._request_director.send(req) File "/usr/local/bin/yt-dlp/yt_dlp/networking/common.py", line 114, in send response = handler.send(request) File "/usr/local/bin/yt-dlp/yt_dlp/networking/_helper.py", line 204, in wrapper return func(self, *args, **kwargs) File "/usr/local/bin/yt-dlp/yt_dlp/networking/common.py", line 325, in send return self._send(request) File "/usr/local/bin/yt-dlp/yt_dlp/networking/_urllib.py", line 401, in _send raise HTTPError(UrllibResponseAdapter(e.fp), redirect_loop='redirect error' in str(e)) from e yt_dlp.networking.exceptions.HTTPError: HTTP Error 503: Service Unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 847, in _request_webpage return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query)) File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 4074, in urlopen raise _CompatHTTPError(e) from e yt_dlp.networking.exceptions._CompatHTTPError: HTTP Error 503: Service Unavailable

dirkf commented 10 months ago

...503 error for multiple Videos since today

Did it actually work on the day before, or was there a different error (eg 410)?

Probably this is #7666 but the site has now given up on returning 410 ("something that was available is no longer there") and is just returning a generic "something went wrong" via 503.

w4grfw commented 10 months ago

Older videos work, i.e. Tagesschau 6.12.2023 https://www.ardmediathek.de/video/Y3JpZDovL3RhZ2Vzc2NoYXUuZGUvOTIxMTc2NzUtYzU1NC00NTJmLTgwNjgtMGUzOTVjNzliNzZkLVNFTkRVTkdTVklERU8 I only get the 410-error for videos from the hr-channel, e.g. https://www.ardmediathek.de/video/Y2VjY2JlYjUtODQ0MC00N2I5LWJmMDYtNjRlYmZjMDU1YTI5

andreygursky commented 10 months ago

I get "HTTP Error 503: Service Unavailable" since 07.12.2023. Luckily, access through https://api.ardmediathek.de/page-gateway/pages/ard/item/VIDEO_ID?embedded=false&mcV6=true (which web site uses itself) continues to work.

Gerd-Die commented 10 months ago

fwiw: It seems to spread - what still worked this morning now gets a 503 as well. On one occasion I got the German version but not the foreign language one - now the German version also gets a 503. It looks as if ARD is transitioning something here.

niehauss commented 10 months ago

fwiw: It seems to spread - what still worked this morning now gets a 503 as well. On one occasion I got the German version but not the foreign language one - now the German version also gets a 503. It looks as if ARD is transitioning something here.

Can confirm. Same here.

shivan2418 commented 10 months ago

Got same error just now.

tkueeu commented 10 months ago

You can try to get the url of master.m3u8 from the browser network protocol when doing playback in the browser and pass that url as argument.

niehauss commented 10 months ago

You can try to get the url of master.m3u8 from the browser network protocol when doing playback in the browser and pass that url as argument.

Ugly but it works. Thank you.

barsnick commented 10 months ago

I observed this a few days ago. While bisecting, I noticed that the site sometimes delivers all the formats and downloads properly, and sometimes yt-dlp terminates with this 503 error.

I'm guessing: There is some intermittent site behavior, or randomly returned pages according to different APIs.

SergiusMohr commented 10 months ago

same issue here for 'BR' provider:

yt-dlp -v https://www.ardmediathek.de/video/tinkas-welt/tag-10-das-grosse-jahrestreffen-s03-e10/br-fernsehen/Y3JpZDovL2JyLmRlL3ZpZGVvLzZmOGIzOGZiLThmMGItNDhlZC04ODkzLWJjNzk4YzJlY2M2OQ
[debug] Command-line config: ['-v', 'https://www.ardmediathek.de/video/tinkas-welt/tag-10-das-grosse-jahrestreffen-s03-e10/br-fernsehen/Y3JpZDovL2JyLmRlL3ZpZGVvLzZmOGIzOGZiLThmMGItNDhlZC04ODkzLWJjNzk4YzJlY2M2OQ']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.10.13 [b634ba742] (zip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-91-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: certifi-2020.06.20, mutagen-1.45.1, pyxattr-0.7.2, secretstorage-3.3.1, sqlite3-3.37.2
[debug] Proxy map: {}
[debug] Loaded 1890 extractors
[debug] Using fake IP 53.68.26.185 (DE) as X-Forwarded-For
[ARDBetaMediathek] Extracting URL: https://www.ardmediathek.de/video/tinkas-welt/tag-10-das-grosse-jahrestreffen-s03-e10/br-fernsehen/Y3JpZDovL2JyLmRlL3ZpZGVvLzZmOGIzOGZiLThmMGItNDhlZC04ODkzLWJjNzk4YzJlY2M2OQ
[ARDBetaMediathek] tinkas-welt/tag-10-das-grosse-jahrestreffen-s03-e10/br-fernsehen: Downloading JSON metadata
ERROR: [ARDBetaMediathek] Y3JpZDovL2JyLmRlL3ZpZGVvLzZmOGIzOGZiLThmMGItNDhlZC04ODkzLWJjNzk4YzJlY2M2OQ: Unable to download JSON metadata: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: Service Unavailable>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 715, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/ard.py", line 595, in _real_extract
    player_page = self._download_json(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 1069, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 1033, in download_handle
    res = self._download_webpage_handle(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 903, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 860, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/usr/local/bin/yt-dlp/yt_dlp/networking/_urllib.py", line 410, in _send
    res = opener.open(urllib_req, timeout=float(request.extensions.get('timeout') or self.timeout))
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Service Unavailable

the workaround with the URL from master.m3u8 works:

yt-dlp https://br-i.akamaihd.net/i/geo/MUJIuUOVBwQIbtCCBLzGiLC1uwQoNA4p_2ZS/_AJS/_2FP_-4g5H1S/6f8b38fb-8f0b-48ed-8893-bc798c2ecc69_,A,E,X,HD,C,.mp4.csmil/master.m3u8?__b__=200

i hope it can be fixed soon, thx!

Meckiilino commented 10 months ago

same issue here:

`yt-dlp-nightly.exe https://www.ardmediathek.de/video/feuer-und-flamme/folge-2-wohnungsbrand-menschen-in-lebensgefahr-s07-e02/wdr/Y3JpZDovL3dkci5kZS9CZWl0cmFnLXNvcGhvcmEtMzM5NGE3OWYtMmRkOS00NjVkLWJkMzYtYTY2ZTcyYWRjY2Ux -vU [debug] Command-line config: ['https://www.ardmediathek.de/video/feuer-und-flamme/folge-2-wohnungsbrand-menschen-in-lebensgefahr-s07-e02/wdr/Y3JpZDovL3dkci5kZS9CZWl0cmFnLXNvcGhvcmEtMzM5NGE3OWYtMmRkOS00NjVkLWJkMzYtYTY2ZTcyYWRjY2Ux', '-vU'] [debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version nightly@2023.12.13.232710 from yt-dlp/yt-dlp-nightly-builds [6b5d93b0b] (win_exe) [debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1k 25 Mar 2021) [debug] exe versions: ffmpeg N-112972-g5256b2fbe6-20231214 (setts), ffprobe N-112972-g5256b2fbe6-20231214 [debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.11.17, mutagen-1.47.0, requests-2.31.0, sqlite3-3.35.5, urllib3-2.1.0, websockets-12.0 [debug] Proxy map: {} [debug] Request Handlers: urllib, requests, websockets [debug] Loaded 1792 extractors [debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest Latest version: nightly@2023.12.13.232710 from yt-dlp/yt-dlp-nightly-builds yt-dlp is up to date (nightly@2023.12.13.232710 from yt-dlp/yt-dlp-nightly-builds) [debug] Using fake IP 53.25.243.41 (DE) as X-Forwarded-For [ARDBetaMediathek] Extracting URL: https://www.ardmediathek.de/video/feuer-und-flamme/folge-2-wohnungsbrand-menschen-in-lebensgefahr-s07-e02/wdr/Y3JpZDovL3dkci5kZS9CZWl0cmFnLXNvcGhvcmEtMzM5NGE3OWYtMmRkOS00NjVkLWJkMzYtYTY2ZTcyYWRjY2Ux [ARDBetaMediathek] feuer-und-flamme/folge-2-wohnungsbrand-menschen-in-lebensgefahr-s07-e02/wdr: Downloading JSON metadata ERROR: [ARDBetaMediathek] Y3JpZDovL3dkci5kZS9CZWl0cmFnLXNvcGhvcmEtMzM5NGE3OWYtMmRkOS00NjVkLWJkMzYtYTY2ZTcyYWRjY2Ux: Unable to download JSON metadata: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: Service Unavailable>); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U File "yt_dlp\extractor\common.py", line 718, in extract File "yt_dlp\extractor\ard.py", line 595, in _real_extract File "yt_dlp\extractor\common.py", line 1072, in download_content File "yt_dlp\extractor\common.py", line 1036, in download_handle File "yt_dlp\extractor\common.py", line 906, in _download_webpage_handle File "yt_dlp\extractor\common.py", line 863, in _request_webpage

File "yt_dlp\YoutubeDL.py", line 4069, in urlopen File "yt_dlp\networking\common.py", line 114, in send File "yt_dlp\networking_helper.py", line 204, in wrapper File "yt_dlp\networking\common.py", line 325, in send File "yt_dlp\networking_requests.py", line 341, in _send yt_dlp.networking.exceptions.HTTPError: HTTP Error 503: Service Unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "yt_dlp\extractor\common.py", line 850, in _request_webpage File "yt_dlp\YoutubeDL.py", line 4101, in urlopen yt_dlp.networking.exceptions._CompatHTTPError: HTTP Error 503: Service Unavailable`

SergiusMohr commented 10 months ago

issue is still open (last update contains no fix), the workaround with master.m3u8 if anybody needs help:

Bildschirmfoto vom 2023-12-19 09-19-00

Unrepentant-Atheist commented 10 months ago

Still present:

[debug] Command-line config: ['https://www.ardmediathek.de/video/Y3JpZDovL3N3ci5kZS9hZXgvbzE5NzIyMzg', '--all-subs', '-vU']
[debug] Portable config "C:\Scoop\apps\yt-dlp\current\yt-dlp.conf": []
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.11.16 from yt-dlp/yt-dlp [24f827875] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-113032-g76cd71bf0c-20231219 (setts), ffprobe N-113032-g76cd71bf0c-20231219
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, requests-2.31.0, sqlite3-3.35.5, urllib3-2.1.0, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests
[debug] Loaded 1901 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2023.11.16 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2023.11.16 from yt-dlp/yt-dlp)
[debug] Using fake IP 53.253.122.9 (DE) as X-Forwarded-For
[ARDBetaMediathek] Extracting URL: https://www.ardmediathek.de/video/Y3JpZDovL3N3ci5kZS9hZXgvbzE5NzIyMzg
[ARDBetaMediathek] Y3JpZDovL3N3ci5kZS9hZXgvbzE5NzIyMzg: Downloading JSON metadata
ERROR: [ARDBetaMediathek] Y3JpZDovL3N3ci5kZS9hZXgvbzE5NzIyMzg: Unable to download JSON metadata: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: Service Unavailable>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 715, in extract
  File "yt_dlp\extractor\ard.py", line 595, in _real_extract
  File "yt_dlp\extractor\common.py", line 1069, in download_content
  File "yt_dlp\extractor\common.py", line 1033, in download_handle
  File "yt_dlp\extractor\common.py", line 903, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 860, in _request_webpage

  File "yt_dlp\YoutubeDL.py", line 4052, in urlopen
  File "yt_dlp\networking\common.py", line 114, in send
  File "yt_dlp\networking\_helper.py", line 204, in wrapper
  File "yt_dlp\networking\common.py", line 325, in send
  File "yt_dlp\networking\_requests.py", line 341, in _send
yt_dlp.networking.exceptions.HTTPError: HTTP Error 503: Service Unavailable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "yt_dlp\extractor\common.py", line 847, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 4074, in urlopen
yt_dlp.networking.exceptions._CompatHTTPError: HTTP Error 503: Service Unavailable
calestyo commented 10 months ago

https://github.com/barsnick/yt-dlp/commit/c02083f8d16c69d3570de5705557a29db7ee5ea2 fixes the issue for me, it seems.

niehauss commented 10 months ago

fixes the issue for me, it seems.

Hm, I got mixed results.

https://www.ardmediathek.de/video/luka-und-das-magische-theater/folge-3-echte-schlitzohren/kika/Y3JpZDovL3dkci5kZS9CZWl0cmFnLTUyMDQ2MDliLTA2ZDItNGQ5Ny1hN2I4LWI4NmYyOGIwZDRlZA?isChildContent still does not work for me

However https://www.ardmediathek.de/video/luka-und-das-magische-theater/folge-2-taschendiebe/kika/Y3JpZDovL3dkci5kZS9CZWl0cmFnLWFlMGRlMTZmLWU0YmYtNDI3YS05NzA2LTA3NWNhNzZlMDExOQ?isChildContent and https://www.ardmediathek.de/video/luka-und-das-magische-theater/folge-1-kleine-spatzen/kika/Y3JpZDovL3dkci5kZS9CZWl0cmFnLTlmZmQyZGI2LWNhZjAtNDYwMy04NzBhLTBmZTVkMzNiMDkwNg?isChildContentworked for me.

calestyo commented 10 months ago

All these three work for me (with the patch)... I mean they start downloading at least (didn't wait until the end).

LNA-DEV commented 10 months ago

They work for me too.

It generally seems to work good.

nolexio commented 10 months ago

I do also get mixed results. I checked the three links aboce. First one worked, the other two ran into the 503 Service Unavailable error.

hutrx commented 10 months ago

Here is a very simple workaround that works for me.

This link to the latest Tatort on the ARD site gives a 503 error. https://www.ardmediathek.de/video/tatort/kontrollverlust/ard/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC8yMDIzLTEyLTI2XzIwLTE1LU1FWg

This link to exactly the same Tatort on ZDF's site works. https://www.zdf.de/ard/tatort/page-video-ard-kontrollverlust-104.html

So just download it from ZDF instead. You can search their site for whatever you want and use their link.

(German public TV channels ZDF and ARD are working together. They are heading towards a single streaming service. Each site now allows you to search for and play the other site’s material. This is quite recent and may explain the changes at ARD. I have not come across any shows where I could not use this approach.)

niehauss commented 9 months ago

Thanks! Works fine now.

Meckiilino commented 9 months ago

Thanks also from my side.

Gerd-Die commented 9 months ago

Yep. It's back to working ... THANKS!!

(those endless file names do take some getting used to, though ... I'll have to think of a clever renaming algorithm now ;-)

seproDev commented 9 months ago

@Gerd-Die If you don't want to have the ID in the filename, you can use output templates. For example: -o "%(title)s.%(ext)s" You can also use output templates to add additional info like the series name.

andreygursky commented 9 months ago

I'm experiencing a regression with #8878. Filename ID should remain contentId from .tracking.atiCustomVars.contentId (page-gateway WEB api) not the weirdly base64-encoded metadataId.

Grub4K commented 9 months ago

The change was intentional. Since the id moved from the url to the described field, we assumed it might be removed at any time.

The old id is still provided for compatibility with the download archive. To restore it for as long as its still available, you may use -o "%(title)s [%(_old_archive_ids.0.17:)s].%(ext)s". %(_old_archive_ids.0.17:)s can be used in metadata replacement to revert to the old behavior completely. Keep in mind that this then stops working when the field disappears.

andreygursky commented 9 months ago

Since the id moved from the url to the described field

This hasn't changed recently, but after transition from classic ARD to ARD-beta few years ago. And after sometime with those unusable IDs in filenames, youtube-dl (and/or yt-dlp) switched back to legacy IDs found as contentId.

The old id is still provided for compatibility with the download archive. ...

Thanks for describing a workaround. Still hope we can have legacy IDs back and using those totally impractically too long strings as ID only as fallback if contentId would disappear.

calestyo commented 9 months ago

Is there a way to convert all the existing (old) IDs to the new ones?

andreygursky commented 9 months ago

Just for example:

with old ID: 'Lupine - regionaler Protein-Protz [93647768].mp4'

with new ID: 'Lupine - regionaler Protein [Y3JpZDovL3JiYi1vbmxpbmUuZGUvc3VwZXJtYXJrdC8yMDIxLTEwLTA0VDIwOjE1OjAwX2EzMzEwM2ZiLTVkMTYtNDdlOC1hZTk0LTk5Y2YzN2JkZWIzNy9sdXBpbmUtc3Vlc3NsdXBpbmUtYm9obmUtcHJvZHVrdGUtaW0tdHJlbmQtYW5iYXUtaW4tYnJhbmRlbmJ1cmctaG9oZXItZWl3ZWlzc2dlaGFsdC1lcnNhdHotZnVlci1mbGVpc2NoLXNvamE].mp4'

The length of the new ID is 263 characters.

andreygursky commented 9 months ago

8916

Thanks!

Mi605 commented 9 months ago

ARD stopped working again. Updating to 2023.12.30 had fixed the issue from early January until now. Yet there seems to be a regression, even in nightly build:

$ yt-dlp -vU 'https://www.ardmediathek.de/live/Y3JpZDovL2Rhc2Vyc3RlLmRlL0xpdmVzdHJlYW0tRGFzRXJzdGU'
[debug] Command-line config: ['-vU', 'https://www.ardmediathek.de/live/Y3JpZDovL2Rhc2Vyc3RlLmRlL0xpdmVzdHJlYW0tRGFzRXJzdGU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.01.21.232719 from yt-dlp/yt-dlp-nightly-builds [f0e8bc7c6] (linux_exe)
[debug] Python 3.10.13 (CPython x86_64 64bit) - Linux-6.1.55-antix.1-amd64-smp-x86_64-with-glibc2.36 (OpenSSL 3.2.0 23 Nov 2023, glibc 2.36)
[debug] exe versions: ffmpeg 5.1.4-0 (setts), ffprobe 5.1.4-0, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.11.17, mutagen-1.47.0, requests-2.31.0, secretstorage-3.3.3, sqlite3-3.44.2, urllib3-2.1.0, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1818 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.01.21.232719 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.01.21.232719 from yt-dlp/yt-dlp-nightly-builds)
[debug] Using fake IP 53.119.133.141 (DE) as X-Forwarded-For
[ARDMediathek] Extracting URL: https://www.ardmediathek.de/live/Y3JpZDovL2Rhc2Vyc3RlLmRlL0xpdmVzdHJlYW0tRGFzRXJzdGU
[ARDMediathek] Y3JpZDovL2Rhc2Vyc3RlLmRlL0xpdmVzdHJlYW0tRGFzRXJzdGU: Downloading JSON metadata
ERROR: [ARDMediathek] Y3JpZDovL2Rhc2Vyc3RlLmRlL0xpdmVzdHJlYW0tRGFzRXJzdGU: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: Not Found>)
  File "yt_dlp/extractor/common.py", line 718, in extract
  File "yt_dlp/extractor/ard.py", line 382, in _real_extract
  File "yt_dlp/extractor/common.py", line 1072, in download_content
  File "yt_dlp/extractor/common.py", line 1036, in download_handle
  File "yt_dlp/extractor/common.py", line 906, in _download_webpage_handle
  File "yt_dlp/extractor/common.py", line 863, in _request_webpage

  File "yt_dlp/extractor/common.py", line 850, in _request_webpage
  File "yt_dlp/YoutubeDL.py", line 4081, in urlopen
  File "yt_dlp/networking/common.py", line 114, in send
  File "yt_dlp/networking/_helper.py", line 204, in wrapper
  File "yt_dlp/networking/common.py", line 325, in send
  File "yt_dlp/networking/_requests.py", line 348, in _send
yt_dlp.networking.exceptions.HTTPError: HTTP Error 404: Not Found

This is the stable URL to "ARD one" livestream, valid for years meanwhile, and works still fine in Web Browser. MPV fails on it, because it is relying on a yt-dlp hook for stream selection.

bashonly commented 9 months ago

@Mi605 open a new issue please

Mi605 commented 9 months ago

Will do so. Many thanks. Just placed it here after having read the instructions for new issues, I was perfectly convinced you want it being reported this way, even in a closed issue, as a comment.