HTTP Error 503: Service Unavailable on www3.nhk.or.jp

tchwpkgorg commented 1 year ago

Checklist

[*] I'm reporting a broken site support
[*] I've verified that I'm running youtube-dl version 2021.12.17
[*] I've checked that all provided URLs are alive and playable in a browser
[*] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[*] I've searched the bugtracker for similar issues including closed ones

Verbose log


$ ./youtube-dl -v https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2032293/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2032293/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.08.07 [86e3cf5e5] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.11.4 (CPython x86_64 64bit) - Linux-6.4.2-060402-generic-x86_64-with-glibc2.37 - OpenSSL 3.0.8 7 Feb 2023 - glibc 2.37
[debug] exe versions: ffmpeg 5.1.2, ffprobe 5.1.2, rtmpdump 2.4
[debug] Proxy map: {}
[NhkVod] 2032-293: Downloading JSON metadata
[Piksel] nw_vod_v_en_2032_293_20230803113000_01_1691031888: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 503: Service Unavailable (caused by <HTTPError 503: 'Service Unavailable'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/storage/tmp/./youtube-dl/youtube_dl/extractor/common.py", line 665, in _request_webpage
    return self._downloader.urlopen(url_or_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/tmp/./youtube-dl/youtube_dl/YoutubeDL.py", line 2465, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

For some time, youtube-dl no longer works on NHK World, i.e. https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2032293/

It errors as above. Tried the last "stable" version 2021.12.17 and the last "nightly" (2023.08.07).

dirkf commented 1 year ago

Looks like we need to merge https://github.com/ytdl-org/youtube-dl/pull/32213. By all means check out the PR if you wish.

Vangelis66 commented 1 year ago

we need to merge #32213. By all means, check out the PR

Woks basically OK, at least with the video in the OP:

yt-dl -vF "https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2032293/" =>

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', 'https://www3.nhk.or.jp/nhkworld/en/ondemand/video/2032293/']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.08.04.2154
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[NhkVod] 2032-293: Downloading JSON metadata
[Piksel] nw_vod_v_en_2032_293_20230803113000_01_1691031888: Downloading webpage
[Piksel] nw_vod_v_en_2032_293_20230803113000_01_1691031888: Downloading JSON metadata
[Piksel] nw_vod_v_en_2032_293_20230803113000_01_1691031888: Downloading JSON metadata
[Piksel] m2k9n530: Downloading m3u8 information
[Piksel] m2k9n530: Downloading SMIL file
[info] Available formats for m2k9n530:
format code  extension  resolution note
http-383     mp4        320x180     383k video@ 256k, audio@127k, 77.40MiB
hls-392      mp4        unknown     392k
http-392     mp4        320x?       392k
http-663-0   mp4        640x360     663k video@ 536k, audio@127k, 133.55MiB
http-663-1   mp4        640x360     663k video@ 536k, audio@127k, 133.55MiB
hls-678      mp4        unknown     678k
http-678     mp4        640x?       678k
http-1242    mp4        1280x720   1242k video@1115k, audio@127k, 249.57MiB
hls-1271     mp4        unknown    1271k
http-1271    mp4        1280x?     1271k
http-2126    mp4        1280x720   2126k video@1999k, audio@127k, 426.57MiB
http-2177    mp4        1280x?     2177k
http-10268   mp4        1920x1080  10268k video@10268k, audio@  0k, 2.06GiB (best)

... Some of the metadata details in the resolution and note columns are still missing (unknown, ?) or appear incorrect (e.g., is the best format really video-only? video@10268k, audio@ 0k suggests that 😉 ) ...

Vangelis66 commented 1 year ago

... To be fair 😜 , "downstream" aren't doing any better in the metadata department, so I guess it's because of inherent limitations in the NHKWorld+Piksel IEs:

[debug] Sort order given by extractor: tbr
[debug] Formats sorted by: hasvid, ie_pref, tbr, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for m2k9n530:
ID         EXT RESOLUTION |   FILESIZE    TBR PROTO | VCODEC     VBR ACODEC   ABR
---------------------------------------------------------------------------------
http-383   mp4 320x180    |   77.40MiB   383k https | unknown   256k unknown 127k
hls-392    mp4 unknown    |              392k m3u8  | unknown        unknown
http-392   mp4 320x?      |              392k https | unknown        unknown
http-663-0 mp4 640x360    |  133.55MiB   663k https | unknown   536k unknown 127k
http-663-1 mp4 640x360    |  133.55MiB   663k https | unknown   536k unknown 127k
hls-678    mp4 unknown    |              679k m3u8  | unknown        unknown
http-678   mp4 640x?      |              679k https | unknown        unknown
http-1242  mp4 1280x720   |  249.57MiB  1242k https | unknown  1115k unknown 127k
hls-1271   mp4 unknown    |             1272k m3u8  | unknown        unknown
http-1271  mp4 1280x?     |             1272k https | unknown        unknown
http-2126  mp4 1280x720   |  426.57MiB  2126k https | unknown  1999k unknown 127k
http-2177  mp4 1280x?     |             2177k https | unknown        unknown
http-10268 mp4 1920x1080  |    2.06GiB 10268k https | unknown 10268k unknown

R-Gerard commented 11 months ago

Looks like we need to merge https://github.com/ytdl-org/youtube-dl/pull/32213. By all means check out the PR if you wish.

Is there any reason to not merge that PR?

dirkf commented 11 months ago

Comments with successful or other experiences there would incentivise the process. Otherwise it has to wait until I have time to exercise it myself.

ytdl-org / youtube-dl