search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.34k stars 10.03k forks source link

[nickjr.com] Broken after site update #31056

Open LazyRips opened 2 years ago

LazyRips commented 2 years ago

Verbose log

C:\Users\Desktop\youtube-dl>youtube-dl.exe -F https://www.nickjr.com/episodes/bzttyk/blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19 -v --geo-bypass-country US
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', 'https://www.nickjr.com/episodes/bzttyk/blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19', '-v', '--geo-bypass-country', 'US']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg git-2019-10-31-7c872df, ffprobe git-2019-10-31-7c872df
[debug] Proxy map: {}
[debug] Using fake IP 6.165.77.245 (US) as X-Forwarded-For.
[generic] blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19: Requesting header
WARNING: Falling back on generic information extractor.
[generic] blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19: Downloading webpage
[generic] blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19: Extracting information
ERROR: Unsupported URL: https://www.nickjr.com/episodes/bzttyk/blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 815, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\YoutubeDL.py", line 836, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\common.py", line 534, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpupik7c6w\build\youtube_dl\extractor\generic.py", line 3489, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.nickjr.com/episodes/bzttyk/blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19

Description

Fails to get video. nickjr and nick.com recently updated their site, so video links also have different names. I believe they just changed the site structure, so if the nick module just gets updated for the new structure it should work again.

dirkf commented 2 years ago

Working in yt-dlp: the extractor needs to be back-ported, and also the MTV extractor on which it depends needs to be updated, eg PR #30697. Also, duplicate of #30848.

However, in both cases this "episode" has three videos: is that what's expected?

$ python -m youtube_dl -v -F 'https://www.nickjr.com/episodes/bzttyk/blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19' --geo-bypass-country US
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'https://www.nickjr.com/episodes/bzttyk/blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19', u'--geo-bypass-country', u'US']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Git HEAD: 44f73880f
[debug] Python version 2.7.17 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP 6.42.79.41 (US) as X-Forwarded-For.
[nick.com] blaze-and-the-monster-machines-the-treasure-of-the-broken-key-a-musical-adventure-season-6-ep-19: Downloading webpage
[nick.com] f6654965-8f7b-11ec-9b1b-0e40cf2fc285: Downloading info
[nick.com] f2f2f808-edaf-11ec-9b1b-0e40cf2fc285: Extracting information
[nick.com] f2f2f808-edaf-11ec-9b1b-0e40cf2fc285: Downloading video urls
[nick.com] f2f2f808-edaf-11ec-9b1b-0e40cf2fc285: Downloading m3u8 information
[nick.com] f3080eaa-edaf-11ec-9b1b-0e40cf2fc285: Extracting information
[nick.com] f3080eaa-edaf-11ec-9b1b-0e40cf2fc285: Downloading video urls
[nick.com] f3080eaa-edaf-11ec-9b1b-0e40cf2fc285: Downloading m3u8 information
[nick.com] f31d2bd3-edaf-11ec-9b1b-0e40cf2fc285: Extracting information
[nick.com] f31d2bd3-edaf-11ec-9b1b-0e40cf2fc285: Downloading video urls
[nick.com] f31d2bd3-edaf-11ec-9b1b-0e40cf2fc285: Downloading m3u8 information
[download] Downloading playlist: The Treasure of the Broken Key: A Musical Adventure
[nick.com] playlist The Treasure of the Broken Key: A Musical Adventure: Collected 3 video ids (downloading 3 of them)
[download] Downloading video 1 of 3
[info] Available formats for f2f2f808-edaf-11ec-9b1b-0e40cf2fc285:
format code  extension  resolution note
hls-362      mp4        384x216     362k , avc1.4D401E, 23.976fps, mp4a.40.2
hls-587      mp4        512x288     587k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1134     mp4        640x360    1134k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1642     mp4        768x432    1642k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-2265     mp4        960x540    2265k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-3287     mp4        1280x720   3287k , avc1.640028, 23.976fps, mp4a.40.2
hls-5293     mp4        1920x1080  5293k , avc1.640028, 23.976fps, mp4a.40.2 (best)
[download] Downloading video 2 of 3
[info] Available formats for f3080eaa-edaf-11ec-9b1b-0e40cf2fc285:
format code  extension  resolution note
hls-348      mp4        384x216     348k , avc1.4D401E, 23.976fps, mp4a.40.2
hls-557      mp4        512x288     557k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1050     mp4        640x360    1050k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1434     mp4        768x432    1434k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-2019     mp4        960x540    2019k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-2931     mp4        1280x720   2931k , avc1.640028, 23.976fps, mp4a.40.2
hls-4893     mp4        1920x1080  4893k , avc1.640028, 23.976fps, mp4a.40.2 (best)
[download] Downloading video 3 of 3
[info] Available formats for f31d2bd3-edaf-11ec-9b1b-0e40cf2fc285:
format code  extension  resolution note
hls-280      mp4        384x216     280k , avc1.4D401E, 23.976fps, mp4a.40.2
hls-326      mp4        512x288     326k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-462      mp4        640x360     462k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-556      mp4        768x432     556k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-759      mp4        960x540     759k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1117     mp4        1280x720   1117k , avc1.640028, 23.976fps, mp4a.40.2
hls-2197     mp4        1920x1080  2197k , avc1.640028, 23.976fps, mp4a.40.2 (best)
[download] Finished downloading playlist: The Treasure of the Broken Key: A Musical Adventure
$
LazyRips commented 2 years ago

Thanks, will try yt-dlp.

nick has episodes in segments, so yes three episodes is expected. :)

gamer191 commented 2 years ago

Also, duplicate of #30848

@dirkf out of interest, is there a reason why this issue, and other duplicates, are being left open? I feel like it makes youtube-dl's issue tracker more complicated

dirkf commented 2 years ago

Just that there was still a discussion pertinent to the specific case in the issue and I hadn't worked out whether that was concluded.

Actually the site (and other pages mediated by the MTV extractor) raised the question of how multi_video results are supposed to be handled.

In yt-dl playlist and multi_video seem to be treated the same regardless of the documented difference in semantics.

yt-dlp can concatenate a multi-file download from a multi_video result if the parts are compatible: this seems like the right thing to do.

In this case the info_dict should mark each episode of three items as multi_video.

I'm not sufficiently familiar with this site or the MTV extractor to know whether the relevant routine of the MTV extractor should always returns a multi_video result rather than a playlist. The relevant line has the comment TODO: should be multi-video (sic), leading one to question why a simple change like that below wasn't made.

--- old/youtube_dl/extractor/mtv.py
+++ new/youtube_dl/extractor/mtv.py
@@ -248,9 +248,12 @@ class MTVServicesInfoExtractor(InfoExtractor):
             if info:
                 entries.append(info)

-        # TODO: should be multi-video
-        return self.playlist_result(
+        if len(entries) == 1:
+            return entries[0]
+        result = self.playlist_result(
             entries, playlist_title=title, playlist_description=description)
+        result['_type'] = 'multi_video'
+        return result

     def _extract_triforce_mgid(self, webpage, data_zone=None, video_id=None):
         triforce_feed = self._parse_json(self._search_regex(
Sipherdrakon commented 1 year ago

MTV Extractor is now broken for mtv.com, so if this multi-video option can be fleshed out, might be a good time to try it. I was reading in yt-dlp that the API is no longer working which is causing the MTV.com part of the extractor to fail. It still works for vh1 and everything else, just mtv.com that stopped working.

dirkf commented 1 year ago

Sites that use an XML feed are working, which includes the test back-port of Nick:

$ python3.9 -m youtube_dl -F 'https://www.nickjr.com/episodes/28csrq/wallykazam-sticky-picnic-season-2-ep-15' -v --geo-bypass-country US
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', 'https://www.nickjr.com/episodes/28csrq/wallykazam-sticky-picnic-season-2-ep-15', '-v', '--geo-bypass-country', 'US']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 774b346f9
[debug] Python version 3.9.15 (CPython) - Linux-4.4.0-210-generic-i686-with-glibc2.23
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP 6.178.155.63 (US) as X-Forwarded-For.
[nick.com] wallykazam-sticky-picnic-season-2-ep-15: Downloading webpage
[nick.com] f5abcb94-76eb-11e5-8ff7-0026b9414f30: Downloading info
[nick.com] 93910c88-0f0b-4381-b910-b4506b79f62b: Extracting information
[nick.com] 93910c88-0f0b-4381-b910-b4506b79f62b: Downloading video urls
[nick.com] 93910c88-0f0b-4381-b910-b4506b79f62b: Downloading m3u8 information
[nick.com] 4baa6285-53ac-4c09-98f4-8e56a7b6e51f: Extracting information
[nick.com] 4baa6285-53ac-4c09-98f4-8e56a7b6e51f: Downloading video urls
[nick.com] 4baa6285-53ac-4c09-98f4-8e56a7b6e51f: Downloading m3u8 information
[nick.com] e501fc71-a9fa-4417-bed7-e9100348e217: Extracting information
[nick.com] e501fc71-a9fa-4417-bed7-e9100348e217: Downloading video urls
[nick.com] e501fc71-a9fa-4417-bed7-e9100348e217: Downloading m3u8 information
[nick.com] 322add57-8ff3-48a9-a96c-238ab4de6b35: Extracting information
[nick.com] 322add57-8ff3-48a9-a96c-238ab4de6b35: Downloading video urls
[nick.com] 322add57-8ff3-48a9-a96c-238ab4de6b35: Downloading m3u8 information
[download] Downloading playlist: Sticky Picnic
[nick.com] playlist Sticky Picnic: Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[info] Available formats for 93910c88-0f0b-4381-b910-b4506b79f62b:
format code  extension  resolution note
hls-344      mp4        384x216     344k , avc1.4D401E, 23.976fps, mp4a.40.2
hls-548      mp4        512x288     548k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-987      mp4        640x360     987k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1290     mp4        768x432    1290k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1867     mp4        960x540    1867k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-2716     mp4        1280x720   2716k , avc1.640028, 23.976fps, mp4a.40.2
hls-4694     mp4        1920x1080  4694k , avc1.640028, 23.976fps, mp4a.40.2 (best)
[download] Downloading video 2 of 4
[info] Available formats for 4baa6285-53ac-4c09-98f4-8e56a7b6e51f:
format code  extension  resolution note
hls-341      mp4        384x216     341k , avc1.4D401E, 23.976fps, mp4a.40.2
hls-547      mp4        512x288     547k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-852      mp4        640x360     852k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1080     mp4        768x432    1080k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1535     mp4        960x540    1535k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-2183     mp4        1280x720   2183k , avc1.640028, 23.976fps, mp4a.40.2
hls-3871     mp4        1920x1080  3871k , avc1.640028, 23.976fps, mp4a.40.2 (best)
[download] Downloading video 3 of 4
[info] Available formats for e501fc71-a9fa-4417-bed7-e9100348e217:
format code  extension  resolution note
hls-341      mp4        384x216     341k , avc1.4D401E, 23.976fps, mp4a.40.2
hls-488      mp4        512x288     488k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-634      mp4        640x360     634k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-753      mp4        768x432     753k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1066     mp4        960x540    1066k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-1499     mp4        1280x720   1499k , avc1.640028, 23.976fps, mp4a.40.2
hls-2751     mp4        1920x1080  2751k , avc1.640028, 23.976fps, mp4a.40.2 (best)
[download] Downloading video 4 of 4
[info] Available formats for 322add57-8ff3-48a9-a96c-238ab4de6b35:
format code  extension  resolution note
hls-267      mp4        384x216     267k , avc1.4D401E, 23.976fps, mp4a.40.2
hls-322      mp4        512x288     322k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-417      mp4        640x360     417k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-479      mp4        768x432     479k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-632      mp4        960x540     632k , avc1.4D401F, 23.976fps, mp4a.40.2
hls-832      mp4        1280x720    832k , avc1.640028, 23.976fps, mp4a.40.2
hls-1385     mp4        1920x1080  1385k , avc1.640028, 23.976fps, mp4a.40.2 (best)
[download] Finished downloading playlist: Sticky Picnic
$

We can make the extractor return multi_video for these sites (as above), but the core yt-dl processing would have to be changed to join the parts automatically.

MTV.com now has a JSON feed, which needs new handling like this:

$ python3.9 -m youtube_dl -F 'https://www.mtv.com/episodes/g230me/teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2' -v --geo-bypass-country US
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', 'https://www.mtv.com/episodes/g230me/teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2', '-v', '--geo-bypass-country', 'US']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 774b346f9
[debug] Python version 3.9.15 (CPython) - Linux-4.4.0-210-generic-i686-with-glibc2.23
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP 6.207.227.45 (US) as X-Forwarded-For.
[mtv] teen-mom-the-next-chapter-you-go-do-you-season-1-ep-2: Downloading webpage
[mtv] 0f294ac5-27d4-11ed-ad8e-0e40cf2fc285: Downloading JSON metadata
[mtv] 0f294ac5-27d4-11ed-ad8e-0e40cf2fc285: Downloading m3u8 information
[info] Available formats for 0f294ac5-27d4-11ed-ad8e-0e40cf2fc285:
format code                  extension  resolution note
hls-audio0-American_English  mp4        audio only 
hls-277                      mp4        384x216     277k , avc1.4d401e, 23.974fps, video only
hls-449                      mp4        512x288     449k , avc1.4d401f, 23.974fps, video only
hls-866                      mp4        640x360     866k , avc1.4d401f, 23.974fps, video only
hls-1156                     mp4        768x432    1156k , avc1.4d401f, 23.974fps, video only
hls-1752                     mp4        960x540    1752k , avc1.4d401f, 23.974fps, video only
hls-2698                     mp4        1280x720   2698k , avc1.640028, 23.974fps, video only
hls-4944                     mp4        1920x1080  4944k , avc1.640028, 23.974fps, video only (best)
$

While there are no combined streams, they aren't multi-part either.

PR #30967 needs more work.