ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.76k stars 10.07k forks source link

BBC iPlayer 1080p video=12000000.m3u8 #30136

Open moftot opened 3 years ago

moftot commented 3 years ago

Checklist

Description

WRITE DESCRIPTION HERE

I've discovered that modifying the playlist download URL for BBC iPlayer from

...-video=5070000.m3u8...

to

-video=12000000.m3u8

will give you access to BBC iPlayer content in 1080p50! I discovered this through this post:

https://www.avforums.com/threads/is-there-a-way-to-get-iplayer-content-in-1080p.2319869/post-29211339

Not sure how the poster discovered this and not sure whether this is hidden from known media selectors etc. But please consider how this can be incorporated into youtube-dl or please help describing how I can experiment with customizing the Python code myself.

Example BBC iPlayer URL that works:

https://www.bbc.co.uk/iplayer/episode/m0010s0w/panorama-online-abuse-why-do-you-hate-me

get-iplayer finds the following playlist URL:

https://vod-hls-uk.live.cf.md.bbci.co.uk/usp/auth/vod/piff_abr_full_hd/e87ac8-m0010s0v/vf_m0010s0v_d284745f-0e70-4763-a378-3c78576b7a97.ism.hlsv2.ism/vf_m0010s0v_d284745f-0e70-4763-a378-3c78576b7a97.ism.hlsv2-audio_eng_1=128000-video=5070000.m3u8?Expires=1634646622&Signature=j74cyVi0oOcLvl74uJJDTiMcPaZqB1AuiRAMqPiKi2Kar-q7hI5AUokDlQb~fVB8mGu3OjqYBy4SWHnCGbTHH7FyYTaq7F6kJhG2hEU8LRE-RLB~aoqkZcELUw4KYZj7HF8Odcel6ofsYYYtQK2aPctmOnIeJot6M-kQeC2JrtTqj6ZNR2ISmq9LwOeuEx1t5cLN-R6LwvpYCAYwBeFlNPWkXpE8wAcvQI-jI~cLO0tZRnG~hULShJBnG83mMheyBaf2h6CqnAgWIhugT0O2Jzfw3NbDstfq-5INZRGWHe1iIumxQNjK5Vi1NgOempm098mLQC12qq8GNHjXNW-TLQ__&Key-Pair-Id=K2VWLYKQ4HU1FJ

The standard 720p content in .mp4 downloaded with get-iplayer is 1.01GB, ffprobe shows:

Duration: 00:28:59.69, start: 0.000000, bitrate: 5037 kb/s     Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 4895 kb/s, 50 fps, 50 tbr, 90k tbn, 100 tbc (default)     Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 125 kb/s (default)

The modified playlist URL for 1080p is:

https://vod-hls-uk.live.cf.md.bbci.co.uk/usp/auth/vod/piff_abr_full_hd/e87ac8-m0010s0v/vf_m0010s0v_d284745f-0e70-4763-a378-3c78576b7a97.ism.hlsv2.ism/vf_m0010s0v_d284745f-0e70-4763-a378-3c78576b7a97.ism.hlsv2-audio_eng_1=128000-video=12000000.m3u8?Expires=1634646622&Signature=j74cyVi0oOcLvl74uJJDTiMcPaZqB1AuiRAMqPiKi2Kar-q7hI5AUokDlQb~fVB8mGu3OjqYBy4SWHnCGbTHH7FyYTaq7F6kJhG2hEU8LRE-RLB~aoqkZcELUw4KYZj7HF8Odcel6ofsYYYtQK2aPctmOnIeJot6M-kQeC2JrtTqj6ZNR2ISmq9LwOeuEx1t5cLN-R6LwvpYCAYwBeFlNPWkXpE8wAcvQI-jI~cLO0tZRnG~hULShJBnG83mMheyBaf2h6CqnAgWIhugT0O2Jzfw3NbDstfq-5INZRGWHe1iIumxQNjK5Vi1NgOempm098mLQC12qq8GNHjXNW-TLQ__&Key-Pair-Id=K2VWLYKQ4HU1FJ

The 1080p version in .mp4 downloaded with youtube-dl using the modified playlist URL is 1.79GB, ffprobe shows:

Duration: 00:28:59.69, start: 0.000000, bitrate: 8882 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 8742 kb/s, 50 fps, 50 tbr, 90k tbn, 100 tbc (default) Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 125 kb/s (default)

dirkf commented 3 years ago

BBC has several different 'media selector' domains, each domain serving a certain set of 'media sets', and each media set targeted at a different application. Fetching a URL whose path depends on the media set name and the programme ID (vpid) gives a playlist for the programme with the various formats for that media set.

The BBC extractor currently uses the open.live.bbc.co.uk media selector with iptv-all and pc media sets. Other sets that can be used include legacy-iptv-all and iptv-uhd: the latter offers the resolution wanted by OP but is only supported by authenticated media selector domains. One way of getting authenticated is to extract a client certificate from a supported set-top box and present it using --client-certificate as added in PR #29590.

yt-dl maintainers deprecate multiple requests to cover a wide range of media sets combined from several media selectors, so this might be a useful shortcut, but how would we guess that the right value is 12000000 rather than say, 11600000 for a particular programme ID on a particular day?

In the current extractor, one could patch this in (at the end of the _process_media_selector() method): [updated]

--- old/youtube_dl/extractor/bbc.py
+++ new/youtube_dl/extractor/bbc.py
@@ -445,6 +445,14 @@
                         formats.append(fmt)
             elif kind == 'captions':
                 subtitles = self.extract_subtitles(media, programme_id)
+        formats += [
+                {'url': fmt['url'].replace('-video=5070000.m3u8', '-video=12000000.m3u8'),
+                 'height': 1080, 'vbr': 12000,
+                 'abr': fmt.get('abr'),
+                 'protocol': fmt.get('protocol'),
+                 'preference': fmt.get('preference'),
+                 'format_id': fmt.get('format_id', 'hack') + '-really12M',
+                } for fmt in formats if '-video=5070000.m3u8' in fmt.get('url', '')]
         return formats, subtitles

     def _download_playlist(self, playlist_id):

Another option for acquiring 'backup' content at >720p for those with a TV licence in range of BBC DVB-T2 broadcasts is a USB TV receiver dongle, but Freeview(xxx) receivers are required to encrypt the content on disk.

moftot commented 3 years ago

Thank you very much for your prompt, detailed and technical reply, extremely helpful. I'll experiment with this on the weekend.

I guess 5070000 is the "bitrate class" for 720p and 12000000 is the "bitrate class" for 1080p. Since 01 Aug 2020 I've successfully downloaded 149 programs through the 5070000 "bitrate class" but so far I've only tried and downloaded 3 programs (all successfully) through the 12000000 "bitrate class".

Perhaps the 12000000 "bitrate class" is hidden. Much like on RaiPlay where the best quality is usually the 720p 2400 "bitrate class" but often there's a hidden 1080p 5000 "bitrate class".

I'll keep you posted.

dirkf commented 3 years ago

It may be that the CDN supplier automatically supplies the 12M version. Is there any visible difference between the versions that you tested? Or perhaps BBC supplies 12M which gets transcoded for the other resolutions? Or it's a test? Etc. I don't see any sign of the 12M versions in the advertised format lists.

Higher definition (eg 4k, 18M, H.265) downloads are available as in the iptv-uhd media set, but may not be available for all shows. Eg: your example Panorama episode, PID m0010s0w, vPID, m0010s0v, has no advertised UHD formats, only <=5M07 rates. But Blue Planet II S01E01, PID p04thmv7, vPID p06mqywx, does have them (through an authenticated media selector).

You might find the BBC Media Availability Tool (JS required) interesting, especially if you change the playlist URLs from .../format/xml to .../format/json.

edent commented 3 years ago

@dirkf I'm a little confused by that patch. If I apply it, I get

SyntaxError: closing parenthesis ')' does not match opening parenthesis '[' on line 449

Am I missing something obvious? Thanks

dirkf commented 3 years ago

If you mean PR #29590, better to discuss there. However you'll need to provide details of exactly what you did. I just applied https://patch-diff.githubusercontent.com/raw/ytdl-org/youtube-dl/pull/29590.diff to a clean 2021-06-06 git repo (patch -p 1 -i ~/29590.diff) and had no errors when running the result.

edent commented 3 years ago

I mean the patch above to enable 1080p in _process_media_selector() - but I'll keep digging. Thanks,

edent commented 3 years ago

Aha, here's a quick and dirty way to force 1080p on iPlayer:

for fmt in formats:
           if '-video=5070000.m3u8' in fmt.get('url', ''):
              formats = [
                {'url': fmt['url'].replace('-video=5070000.m3u8', '-video=12000000.m3u8'),
                 'height': 1080, 'vbr': 12000,
                 'abr': fmt.get('abr'),
                 'protocol': fmt.get('protocol'),
                 'preference': fmt.get('preference'),
                 'format_id': fmt.get('format_id', 'hack') + '-really12M',
                }, fmt]

Saves the end file as a .m3u8 - but a quick rename to .mp4 fixes it. Now... How to get something better than stereo? ☺

dirkf commented 3 years ago

Sorry, I was being obtuse. You're right: somehow I managed to paste , fmt) into the diff. If you try the updated patch above it should be more useful.

nomis commented 3 years ago

That 1080p version of Panorama is only marginally better than the 720p version. It looks like upscaled 720p content using a higher bitrate (that doesn't even reach the advertised 12Mbit/s).

If you try this on Blue Planet II S01E01 you get a playlist with no video.

dirkf commented 3 years ago

Seems like my first guess might have been right:

the CDN supplier automatically supplies the 12M version

The UHD formats for BP S01E01 are all DASH MP4/H.265.

macmenot commented 2 years ago

A slightly improved version of the patch listed above that clones the fmt dict and just overrides/updates the differing attributes:

--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@@ -445,6 +445,20 @@ class BBCCoUkIE(InfoExtractor):
                         formats.append(fmt)
             elif kind == 'captions':
                 subtitles = self.extract_subtitles(media, programme_id)
+
+        # add 1080p equivalent of 720p version if found
+        for fmt in formats:
+            if '-video=5070000.m3u8' not in fmt.get('url', ''):
+                continue
+            fhd = dict(fmt)
+            fhd['url'] = fmt['url'].replace('-video=5070000.m3u8', '-video=12000000.m3u8')
+            fhd['height'] = 1080
+            fhd['tbr'] = fmt['abr'] + 12000
+            fhd['vbr'] = 12000
+            fhd['width'] = 1920
+            fhd['format_id'] = fmt['format_id'].replace(str(int(fmt['tbr'])), str(int(fhd['tbr'])))
+            formats.append(fhd)
+
         return formats, subtitles

     def _download_playlist(self, playlist_id):

This makes the entry show up in --list-formats correctly and writes directly to a .mp4 file etc.

macmenot commented 2 years ago

It is still not clear to me why this 1080p format doesn't seem to show up in any of the mediaselector responses for a given vpid? Even checking iptv-all and iptv-uhd on the securegate domain it is not found.

dirkf commented 2 years ago

Based on this, isn't the 1080p format just an upscaled artefact of the CDN arrangement?

Kudos for the improved patch, though.

macmenot commented 2 years ago

Yeah hard to tell. Would have to eyeball side-by-side to determine if it's upscaled from 720p in all cases, or whether that's dependant on the source material.

Picking an arbitrary film, the avg video bitrate was ~9830 kb/s which was double the 720p version at least.

macmenot commented 2 years ago

Based on https://www.bbc.co.uk/iplayer/help/questions/supported-devices/1080-streams I am also assuming that some devices are getting served the 1080p URLs, perhaps based on User-Agent identifier or something

dirkf commented 2 years ago

But aren't those just the media set iptv-uhd formats? No, because those are all H.265.

Perhaps the iPlayer JS running on a 1080p-supported device knows how to pick up the secret, possibly upscaled, 1080p streams so that it can be claimed to be showing 1080p all the time, also avoiding multiple client-side upscaling or resolution switching?

macmenot commented 2 years ago

well the iptv-uhd formats are only available on a very few select videos, and whilst they do have 1080p variants those are more clearly lower bitrate downscales from the 2160p to use when network bandwidth is limited:

e.g.,

[info] Available formats for xxxxxxxx:
format code                                                                       extension  resolution note
mf_akamai_uhd-f0dc4db3-4908-92ca-9a7d-5718b84569c5_audio=192000_dur=3840-0        m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai_uhd-f0dc4db3-4908-92ca-9a7d-5718b84569c5_audio=192000_dur=3840-1        m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_cloudfront_uhd-f0dc4db3-4908-92ca-9a7d-5718b84569c5_audio=192000_dur=3840-0    m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_cloudfront_uhd-f0dc4db3-4908-92ca-9a7d-5718b84569c5_audio=192000_dur=3840-1    m4a        audio only [eng] DASH audio  190k , m4a_dash container, mp4a.40.2 (48000Hz)
mf_akamai_uhd-815ce70a-2144-858b-8863-b0257de49fd2_video=1100000_dur=3840-0       mp4        768x432    DASH video 1652k , mp4_dash container, hev1.2.4.L90.00.00.90, 25fps, video only
mf_akamai_uhd-815ce70a-2144-858b-8863-b0257de49fd2_video=1100000_dur=3840-1       mp4        768x432    DASH video 1652k , mp4_dash container, hev1.2.4.L90.00.00.90, 25fps, video only
mf_cloudfront_uhd-815ce70a-2144-858b-8863-b0257de49fd2_video=1100000_dur=3840-0   mp4        768x432    DASH video 1652k , mp4_dash container, hev1.2.4.L90.00.00.90, 25fps, video only
mf_cloudfront_uhd-815ce70a-2144-858b-8863-b0257de49fd2_video=1100000_dur=3840-1   mp4        768x432    DASH video 1652k , mp4_dash container, hev1.2.4.L90.00.00.90, 25fps, video only
mf_akamai_uhd-069d92d8-2b7a-ebef-309e-ed5432e5a861_video=2000000_dur=3840-0       mp4        960x540    DASH video 2851k , mp4_dash container, hev1.2.4.L93.00.00.90, 25fps, video only
mf_akamai_uhd-069d92d8-2b7a-ebef-309e-ed5432e5a861_video=2000000_dur=3840-1       mp4        960x540    DASH video 2851k , mp4_dash container, hev1.2.4.L93.00.00.90, 25fps, video only
mf_cloudfront_uhd-069d92d8-2b7a-ebef-309e-ed5432e5a861_video=2000000_dur=3840-0   mp4        960x540    DASH video 2851k , mp4_dash container, hev1.2.4.L93.00.00.90, 25fps, video only
mf_cloudfront_uhd-069d92d8-2b7a-ebef-309e-ed5432e5a861_video=2000000_dur=3840-1   mp4        960x540    DASH video 2851k , mp4_dash container, hev1.2.4.L93.00.00.90, 25fps, video only
mf_akamai_uhd-825ea854-ea36-c11e-5a4b-f5db069fa44b_video=3000000_dur=3840-0       mp4        1280x720   DASH video 4010k , mp4_dash container, hev1.2.4.L120.00.00.90, 25fps, video only
mf_akamai_uhd-825ea854-ea36-c11e-5a4b-f5db069fa44b_video=3000000_dur=3840-1       mp4        1280x720   DASH video 4010k , mp4_dash container, hev1.2.4.L120.00.00.90, 25fps, video only
mf_cloudfront_uhd-825ea854-ea36-c11e-5a4b-f5db069fa44b_video=3000000_dur=3840-0   mp4        1280x720   DASH video 4010k , mp4_dash container, hev1.2.4.L120.00.00.90, 25fps, video only
mf_cloudfront_uhd-825ea854-ea36-c11e-5a4b-f5db069fa44b_video=3000000_dur=3840-1   mp4        1280x720   DASH video 4010k , mp4_dash container, hev1.2.4.L120.00.00.90, 25fps, video only
mf_akamai_uhd-50e7b5e7-9787-e47d-41d9-12e7162e858a_video=4600000_dur=3840-0       mp4        1600x900   DASH video 6411k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-50e7b5e7-9787-e47d-41d9-12e7162e858a_video=4600000_dur=3840-1       mp4        1600x900   DASH video 6411k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-50e7b5e7-9787-e47d-41d9-12e7162e858a_video=4600000_dur=3840-0   mp4        1600x900   DASH video 6411k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-50e7b5e7-9787-e47d-41d9-12e7162e858a_video=4600000_dur=3840-1   mp4        1600x900   DASH video 6411k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-c8cc7b90-e7a6-5a1d-9149-77171098823b_video=6300000_dur=3840-0       mp4        1920x1080  DASH video 8223k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-c8cc7b90-e7a6-5a1d-9149-77171098823b_video=6300000_dur=3840-1       mp4        1920x1080  DASH video 8223k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-c8cc7b90-e7a6-5a1d-9149-77171098823b_video=6300000_dur=3840-0   mp4        1920x1080  DASH video 8223k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-c8cc7b90-e7a6-5a1d-9149-77171098823b_video=6300000_dur=3840-1   mp4        1920x1080  DASH video 8223k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-1f589e04-db65-150e-fd47-12cd9ae0defc_video=8080000_dur=3840-0       mp4        2560x1440  DASH video 10346k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-1f589e04-db65-150e-fd47-12cd9ae0defc_video=8080000_dur=3840-1       mp4        2560x1440  DASH video 10346k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-1f589e04-db65-150e-fd47-12cd9ae0defc_video=8080000_dur=3840-0   mp4        2560x1440  DASH video 10346k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_cloudfront_uhd-1f589e04-db65-150e-fd47-12cd9ae0defc_video=8080000_dur=3840-1   mp4        2560x1440  DASH video 10346k , mp4_dash container, hev1.2.4.L150.00.00.90, 25fps, video only
mf_akamai_uhd-e1aac700-b2e8-d0f5-f2db-b29a35bb2ac8_video=12640000_dur=3840-0      mp4        3200x1800  DASH video 16260k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-e1aac700-b2e8-d0f5-f2db-b29a35bb2ac8_video=12640000_dur=3840-1      mp4        3200x1800  DASH video 16260k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-e1aac700-b2e8-d0f5-f2db-b29a35bb2ac8_video=12640000_dur=3840-0  mp4        3200x1800  DASH video 16260k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-e1aac700-b2e8-d0f5-f2db-b29a35bb2ac8_video=12640000_dur=3840-1  mp4        3200x1800  DASH video 16260k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-f31854f1-293e-9f11-f7ad-80e00033b951_video=18160000_dur=3840-0      mp4        3840x2160  DASH video 21744k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_akamai_uhd-f31854f1-293e-9f11-f7ad-80e00033b951_video=18160000_dur=3840-1      mp4        3840x2160  DASH video 21744k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-f31854f1-293e-9f11-f7ad-80e00033b951_video=18160000_dur=3840-0  mp4        3840x2160  DASH video 21744k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only
mf_cloudfront_uhd-f31854f1-293e-9f11-f7ad-80e00033b951_video=18160000_dur=3840-1  mp4        3840x2160  DASH video 21744k , mp4_dash container, hev1.2.4.L153.00.00.90, 25fps, video only (best)

And whilst they list supported devices and the limited set of programmes as part of the UHD trials pages, the 1080p page I linked earlier just states "1080p streams for both live and on-demand" and doesn't seem to suggest there's any limited set of programmes that those are eligible for. Similarly I don't see why they'd advertise these special 1080p devices if they were just going to serve them upscaled content.

nomis commented 2 years ago

Similarly I don't see why they'd advertise these special 1080p devices if they were just going to serve them upscaled content.

This is likely to be a quirk of the CDN infrastructure. We know the real HD streams require a TLS client certificate to access.

These 1080p streams are just slightly higher bitrate versions of the 720p content. It's not worth the extra bandwidth for a very small increase in quality or to lose the video altogether on some programmes (that would need to be mitigated automatically).

All that routinely using this is going to do is get that playlist bitrate properly disabled on the CDN infrastructure.

macmenot commented 2 years ago

This is likely to be a quirk of the CDN infrastructure. We know the real HD streams require a TLS client certificate to access.

I'm testing with a certificate against sercuregate, hence why I can list the 2160p formats where they're available, but I still don't see any 1080p versions in either the iptv-all or iptv-uhd mediasets for any of the non-UHD programmes

dirkf commented 2 years ago

A strange thing is that the playlists served by both iptv-all and legacy-iptv-all for vpids m0010s0v (not on UHD) and p06mqywx (on UHD) all claim 1920x1080 in the playlist metadata but actually offer max 720p.

lewisflakes commented 2 years ago

A slightly improved version of the patch listed above that clones the fmt dict and just overrides/updates the differing attributes:

--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@@ -445,6 +445,20 @@ class BBCCoUkIE(InfoExtractor):
                         formats.append(fmt)
             elif kind == 'captions':
                 subtitles = self.extract_subtitles(media, programme_id)
+
+        # add 1080p equivalent of 720p version if found
+        for fmt in formats:
+            if '-video=5070000.m3u8' not in fmt.get('url', ''):
+                continue
+            fhd = dict(fmt)
+            fhd['url'] = fmt['url'].replace('-video=5070000.m3u8', '-video=12000000.m3u8')
+            fhd['height'] = 1080
+            fhd['tbr'] = fmt['abr'] + 12000
+            fhd['vbr'] = 12000
+            fhd['width'] = 1920
+            fhd['format_id'] = fmt['format_id'].replace(str(int(fmt['tbr'])), str(int(fhd['tbr'])))
+            formats.append(fhd)
+
         return formats, subtitles

     def _download_playlist(self, playlist_id):

This makes the entry show up in --list-formats correctly and writes directly to a .mp4 file etc.

I'm on a mac, and a noob, where do I find this bbc.py? I've added the above into bbc.py located at /library/frameworks/python.framework/versions/3.9/lib/python3.9/site-packages/youtube-dl/extractor/bbc.py (see screenshot below) But it doesn't work, what am I doing wrong?

screenshot
dirkf commented 2 years ago

If your yt-dl was installed with pip or similar, that looks like the right place.

Otherwise if ls -l "$(which youtube-dl)" shows a large (~2MB) file, you also have (and are running) the self-extracting archive version, so get rid of it and (re-)install with pip/pip3 (in this case).

After copying the updated extractor, run youtube-dl --version as admin (with privilege to write to subdirectories of /library/frameworks/python.framework/versions/3.9/lib/python3.9/site-packages/youtube-dl) to overwrite any existing bytecode-compiled version.

lewisflakes commented 2 years ago

If your yt-dl was installed with pip or similar, that looks like the right place.

Otherwise if ls -l "$(which youtube-dl)" shows a large (~2MB) file, you also have (and are running) the self-extracting archive version, so get rid of it and (re-)install with pip/pip3 (in this case).

After copying the updated extractor, run youtube-dl --version as admin (with privilege to write to subdirectories of /library/frameworks/python.framework/versions/3.9/lib/python3.9/site-packages/youtube-dl) to overwrite any existing bytecode-compiled version.

Thank you. The youtube-dl file was indeed around 2MB, so I removed it and reinstalled with pip3. I can get 1080p now!

jimbo2018 commented 2 years ago

I have ytdl on my windows laptop. Do I just change the code in the BBC file? Thx

dirkf commented 2 years ago

It depends which type of installation you have. You can update an expanded installation, as with pip, quite easily, as above, but typically Windows users have a self-extracting executable (~8MB) bundled with a version of Python 3 that is harder to patch.

jimbo2018 commented 2 years ago

Thank you. But dies anyone know if English regions of BBC1 have an HD stream yet? Thanks

The-Torrent commented 2 years ago

anyone got an easier tutorial to follow?

madindehead commented 1 year ago

A slightly improved version of the patch listed above that clones the fmt dict and just overrides/updates the differing attributes:

--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@@ -445,6 +445,20 @@ class BBCCoUkIE(InfoExtractor):
                         formats.append(fmt)
             elif kind == 'captions':
                 subtitles = self.extract_subtitles(media, programme_id)
+
+        # add 1080p equivalent of 720p version if found
+        for fmt in formats:
+            if '-video=5070000.m3u8' not in fmt.get('url', ''):
+                continue
+            fhd = dict(fmt)
+            fhd['url'] = fmt['url'].replace('-video=5070000.m3u8', '-video=12000000.m3u8')
+            fhd['height'] = 1080
+            fhd['tbr'] = fmt['abr'] + 12000
+            fhd['vbr'] = 12000
+            fhd['width'] = 1920
+            fhd['format_id'] = fmt['format_id'].replace(str(int(fmt['tbr'])), str(int(fhd['tbr'])))
+            formats.append(fhd)
+
         return formats, subtitles

     def _download_playlist(self, playlist_id):

This makes the entry show up in --list-formats correctly and writes directly to a .mp4 file etc.

@macmenot

I've tried this with my yt-dlp. The 1080p shows up when listing formats and it seems to start downloading the 1080p. However, it doesn't actually download the 1080p one because it doesn't even up being the correct file size or bit rate.

I want to say it's caused by the line if '-video=5070000.m3u8' not in fmt.get('url', ''): because the 5070000.m3u8 stream is available, so surely the next bit of code doesn't get implemented? Or am I missing something?

I'm convinced that it doesn't work and that the certificate maybe becomes an issue too? Honestly, what a faff.

dirkf commented 1 year ago

When the loop over formats finds a format url containing -video=5070000.m3u8, it copies the format (fhd = dict(fmt)), updates the copied format with the putative 1080p data, and adds the copied format to the format list.

No-one is guaranteeing that any of these putative 1080p formats is available or valid or better quality than the known 720p formats, or even actually 1080p.

If using the patch with Python 2, or anyway, use compat_str, and the int(..) coercion isn't required given the lines above:

+            fhd['format_id'] = fmt['format_id'].replace(compat_str(fmt['tbr']), compat_str(fhd['tbr']))
zamroni111 commented 7 months ago

This is likely to be a quirk of the CDN infrastructure. We know the real HD streams require a TLS client certificate to access.

I'm testing with a certificate against sercuregate, hence why I can list the 2160p formats where they're available, but I still don't see any 1080p versions in either the iptv-all or iptv-uhd mediasets for any of the non-UHD programmes

can you share the certificate file?

dirkf commented 7 months ago

Don't post any such thing here, though. What people do elsewhere is between themselves and any interested legal representatives.