search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.98k stars 10.01k forks source link

Previously supported videos now unsupported after site update UKTV UKTVPLAY #17909

Closed Rencroft closed 3 years ago

Rencroft commented 6 years ago

Hi,

C:\Program Files (x86)\youtubedl>youtube-dl --version
2018.10.05

C:\Program Files (x86)\youtubedl>youtube-dl.exe -o -v "https://uktvplay.uktv.co.
uk/shows/casualty/watch-online/5839038888001"
[generic] 5839038888001: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 5839038888001: Downloading webpage
[generic] 5839038888001: Extracting information
ERROR: Unsupported URL: https://uktvplay.uktv.co.uk/shows/casualty/watch-online/
5839038888001

C:\Program Files (x86)\youtubedl>
...
<end of log>

Support for downloading these videos was working but the site hosting them was updated and now the unsupported url error is given. The URLs for the videos have not changed format so hopefully it will be possible/easy to fix support for these videos. Note a free account and login is needed to play the videos in a browser but youtube-dl was still previously able to download them without login before the site update.

Kind Regards Rencroft

darthhaggis commented 6 years ago

To expand on the previous post, URL formatting has changed slightly. For example (for same video page): URL before site update (still works with youtube-dl): https://uktvplay.uktv.co.uk/shows/secrets-of-the-castle/watch-online/?video=5307684600001 URL after site update (doesn't work with youtube-dl): https://uktvplay.uktv.co.uk/shows/secrets-of-the-castle/watch-online/5818502556001 The old URL however is not viewable in a browser; it just defaults to the series landing page whereas the new URL will direct you to the specific episode within the browser. Looking at the page source for the new URL, the old video_id can not be found within the html.

Rencroft commented 6 years ago

darthhaggis, thanks for the input your quite right.

thrnz commented 6 years ago

Note that some videos still fail to download, even having changed to the old URL format. For instance: 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://uktvplay.uktv.co.uk/shows/taskmaster/watch-online/?video=5827355680001']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.10.05
[debug] Python version 3.6.6 (CPython) - Linux-4.15.0-36-generic-x86_64-with
[debug] exe versions: ffmpeg 3.4.4, ffprobe 3.4.4
[debug] Proxy map: {}
[brightcove:new] 5827355680001: Downloading webpage
[brightcove:new] 5827355680001: Downloading JSON metadata
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 507, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/brightcove.py", line 743, in _real_extract
    json_data, video_id, headers=headers)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/brightcove.py", line 648, in _parse_brightcove_metadata
    self._sort_formats(formats)
  File "/usr/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 1286, in _sort_formats
    raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
darthhaggis commented 6 years ago

@thrnz: Notice in the example given, the video_id associated with the old format is not the same video_id associated with the new format URL, even though they are for the same video. Using the video_id associated with the other format does not work and produces errors. I have not found reference to the old video_id within the new URL source code, so I don't see a simple way of extracting the old video_id to be able to continue using the old format.

EDIT: If there were a quick and easy way of extracting the old video_id from the new URL, we could continue with 'business as usual' but such a way has so far eluded us. If anyone can prove us wrong, please enlighten us.

In situations like this, it would be so much simpler if we could just throw the Blob file address at the program...

dankargo commented 6 years ago

Just a very amateur guess but could the value after "/fairplay/' in this example be a potential VideoID?

src="https://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?...

For URL https://uktvplay.uktv.co.uk/shows/classic-eastenders/watch-online/5844243441001

darthhaggis commented 6 years ago

Just a very amateur guess but could the value after "/fairplay/' in this example be a potential VideoID?

src="https://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?...

For URL https://uktvplay.uktv.co.uk/shows/classic-eastenders/watch-online/5844243441001

I have tried using that number with no luck. It also appears elsewhere in the source code as: data-account="1242911124001"

Vangelis66 commented 6 years ago

after "/fairplay/"

Well, if they're now using Apple Fairplay DRM for their mobile streams, then it's GAME OVER (it hasn't been cracked) 💢 . Prior to their site overhaul, they used the brightcove service and AppleHLS encrypted (HLSe) streams for mobile devices (which use AES-128 encryption supported by yt-dl) ...

I have kept records on disk and for a show in the old site format

http://uktvplay.uktv.co.uk/shows/narrow-escapes-of-world-war-ii/watch-online/?video=2776248288001

the HLSe master playlist would have been

http://c.brightcove.com/services/mobile/streaming/index/master.m3u8?videoId=2776248288001

The master playlist posted by @dankargo suggests a completely new playlist API... BTW, it would help other people troubleshoot this further if valid login credentials were shared here, even if temporarily... 😉

Rencroft commented 6 years ago

Here's a login :)

cemud@oranek.com devtempy1#

Vangelis66 commented 6 years ago

@Rencroft

Many thanks for the login credentials 👍 HOWEVER, my worst fears have been realised 😞

Trying, for instance,

https://uktvplay.uktv.co.uk/shows/classic-eastenders/watch-online/5844243441001

in a desktop browser and inspecting network activity via Web Console, one can identify, even before login-in, the following GET XHR (XMLHttpRequest) request to

https://edge.api.brightcove.com/playback/v1/accounts/1242911124001/videos/5844243441001

The raw response from that request is the following JSON file: 

{
 "ad_keys": null,
 "sources": [
  {
   "src": "http://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?fastly_token=NWJlZTE2N2RfZjg1NTNmYzZiNGU2YjA1OWE4ZWMyMjg2ZTFkYjc1MWM4OWU3MmE4ZjVhY2U4ZGZiOTJiNGNhMWFhMTkyMTQyNA%3D%3D",
   "key_systems": {
    "com.apple.fps.1_0": {
     "certificate_url": "https://manifest.prod.boltdns.net/license/v1/fairplay_app_cert/1242911124001",
     "key_request_url": "https://manifest.prod.boltdns.net/license/v1/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6a19b118-ad29-4a53-8fa5-8abbdea10fca?fastly_token=NWJlZTE2N2RfZmU1MWQ5OTFhODc3MTcwYjAxMTQxZTAzMmE0MmQ2YTUzMzgyNTEwYTkyNmYxODM4ZDc4N2VhZDkyNGE2ZjljNw%3D%3D"
    }
   },
   "ext_x_version": "5",
   "type": "application/x-mpegURL"
  },{
   "src": "https://manifest.prod.boltdns.net/manifest/v1/hls/v5/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/10s/master.m3u8?fastly_token=NWJlZTE2N2RfZjg1NTNmYzZiNGU2YjA1OWE4ZWMyMjg2ZTFkYjc1MWM4OWU3MmE4ZjVhY2U4ZGZiOTJiNGNhMWFhMTkyMTQyNA%3D%3D",
   "key_systems": {
    "com.apple.fps.1_0": {
     "certificate_url": "https://manifest.prod.boltdns.net/license/v1/fairplay_app_cert/1242911124001",
     "key_request_url": "https://manifest.prod.boltdns.net/license/v1/fairplay/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6a19b118-ad29-4a53-8fa5-8abbdea10fca?fastly_token=NWJlZTE2N2RfZmU1MWQ5OTFhODc3MTcwYjAxMTQxZTAzMmE0MmQ2YTUzMzgyNTEwYTkyNmYxODM4ZDc4N2VhZDkyNGE2ZjljNw%3D%3D"
    }
   },
   "ext_x_version": "5",
   "type": "application/x-mpegURL"
  },{
   "src": "http://manifest.prod.boltdns.net/manifest/v1/dash/live-baseurl/bccenc/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfOTlkMTAwNDg0NTQ2YTdjNDgyZTViODc3YThjYzgxYzM0MWRmNGM5OTY5ZTEyYzZjZTI2YmFhYzhmZTNlMzE5ZA%3D%3D",
   "key_systems": {
    "com.widevine.alpha": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfMDE1OWUwY2U2ZmY5ZDk5OTYxYmFhMGU2MWEzNzhkNjE4ZWEyYTllNzhiMmI1NTMwNmQ1MGYwYTZhNjgzMDJmOA%3D%3D"
    },
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfNjhlOWU1YzY3OWE2Yjk1OWU2MTYxZjcwZmIwMzczZDRlYThjNGM5MjM2NmYzNzUyYmY1ODhjY2I0NjM4MGFhNg%3D%3D"
    }
   },
   "profiles": "urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "https://manifest.prod.boltdns.net/manifest/v1/dash/live-baseurl/bccenc/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfOTlkMTAwNDg0NTQ2YTdjNDgyZTViODc3YThjYzgxYzM0MWRmNGM5OTY5ZTEyYzZjZTI2YmFhYzhmZTNlMzE5ZA%3D%3D",
   "key_systems": {
    "com.widevine.alpha": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/widevine/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfMDE1OWUwY2U2ZmY5ZDk5OTYxYmFhMGU2MWEzNzhkNjE4ZWEyYTllNzhiMmI1NTMwNmQ1MGYwYTZhNjgzMDJmOA%3D%3D"
    },
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/cenc/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/ecc64650-eade-4f4a-89d5-51b0505ea674?fastly_token=NWJlZTE2N2RfNjhlOWU1YzY3OWE2Yjk1OWU2MTYxZjcwZmIwMzczZDRlYThjNGM5MjM2NmYzNzUyYmY1ODhjY2I0NjM4MGFhNg%3D%3D"
    }
   },
   "profiles": "urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "http://manifest.prod.boltdns.net/manifest/v1/dash/live-hbbtv15/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfM2NkYjAzZmY0NTExYTUzYzQ1ZGY4YTAxZjJjOTc5NGJhMDk5ODJkMDFmOWQ5YWE2YWFkYWFlZGM4ZDJiOGQ3Mg%3D%3D",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "profiles": "urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "https://manifest.prod.boltdns.net/manifest/v1/dash/live-hbbtv15/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/6s/manifest.mpd?fastly_token=NWJlZTE2N2RfM2NkYjAzZmY0NTExYTUzYzQ1ZGY4YTAxZjJjOTc5NGJhMDk5ODJkMDFmOWQ5YWE2YWFkYWFlZGM4ZDJiOGQ3Mg%3D%3D",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "profiles": "urn:hbbtv:dash:profile:isoff-live:2012,urn:mpeg:dash:profile:isoff-live:2011",
   "type": "application/dash+xml"
  },{
   "src": "http://bcboltuktv-a.akamaihd.net/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest?akamai_token=exp=1542329981~acl=/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest*~hmac=e3e9c3053b1043edd71c6c0a8aeaf8b90a35fe1e90890821dad8ee27893ebf3e",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "type": "application/vnd.ms-sstr+xml"
  },{
   "src": "https://bcboltuktv-a.akamaihd.net/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest?akamai_token=exp=1542329981~acl=/media/v1/smooth/simple/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e/manifest.ism/Manifest*~hmac=e3e9c3053b1043edd71c6c0a8aeaf8b90a35fe1e90890821dad8ee27893ebf3e",
   "key_systems": {
    "com.microsoft.playready": {
     "license_url": "https://manifest.prod.boltdns.net/license/v1/playready/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/7ef1386e-c391-4256-ba13-069340a0345e?fastly_token=NWJlZTE2N2RfYjgyODk4NjZmNDRjZjE1YTYxODMzYWY5MmFmMGM5YmUxZWI4Y2M3OTU5MzhhNDBmYzAwN2E3ZjUyN2NkYTYyOQ%3D%3D"
    }
   },
   "type": "application/vnd.ms-sstr+xml"
  }],
 "updated_at": "2018-10-05T05:21:19.086Z",
 "poster_sources": [
  {
   "src": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/1024x576/14m46s48ms/match/image.jpg"
  }],
 "reference_id": null,
 "offline_enabled": false,
 "text_tracks": [],
 "economics": "AD_SUPPORTED",
 "description": null,
 "poster": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/1024x576/14m46s48ms/match/image.jpg",
 "long_description": null,
 "created_at": "2018-10-04T09:41:43.440Z",
 "id": "5844243441001",
 "custom_fields": {
  "springstream": "OD",
  "housenum": "CTBT132W82-dd",
  "springcq": "CTBT132W"
 },
 "duration": 1772096,
 "account_id": "1242911124001",
 "tags": [],
 "name": "CTBT132W82-dd",
 "link": null,
 "thumbnail": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/320x180/14m46s48ms/match/image.jpg",
 "cue_points": [
  {
   "metadata": null,
   "force_stop": false,
   "id": "5844706303001",
   "type": "AD",
   "time": 673.64,
   "name": "cp_16243983"
  },{
   "metadata": null,
   "force_stop": false,
   "id": "5844706304001",
   "type": "AD",
   "time": 1772,
   "name": "cp_16243984"
  }],
 "thumbnail_sources": [
  {
   "src": "https://cf-images.eu-west-1.prod.boltdns.net/v1/jit/1242911124001/436b2bb0-3519-421c-aebb-eb1b1be02534/main/320x180/14m46s48ms/match/image.jpg"
  }],
 "published_at": "2018-10-04T09:41:43.440Z"
}

As anyone can see, the place reeks of DRM: AppleHLS streams (mobile devices, mobile apps) use FairPlay, MPEG-DASH streams (desktop browsers) use Common Encryption (cenc) with Widevine, plus there exist Microsoft friendly "Smooth Streaming" streams which use PlayReady DRM!

All mentioned DRM schemes have not been reverse-engineered (if someone has, he/she'd be wise enough to not disclose his/her feat...), so it is currently unrealistic to expect yt-dl to support the new site of UKTVplay...

Unless someone figures out an algorithm to convert the "new" video_ids to the "old" value that would still work (for how long?) with the previous site template, as I said, it's GAME OVER! If, like me, you abhor any form of DRM, you'd better stay away from the new UKTVPlay iteration 👎

rautamiekka commented 6 years ago

All mentioned DRM schemes have not been reverse-engineered (if someone has, he/she'd be wise enough to not disclose his/her feat...), so it is currently unrealistic to expect yt-dl to support the new site of UKTVplay...

Why ?

Rencroft commented 6 years ago

Well that's a shame, they were better quality than recording of TV.

Thanks to everyone who looked into it.

darthhaggis commented 6 years ago

Let me see if I understand this correctly. Long story short, even though we can view an episode without issue in a browser, due to the change in how their website is structured, there is now currently no way we will be able to use yt-dl to save that same file for offline viewing, correct?

dankargo commented 6 years ago

I just discovered this Kodi plugin which has recently been updated to allow streaming of UKTV Play's new DRM content in Kodi. Maybe it may give us some hope?

https://github.com/Catch-up-TV-and-More/plugin.video.catchuptvandmoretestdrmkodi18

Vangelis66 commented 6 years ago

@dankargo

Their UKTV Play plugin can be found here: https://github.com/Catch-up-TV-and-More/plugin.video.catchuptvandmoretestdrmkodi18/blob/master/resources/lib/channels/uk/uktvplay.py but, as I said previously, it's useless for yt-dl purposes!

Widevine DRM hasn't been cracked; in fact, Kodi does not circumvent that DRM scheme but relies upon a system installed Chrome browser (with a supported and working WidevineCDM, currently at version >= 1.4.8.984) for the actual decryption of the streams; you can read more at: https://kodi.tv/article/dev-journal-kodi-and-drm

As another example, in Windows, there isn't a very good 10-foot interface for Netflix, but if you have installed Chrome, you can watch Netflix from your desktop. Supporting low-level DRM means that Kodi could hook into the binary blob inside Chrome and use it purely for handling the DRM while video playback and control stays with Kodi.

dankargo commented 5 years ago

Such a shame there’s no way around this :(

darthhaggis commented 5 years ago

Does this mean UKTV will now need to be removed from the 'Supported Sites' page?

jmiskinis commented 5 years ago

Any updates on this?

mdcfe commented 5 years ago

@jmiskinis If there were updates on this, someone would have posted an update here.

Edit: in #18417, people found a DRM-free link for RTE Player, but their platform seems to have little in common with UKTV Play except the DRM that is used (which is the opposite of what we need).

jmiskinis commented 5 years ago

Thanks for the info. I will agree with darthhaggis that UKTV should be removed from the supported sites list.