search

yt-dlp / yt-dlp

A feature-rich command-line audio/video downloader
https://discord.gg/H5MNcFW63r
The Unlicense
82.37k stars 6.42k forks source link

ITV-X #5570

Open Twosheds-Jackson opened 1 year ago

Twosheds-Jackson commented 1 year ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

Checklist

Region

UK

Provide a description that is worded well enough to be understood

ITV-Hub has just been replaced with the newly reformatted "ITV-X". Up until a few hours ago, downloads from ITV-Hub still worked, but now that they have all been replaced by ITV-X the downloads fail:

[ITV] 1a7314a0014: Downloading webpage ERROR: [ITV] 1a7314a0014: Unable to extract params; please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U

Provide verbose output that clearly demonstrates the problem

Complete Verbose Output

[GUI] executing command line: "D:\Drive D\YouTube-DL\ylp\yt-dlp.exe" -S "res:1080,vext:mp4,aext:m4a,fps" --windows-filenames --embed-chapters --embed-subs --embed-thumbnail --no-mtime --ffmpeg-location "D:\Drive D\YouTube-DL\ylp\ffmpeg.exe" -vU  -o "C:\Users\xxx\Downloads\%(title)s.%(ext)s" https://www.itv.com/hub/vera/1a7314a0014

[debug] Command-line config: ['-S', 'res:1080,vext:mp4,aext:m4a,fps', '--windows-filenames', '--embed-chapters', '--embed-subs', '--embed-thumbnail', '--no-mtime', '--ffmpeg-location', 'D:\\Drive D\\YouTube-DL\\ylp\\ffmpeg.exe', '-vU', '-o', 'C:\\Users\\xxx\\Downloads\\%(title)s.%(ext)s', 'https://www.itv.com/hub/vera/1a7314a0014']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out cp1252 (No VT), error cp1252 (No VT), screen cp1252 (No VT)
[debug] yt-dlp version 2022.11.11 [8b64402] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-7-6.1.7601-SP1 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-106386-ge78173557d-20220322 (setts), ffprobe N-106386-ge78173557d-20220322
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[debug] Using fake IP 25.17.235.224 (GB) as X-Forwarded-For
[debug] [ITV] Extracting URL: https://www.itv.com/hub/vera/1a7314a0014
[ITV] 1a7314a0014: Downloading webpage
ERROR: [ITV] 1a7314a0014: Unable to extract params; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 674, in extract
  File "yt_dlp\extractor\itv.py", line 137, in _real_extract
  File "yt_dlp\extractor\common.py", line 1228, in _search_regex

[GUI] yt-dlp.exe process has exited
bashonly commented 1 year ago

Looks like the new webpage format is using next.js; the __NEXT_DATA__ block contains a lot of metadata and playlist URLs. Webpage also makes a request to https://www.itv.com/_bm/get_params?type=web-jsto. Still geo-restricted

dirkf commented 1 year ago

The get_params page is also 4xx from the UK when fetched for the problem page and give what looks like a CF block page when opened separately. Possibly it requires you to log in.

Twosheds-Jackson commented 1 year ago

I see now that the ITV-X has different URL's to ITV-Hub.

For example, what was previously: https://www.itv.com/hub/vera/1a7314a0014

is now: https://www.itv.com/watch/vera/1a7314/1a7314a0014

(Confusingly, the old ITV-Hub URL's still work in a browser, but they are redirecting to the new ITV-X pages)

Using the new ITV-X URL's, YT-DLP produces an "Unsupported URL" error:

[GUI] executing command line: "D:\Drive D\YouTube-DL\ylp\yt-dlp.exe" -S "res:1080,vext:mp4,aext:m4a,fps" --windows-filenames --embed-chapters --embed-subs --embed-thumbnail --no-mtime --ffmpeg-location "D:\Drive D\YouTube-DL\ylp\ffmpeg.exe" -vU  -o "C:\Users\xxx\Downloads\%(title)s.%(ext)s" https://www.itv.com/watch/vera/1a7314/1a7314a0014

[debug] Command-line config: ['-S', 'res:1080,vext:mp4,aext:m4a,fps', '--windows-filenames', '--embed-chapters', '--embed-subs', '--embed-thumbnail', '--no-mtime', '--ffmpeg-location', 'D:\\Drive D\\YouTube-DL\\ylp\\ffmpeg.exe', '-vU', '-o', 'C:\\Users\xxx\\Downloads\\%(title)s.%(ext)s', 'https://www.itv.com/watch/vera/1a7314/1a7314a0014']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out cp1252 (No VT), error cp1252 (No VT), screen cp1252 (No VT)
[debug] yt-dlp version 2022.11.11 [8b64402] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-7-6.1.7601-SP1 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-106386-ge78173557d-20220322 (setts), ffprobe N-106386-ge78173557d-20220322
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[debug] [generic] Extracting URL: https://www.itv.com/watch/vera/1a7314/1a7314a0014
[generic] 1a7314a0014: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] 1a7314a0014: Extracting information
[debug] Looking for embeds
ERROR: Unsupported URL: https://www.itv.com/watch/vera/1a7314/1a7314a0014
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1485, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1561, in __extract_info
  File "yt_dlp\extractor\common.py", line 674, in extract
  File "yt_dlp\extractor\generic.py", line 2721, in _real_extract
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.itv.com/watch/vera/1a7314/1a7314a0014

[GUI] yt-dlp.exe process has exited
dirkf commented 1 year ago

"Never mind the streaming site, people want to know more about your sheds..."

That's right. A Hub URL passes the pattern test for yt-dlp, then redirects when loaded and fails because the page structure is different. With the trivial pattern change ...hub... -> ...(?:watch|hub)..., the second step happens directly.

Twosheds-Jackson commented 1 year ago

The sheds really aren't that important. (A few of my friends call me 'Two Sheds'. I've only got one shed, but a few years ago, I said I was thinking of buying another one, and since then some people started calling me 'Two Sheds')

This is all very new to me, but I imagine this error needs a change to itv.py, and there appear to be 4 regular contributors to that code, so I guess we need some help from one of those: @pukkandan, @staubichsauger, @ajj8, @coletdjnz

Is there any other information which I can usefully provide?

Samuel98 commented 1 year ago

ITV (ITVX) now has Widevine (DRM) by the looks of it. As a result ITV will probably be removed from supported sites.

dirkf commented 1 year ago

The second of those did the last major work on the Hub extractor. See #5010 for discussion of the limited scope extraction that could be achieved. Possibly the new site has cut that off.

The equivalent HTML element to the extractor's initial target #video is [data-testid=genie-container]. However there is no equivalent to the data-video-hmac attribute that is needed to look up the playlist data. The playlist URL https://magni.itv.com/playlist/itvonline/ITV/1_7314_0014.003 now gives 404 anyway, so that's moot.

Two next steps could be

If any playable links are available, we need to reproduce whatever happens with the browser.

Vangelis66 commented 1 year ago

ITV (ITVX) now has Widevine (DRM) by the looks of it.

... The previous incarnation, ITV Hub, did also employ cenc (Google Widevine/Microsoft Playready) for their desktop browser streams, see my previous analysis (FWIW, they had been using ClearKey MPEG-DASH encryption a few years ago) ...

And one post above the one I linked to, I wrote:

The crux of the matter is that the existing itvIE on both youtube-dl and yt-dlp relies upon the HLSe (AES-128 encrypted) streams targeting older Apple MOBILE devices (iPhones/iPads) and these streams max out at 504p SD...

As I see it, the route one should go to tackle this would be to ascertain whether HLSe AES-128 streams are STILL available for old iOS devices, preferably by using such a device (inside the UK) and monitoring web traffic...

If such streams exist no more, then it's Game Over 😡 ...

[OT: All this coincided with the American CBS Network going, too, DRM-only; by the looks of it and the rate it is happening, all major media outlets, the world over, will have deprecated "downloading" apps pretty soon (by implementing full-blown DRM) 😭 ; be damned, Google! 😞 ]

EDIT: Didn't actually see @dirkf's comment above, posted a mere 2min before mine...

pukkandan commented 1 year ago

Most of the current itv code is based on https://github.com/ytdl-org/youtube-dl/pull/28955 by @sleaux-meaux and https://github.com/ytdl-org/youtube-dl/issues/28906#issuecomment-831008270

Vangelis66 commented 1 year ago

... A very hasty investigation revealed that the mobile streams have all switched from HLSe (AES-128) encryption to full-blown AppleFairPlay DRM 😢 ...

Faking an old iPhone in my desktop browser when requesting https://www.itv.com/watch/family-guy/2a4259/2a4259a0395 from a whitelisted UK IP got me served below m3u8 manifest (I didn't have to log in): 

https://itvpnpdotcom.blue.content.itv.com/2-4259-0395-001/18/1/VAR026/2-4259-0395-001_18_1_VAR026.ism/.m3u8?filter=%28%28type%3D%3D%22video%22%26%26DisplayHeight%3C%3D576%29%7C%7C%28type%21%3D%22video%22%29%29&hdnea=st%3D1668741371~exp%3D1668762971~acl%3D/2-4259-0395-001/%2A~data%3Dnohubplus~hmac%3Dbc60b7715e6e479a35678b395d91b4e037fc770f2fdc43af5f0def7cabcecc0b

URL-decoded:

https://itvpnpdotcom.blue.content.itv.com/2-4259-0395-001/18/1/VAR026/2-4259-0395-001_18_1_VAR026.ism/.m3u8?filter=((type=="video"&&DisplayHeight<=576)||(type!="video"))&hdnea=st=1668741371~exp=1668762971~acl=/2-4259-0395-001/*~data=nohubplus~hmac=bc60b7715e6e479a35678b395d91b4e037fc770f2fdc43af5f0def7cabcecc0b

This is geo-fenced and short-lived, its content posted below: 

#EXTM3U
#EXT-X-VERSION:4
## Created with OEM  (version=1.11.3-24483)
#EXT-X-SESSION-KEY:METHOD=SAMPLE-AES,URI="skd://itvpnp.live.ott.irdeto.com/Streaming/getckc?CrmId=itvpnp&AccountId=itvpnp&ContentId=2-4259-0395-001_18&KeyId=4af56b7f-504e-4f61-a40d-3f64af703770",KEYFORMAT="com.apple.streamingkeydelivery",KEYFORMATVERSIONS="1"

# AUDIO groups
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-aacl-96",NAME="audio",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2"

# variants
#EXT-X-STREAM-INF:BANDWIDTH=636000,CODECS="mp4a.40.2,avc1.64001F",RESOLUTION=512x288,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_18_1_VAR026-audio=96000-video=503976.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=849000,CODECS="mp4a.40.2,avc1.64001F",RESOLUTION=512x288,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_18_1_VAR026-audio=96000-video=704051.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1272000,CODECS="mp4a.40.2,avc1.64001F",RESOLUTION=896x504,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_18_1_VAR026-audio=96000-video=1103978.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1590000,CODECS="mp4a.40.2,avc1.64001F",RESOLUTION=896x504,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_18_1_VAR026-audio=96000-video=1403946.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1908000,CODECS="mp4a.40.2,avc1.64001F",RESOLUTION=896x504,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_18_1_VAR026-audio=96000-video=1703979.m3u8

# variants
#EXT-X-STREAM-INF:BANDWIDTH=102000,CODECS="mp4a.40.2",AUDIO="audio-aacl-96"
2-4259-0395-001_18_1_VAR026-audio=96000.m3u8

# keyframes
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=67000,CODECS="avc1.64001F",RESOLUTION=512x288,VIDEO-RANGE=SDR,URI="keyframes/2-4259-0395-001_18_1_VAR026-video=503976.m3u8"
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=147000,CODECS="avc1.64001F",RESOLUTION=896x504,VIDEO-RANGE=SDR,URI="keyframes/2-4259-0395-001_18_1_VAR026-video=1103978.m3u8"

Unless proven wrong, I think it's all over... :rage: :rage: :rage: ("every change is for the worse", that's my moto in the recent years and not only because I've grown old-er :wink: ...)

Vangelis66 commented 1 year ago

POST request to the magni playlist API for the ITVX programme above:

POST

No featureset ["hls","aes"] anymore... From my May 31st, 2021 comment that pukkandan linked:

As you can see, ITV Hub are going full-steam ahead with fully DRM-ing all their streams;

Twosheds-Jackson commented 1 year ago

Ah... I feared they might do this when ITV-X launched because there will also be subscription-only content provided on the same site.. (Britbox: as was)

Is DRM content something which YT-DLP doesn't support? (I've seen Britbox downloads online, so there must be a way around this)

Personally, I'm not looking to download any paid-for content, just some of the stuff broadcast clear and free in the UK.

dirkf commented 1 year ago

I suppose it's possible that some shows (News, eg) not encumbered by 3rd-party rights might still be offered without DRM, but probably not.

The re-launched 'Spitting Image', now canned, was available on US YouTube for non-BB viewers.

As to the availability of other BB material and DRM in general, see my recent comment in the upstream project. The projects operate similar policies on this.

Vangelis66 commented 1 year ago

... Right... The bright Greek sunshine plus 22Celsius earlier today (mind you, winter will finally hit these places next Monday 😞 ) made me feel more optimistic 😉 and cleared my mind considerably ... Plus, I won't throw in the towel that easily... So far, we know that:

  1. ITV Hub became itvX, with a major restructuring/overhaul of the webpage;

  2. ITV Hub URIs now redirect in the browser to itvX ones, e.g. https://www.itv.com/hub/family-guy/2a4259a0395 => https://www.itv.com/watch/family-guy/2a4259/2a4259a0395 or, more generally, 

    https://www.itv.com/hub/<seriesName>/<episodeID> => 
https://www.itv.com/watch/<seriesName>/<seriesID>/<episodeID>

    1.+2. are enough on their own to break metadata extraction in the current (broken) itvIE 😢 ...

  3. As in the case of ITV Hub, itvX on desktop browsers will only offer MPEG-DASH streams encrypted with full-blown DRM (Widevine/MSPR), no good for yt-dlp purposes...

  4. ITV Hub would offer mobile useragents HLSe streams, exactly the ones current itvIE used to exploit to fetch ITV offerings...

  5. itvX now offers mobile useragents exclusively HLS streams protected (encrypted) with full-blown Apple Fairplay DRM, no good for yt-dlp purposes...

But, despite all the changes involved in the transition of ITV Hub to itvX, I have verified that the underlying playlist API remained the same! As shown in my previous post above, to get available streams (and subs), no matter the useragent ("platformTag"= ("dotcom" | | "mobile")), itvX POST requests magni.itv.com/playlist/itvonline/<string-derived-from-episodeID>; NB, that request is "geo-sensitive" and requires several headers/cookies...

As described in 5., we no longer get offered any non-DRM HLS streams on mobile useragents by default, but (and here is where I had an epiphany) ... what if... ?

At this point in time, I'm not prepared to publicly disclose the whole story, but I devised and successfully tested a "hack" to access the same HLSe non-DRM streams that ITV Hub offered, but within the new itvX interface...
Not prepared, because ITV staff are always vigilant and probably monitoring this issue right now, hence I fear my "idea" could be dead in the water before it even takes off...

Proof: For "https://www.itv.com/watch/family-guy/2a4259/2a4259a0395", I got below HLSe master manifest:

https://itvpnpmobile.blue.content.itv.com/2-4259-0395-001/6/1/VAR003/2-4259-0395-001_6_1_VAR003.ism/.m3u8?filter=%28%28type%3D%3D%22video%22%26%26DisplayHeight%3C%3D576%29%7C%7C%28type%21%3D%22video%22%29%29&hdnea=st%3D1668797376~exp%3D1668818976~acl%3D/2-4259-0395-001/%2A~data%3Dnohubplus~hmac%3D7f1a8705d42cd5f54334012ca8655e89e3a1e1d7b67a94596f762b11bb189958

whose content is: 

#EXTM3U
#EXT-X-VERSION:4
## Created with OEM  (version=1.11.3-24483)

# AUDIO groups
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-aacl-64",NAME="audio",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-aacl-96",NAME="audio",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2"

# variants
#EXT-X-STREAM-INF:BANDWIDTH=285000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=512x288,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-64",CLOSED-CAPTIONS=NONE
2-4259-0395-001_6_1_VAR003-audio=64000-video=203953.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=636000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=512x288,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_6_1_VAR003-audio=96000-video=503980.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=848000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=512x288,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_6_1_VAR003-audio=96000-video=703986.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1272000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=896x504,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_6_1_VAR003-audio=96000-video=1103978.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1908000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=896x504,FRAME-RATE=25,VIDEO-RANGE=SDR,AUDIO="audio-aacl-96",CLOSED-CAPTIONS=NONE
2-4259-0395-001_6_1_VAR003-audio=96000-video=1703960.m3u8

# variants
#EXT-X-STREAM-INF:BANDWIDTH=68000,CODECS="mp4a.40.2",AUDIO="audio-aacl-64"
2-4259-0395-001_6_1_VAR003-audio=64000.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=102000,CODECS="mp4a.40.2",AUDIO="audio-aacl-96"
2-4259-0395-001_6_1_VAR003-audio=96000.m3u8

# keyframes
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=28000,CODECS="avc1.4D401F",RESOLUTION=512x288,VIDEO-RANGE=SDR,URI="keyframes/2-4259-0395-001_6_1_VAR003-video=203953.m3u8"
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=147000,CODECS="avc1.4D401F",RESOLUTION=896x504,VIDEO-RANGE=SDR,URI="keyframes/2-4259-0395-001_6_1_VAR003-video=1103978.m3u8"

Best variant playlist (504p25@1900kbps) would be: 

https://itvpnpmobile.blue.content.itv.com/2-4259-0395-001/6/1/VAR003/2-4259-0395-001_6_1_VAR003.ism/2-4259-0395-001_6_1_VAR003-audio=96000-video=1703960.m3u8

... its first lines shown below: 

#EXTM3U
#EXT-X-VERSION:4
## Created with OEM  (version=1.11.3-24483)
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-TARGETDURATION:6
#USP-X-TIMESTAMP-MAP:MPEGTS=900000,LOCAL=1970-01-01T00:00:00Z
#EXT-X-KEY:METHOD=AES-128,URI="https://itvpnp.live.ott.irdeto.com/HLS/GetKey.aspx?CrmId=itvpnp&AccountId=itvpnp&ContentId=2-4259-0395-001_6&KeyId=b3c85a7a-96b8-4852-96a4-525a08da8423"
#EXTINF:6, no desc
2-4259-0395-001_6_1_VAR003-audio=96000-video=1703960-1.ts
#EXTINF:6, no desc
2-4259-0395-001_6_1_VAR003-audio=96000-video=1703960-2.ts
#EXTINF:6, no desc

and (over VPN) : 

yt-dlp -vF "https://itvpnpmobile.blue.content.itv.com/2-4259-0395-001/6/1/VAR003/2-4259-0395-001_6_1_VAR003.ism/.m3u8?filter=%28%28type%3D%3D%22video%22%26%26DisplayHeight%3C%3D576%29%7C%7C%28type%21%3D%22video%22%29%29&hdnea=st%3D1668797376~exp%3D1668818976~acl%3D/2-4259-0395-001/%2A~data%3Dnohubplus~hmac%3D7f1a8705d42cd5f54334012ca8655e89e3a1e1d7b67a94596f762b11bb189958" -o "ITV2_FamilyGuy_Se20Ep15_HardBoiledMeg[2a4259a0395].mp4" => 

[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-vF', 'https://itvpnpmobile.blue.content.itv.com/2-4259-0395-001/6/1/VAR003/2-4259-0395-001_6_1_VAR003.ism/.m3u8?filter=%28%28type%3D%3D%22video%22%26%26DisplayHeight%3C%3D576%29%7C%7C%28type%21%3D%22video%22%29%29&hdnea=st%3D1668797376~exp%3D1668818976~acl%3D/2-4259-0395-001/%2A~data%3Dnohubplus~hmac%3D7f1a8705d42cd5f54334012ca8655e89e3a1e1d7b67a94596f762b11bb189958', '-o', 'ITV2_FamilyGuy_Se20Ep15_HardBoiledMeg[2a4259a0395].mp4']
[debug] Encodings: locale cp1253, fs utf-8, pref cp1253, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version 2022.11.18.152719 [ed027fd] (win_x86_exe)
[debug] Python 3.7.9 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.1.1g  21 Apr 2020)
[debug] exe versions: ffmpeg 5.0 (fdk,setts), ffprobe 5.0, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1724 extractors
[debug] [generic] Extracting URL: https://itvpnpmobile.blue.content.itv.com/2-4259-0395-001/6/1/VAR003/2-4259-0395-001_6_1_VAR003.ism/.m3u8?filter=%28%28type%3D%3D%22video%22%26%26DisplayHeight%3C%3D576%29%7C%7C%28type%21%3D%22video%22%29%29&hdnea=st%3D1668797376~exp%3D1668818976~acl%3D/2-4259-0395-001/%2A~data%3Dnohubplus~hmac%3D7f1a8705d42cd5f54334012ca8655e89e3a1e1d7b67a94596f762b11bb189958
[generic] *~data=nohubplus~hmac=7f1a8705d42cd5f54334012ca8655e89e3a1e1d7b67a94596f762b11bb189958: Downloading webpage
[debug] Identified a direct video link
[generic] *~data=nohubplus~hmac=7f1a8705d42cd5f54334012ca8655e89e3a1e1d7b67a9459
6f762b11bb189958: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for *~data=nohubplus~hmac=7f1a8705d42cd5f54334012ca8655e89e3a1e1d7b67a94596f762b11bb189958:
ID   EXT RESOLUTION FPS |   TBR PROTO | VCODEC        VBR ACODEC    ABR
-----------------------------------------------------------------------
68   mp4 audio only     |   68k m3u8  | audio only        mp4a.40.2 64k
102  mp4 audio only     |  102k m3u8  | audio only        mp4a.40.2 96k
285  mp4 512x288     25 |  285k m3u8  | avc1.4D401F  204k mp4a.40.2 64k
636  mp4 512x288     25 |  636k m3u8  | avc1.4D401F  504k mp4a.40.2 96k
848  mp4 512x288     25 |  848k m3u8  | avc1.4D401F  704k mp4a.40.2 96k
1272 mp4 896x504     25 | 1272k m3u8  | avc1.4D401F 1104k mp4a.40.2 96k
1908 mp4 896x504     25 | 1908k m3u8  | avc1.4D401F 1704k mp4a.40.2 96k

So far, I regained access to streams and subtitles, only... Since I'm not a Python coder (just a semi-educated advanced yt-dl user), I don't have the skills to fix myself the broken itvIE - honestly, I think itvIE should be scratched in favour of a new itvXIE one...

@dirkf, in my eyes, you appear to be the ideal person to tackle this, by simple virtue of the fact you're UK-based, so it'd be much easier for you to test things (I have to depend on browser "proxy" extensions and system-wide VPN to work on ITV-related stuff...); would you please be willing to temporarily share an e-mail address here, so I could contact you in private, initially to explore whether "my" hack could materialise inside Python code? (probably yes, because I sense Python is very powerful) ... Of course, you then can fix the extractor fully, with metadata, thumbnail and the lot... 😄

I'll be unavailable for the next 3 - 4 hours, though, so you know...

=== All hope is not lost, yet... ===

Twosheds-Jackson commented 1 year ago

The power of Greek sunshine is clearly quite amazing. It would be great if your inspiration leads a viable new version.

dirkf commented 1 year ago

Possibly the Greek clouds will return once the transition to ITV-X is complete. Let's give it a few days.

https://rxtvinfo.com/2022/itvx-rollout-update-premium-access-details-and-latest-bugs

Also, see (eg) https://github.com/ytdl-org/youtube-dl/commit/27ed77aabba8c9eb08d66f34092b1bfcc22c482e for contact details.

dirkf commented 1 year ago

The "launch" of ITV-X is set for 8 Dec (although it's been running and been promoted on broadcast TV since this issue was created). I guess that will be a good point to see if anything further can be done.

Samuel98 commented 1 year ago

"Best variant playlist (504p25@1900kbps) would be:"

I'm assuming that the better quality streams are locked behind ITVX Premium. You'll probably get different manifests served to you if you're logged in with an ITV X Premium account.

dirkf commented 1 year ago

There wasn't any evidence of higher resolution non-DRM formats alongside the 504p25 mobile formats that the ITV Hub extractor found. I doubt that this has changed with ITVX.

Samuel98 commented 1 year ago

There wasn't any evidence of higher resolution non-DRM formats alongside the 504p25 mobile formats that the ITV Hub extractor found. I doubt that this has changed with ITVX.

There will probably have to be FHD streams available with ITVX Premium original content at least or why would anybody pay for it?

gaborbernat commented 1 year ago

For removing ads mostly 😅

Samuel98 commented 1 year ago

For removing ads mostly 😅

You can do that with a browser extension…

gaborbernat commented 1 year ago

Not everyone knows about it though 👍

Samuel98 commented 1 year ago

Also...Just seen this...

"ITVX will be rolling out across devices and platforms in the coming weeks, to ensure viewers have full access to all content, including the new, exclusive to streaming titles which will go live on Thursday 8th December.

At launch, viewers will have access to over 10,000 hours of content in high definition, and ITVX will also drop new and exclusive programmes every week of the year."

dirkf commented 1 year ago

What they forgot to add: "... and it will all be DRMed."

Samuel98 commented 1 year ago

First originals launched and only SD without Premium account. Don't know if they are HD with one but are encrypted with Widevine so looks like support for ITV needs to be removed.

dirkf commented 1 year ago

Now that launch-date has arrived, maybe @Vangelis66 can re-visit his work-around and perhaps some support will be possible.

Vangelis66 commented 1 year ago

maybe @Vangelis66 can re-visit his work-around and perhaps some support will be possible.

"My" workaround still works, but if the powers that be are made aware, they can probably crush it like an insect 😭 ...

Last Monday's Family Guy: Series 20, Episode 19 (not to be found in STV, as it's an ITV2 offering) :

https://www.itv.com/watch/family-guy/2a4259/2a4259a0399

The content of this attached file was generated ca. 15min ago... The "MediaFiles" link should be good for another 5h, at least... 😜 

#EXTM3U
#EXT-X-VERSION:4
## Created with OEM  (version=1.11.3-24483)
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-TARGETDURATION:6
#USP-X-TIMESTAMP-MAP:MPEGTS=900000,LOCAL=1970-01-01T00:00:00Z
#EXT-X-KEY:METHOD=AES-128,URI="https://itvpnp.live.ott.irdeto.com/HLS/GetKey.aspx?CrmId=itvpnp&AccountId=itvpnp&ContentId=2-4259-0399-001_6&KeyId=8002589f-bac7-4f6f-83c2-92850e054901"
Samuel98 commented 1 year ago

Still no HD though? Where are they hiding those manifests

dirkf commented 1 year ago

I expect you only get the HD if you're logged in, and only in DRM.

But indeed a small change to the old yt-dl extractor does still work.

Samuel98 commented 1 year ago

I expect you only get the HD if you're logged in, and only in DRM.

But indeed a small change to the old yt-dl extractor does still work.

Seems they only offer the 720/1080p streams on specific devices, all L1 android based.

dirkf commented 1 year ago

If you can find a device (this probably means a web browser) that is offered formats >= 720p and that can play them with DRM disabled, please let us know.

torbs723 commented 1 year ago

Any chance of the extractor being updated?

whitebear60 commented 1 year ago

Is the old API still working, at least for 896x504 SD video?

dirkf commented 1 year ago

The old media links are still available behind the new front-end, for now.

Extractor PR updated at yt-dl: https://github.com/ytdl-org/youtube-dl/pull/30266. Thanks @Vangelis66.

torbs723 commented 1 year ago

@dirkf Will pull requests from youtube-dl also be merged with yt-dlp? While we wait for a new version, would it be possible for someone to compile an exe from the master and post it for those of us who could do with an easier option than using python?

Sleaux-meaux did that when his fix never made it into the official build.

pukkandan commented 1 year ago

Will pull requests from youtube-dl also be merged with yt-dlp?

Yes, I periodically pull all commits from upstream

Edit: Help needed - see https://github.com/yt-dlp/yt-dlp/issues/21

YewTreeWeb commented 1 year ago

Hey guys, stumbled across this issue. Have there been any progress with this itv update? I'm on a mac and when trying to download from itv, I'm getting ERROR: Unsupported URL: https://www.itv.com/watch/doc-martin/26104/1a7665a0022

Vangelis66 commented 1 year ago

Have there been any progress with this itv update?

... Well, this issue you commented on is still labeled as Open 😉 , ergo...

Upstream implementation, announced previously in this thread, has been since merged (Feb 4th 2023) in youtube-dl itself (master branch), so I guess the dev team here would have to backport the "backported" ( 😜 ) ITV InfoExtractor into yt-dlp... It has been a low priority item, I guess... If you can code yourself, a PR will be most welcome, as per #21 and https://github.com/yt-dlp/yt-dlp/issues/21#issuecomment-1434677016 ...

Regards 😄 ...

dirkf commented 1 year ago

For a yt-dlp candidate, delete the first 3 lines and the ITVBaseIE._search_nextjs() method from the yt-dl extractor. For extra credit tweak the title processing.

mrg9999 commented 1 year ago

I asked ChatGPT for the fix, and it was most obliging

dirkf commented 1 year ago

Also https://github.com/ytdl-org/youtube-dl/issues/32217#issuecomment-1556007887.

torbs723 commented 1 year ago

I think ITV might have changed something. Anything published since last Friday was failing to download with with youtube-dl and now all programmes fail to download with the message "Unable to extract parameters".

@Vangelis66 Can you still get downloads to work with the modified playlist addresses or have they stopped uploading the DRM free versions now?

dirkf commented 1 year ago

See https://github.com/ytdl-org/youtube-dl/issues/32410#issuecomment-1624397936

unknown-- commented 1 year ago

As of today requesting HLS returns: Missing or incorrectly formed POST body: Invalid featureSet Looks like fairplay is now enforced.