ITV failure to download after site change

[njtaylor]

ITV not longer downloads. It appears flash videos have been disabled and replace with dash. This doesn't work with firefox only chrome, firefox still ends up trying to download a flash plugin. Video does play in chrome should be possible to replace flash with dash, dash has been implemented for other sites.

[Vangelis66]

Hello @njtaylor (I'm an "old" friend from the Get iPlayer mailing list)!

I can confirm the latest itv.pm v0.08 is broken since ca. July 17th 2017; I am on Windows Vista SP2 32bit, using portable Strawberry Perl 5.24.01 (x86) and the combined GFV script, v1.25.92

perl Scripts/c-gfv-1_25_92 "https://www.itv.com/hub/amazing-animal-births/2a3939a0006" --debug

~Trying to open plugin C:\Users\<snip>\AppData\Roaming/get_flash_videos/plugins/Ww
w.pm
Using method 'itv' plugin version 0.08 for https://www.itv.com/hub/amazing-anima
l-births/2a3939a0006
Downloading https://www.itv.com/hub/amazing-animal-births/2a3939a0006
-> GET https://www.itv.com/hub/amazing-animal-births/2a3939a0006
<- 200 text/html; charset=UTF-8 12558 (81252): UTF8 on, ASCII, 81252 characters
81252 bytes
Production ID https://old-origin-api.itv.com/playlist/itvonline/ITV/2_3939_0006.
003

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><GetPlay
listResponse xmlns="http://tempuri.org/"><GetPlaylistResult><Fault xmlns=""><fau
ltcode>InvalidVodcrid</faultcode><faultstring>No match found for production id:
https://old-origin-api.itv.com/playlist/itvonline/ITV/2_3939_0006.003</faultstri
ng></Fault></GetPlaylistResult></GetPlaylistResponse></s:Body></s:Envelope>
Error: Unable to find <Video> in XML at Scripts/c-gfv-1_25_92 line 16330.

Couldn't extract Flash movie URL. This site may need specific support adding,
or fixing.

Please confirm the site is using Flash video and if you have Flash available
check that the URL really works(!).

Check for updates by running: Scripts/c-gfv-1_25_92 --update

HOWEVER, I have some important additional info to contribute here:

1. The itv.pm plugin used to fetch RTMP (Flash) streams for ITV Hub VOD; their live TV streams still continue to use RTMP protocol (which live streams GFV does not support, but other tools do...).
2. For supported combinations of OS+desktop browsers, they (ITV hub) have apparently switched to ENCRYPTED MPEG-DASH streams, delivered (over HTTP) via the native (plugin-free) HTML5 format; on Vista SP2, the latest Firefox version I can run is ESR v52.2.1; fully updated Vista has native (OS) h264+aac decoding support via Windows Media Foundation (WMF), Firefox ESR v52.2.1 has Media Source Extensions (MSE) support plus DRM (EME) modules (CDM), so I can perfectly watch VOD on ITV Hub on Firefox (through a whitelisted UK VPN - but only the playlist API is geo-blocked, DASH streams are not - and an ITV hub account).
3. If you disable MSE support in your browser (via about:config) or in the case of your browser not supporting MSE in the first place, then you sadly get no playback at all, even with Adobe Flash plugin installed and enabled, and the ITV hub site notifies you: Sorry! We no longer support this browser, please update or try a different browser.
4. While logged in to ITV Hub, you can perform URL sniffing via Firefox's web console (or similar in other browsers); when requesting a VOD on a compatible OS+firefox version, you can filter out in the web console the URL of the MPEG-DASH playlist (MPD, Media Presentation Description); this is tokenised and is of the form:

https://itvpnpdotcom-a.akamaihd.net/2-3939-0006-003/18/1/VAR011/2-3939-0006-003_18_1_VAR011.ism/.mpd?hdnea=st=1500375357~exp=1500396957~acl=/*~hmac=2286109d6f65f41c9bae5240a4875118d167a80a1d7e23194eff98575b0a1b1b#s=0.02

However, it is only when one downloads to disk and then opens in a text editor that one realises both video+audio streams are encrypted; I am attaching a previously downloaded MPD for you to inspect (extension changed to .txt to become palatable to GitHub): ITVhub_VOD.mpd.txt

5. To the best of my knowledge, encrypted DASH hasn't been cracked, so am afraid your initial plan to replace RTMP streams with DASH is far from applicable :-(
6. At the same time itv.pm broke, so did various other ITV VOD downloading tools that relied on the RTMP streams, such as Free Rapid Downloader (written in Java) and youtube-dl (written in Python).
7. Oddly enough, users of Get-iPlayer-Automator, a Mac (OS X) only app, reported that they are still able to fetch those RTMP streams that are broken for others...
8. Being on Windows, I couldn't test GiA myself, but, lo and behold, another Python app I tested for myself, streamlink, continues to be able to fetch ITV VOD RTMP streams:

   
   streamlink -l debug --http-header "X-Forwarded-For=xx.x.xxx.xxx" -o "ITV2_FamilyGuy_Se4Ep24_DeepThroats_201707192250[2a4259a0073].flv" "https://www.itv.com/hub/family-guy/2a4259a0073" best

[cli][info] Found matching plugin itvplayer for URL https://www.itv.com/hub/fami ly-guy/2a4259a0073 [cli][info] Available streams: 600k (worst), 800k, 1200k, 1500k, 1800k (best) [cli][info] Opening stream: 1800k (rtmp) [cli][debug] Pre-buffering 8192 bytes [cli][debug] Checking file output [cli][debug] Writing stream to output [download][..192250[2a4259a0073].flv] Written 242.4 MB (3m21s @ 1.2 MB/s) [ cli][info] Stream ended

(xx.x.xxx.xxx is a random UK IP address to fool the geo-location+playlist generation APIs)

So, **the RTMP (Flash) unencrypted streams ARE STILL THERE**, somehow the itv.pm code must be patched to re-gain access to them; FWIW, python code for itvplayer.py (from streamlink) is [here](https://github.com/streamlink/streamlink/blob/master/src/streamlink/plugins/itvplayer.py); sadly, streamlink doesn't fetch subtitles, it is targeted more to live streams (being a fork of livestreamer); VOD support is just an added bonus, I guess...
9. If the worst comes, then there is another avenue that could be exploited: I have installed a firefox addon that made my desktop Firefox look in the eyes of the ITV Hub APIs like **Safari on an iPad tablet**; iPads can't cope with encrypted DASH (yet?), so the ITV Hub CDNs tried to serve my "masked" Firefox an unencrypted AppleHLS playlist that I was able to sniff in the web console: 

`https://itvpnpmobile-a.akamaihd.net/1-7842-0468-001/6/1/VAR003/1-7842-0468-001_6_1_VAR003.ism/.m3u8?hdnea=st=1500436150~exp=1500457750~acl=/*~hmac=fae2ed1e90b8591e2fe969f84f595d9295752078bdf0d1cd8d7c83a861a21804`

That was then quickly (before the token expired) dumped to disk via either FFmpeg or streamlink/livestreamer: 

`streamlink -o mobile.ts "hlsvariant://https://itvpnpmobile-a.akamaihd.net/1-7842-0468-001/6/1/VAR003/1-7842-0468-001_6_1_VAR003.ism/.m3u8?hdnea=st=1500436150~exp=1500457750~acl=/*~hmac=fae2ed1e90b8591e2fe969f84f595d9295752078bdf0d1cd8d7c83a861a21804  name_key=bitrate" best`

The drawback of the mobile streams is the highest available quality is **504p@1500kbps**, while the highest Flash quality variant is **504p@1800kbps**; but it should be adequate if all other routes become inaccessible... 

I trust you to make good use of my invaluable (hopefully) info... 

Hoping for a GFV fix soon!

[njtaylor]

That's very helpful. I'm using OpenBSD, and chrome which has a user agent string of Mozilla/5.0 (X11; OpenBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 which is the unencrypted dash, chrome allowed me to dump the network traffic.

I will look at the itvplayer.py and rtmp streams first.

[njtaylor]

Issue has been identified as a change to the productionId=x/xxxx/xxxx#xxx to productionId=url/x_xxxx_xxxx.xxx so the extracted id is wrong.

Itv.pm has been updated.

[Vangelis66]

Hi again, Nigel :smile:

It is indeed weird that on OpenBSD+Google Chrome v59.0.3071.115 you are being served unencrypted MPEG-DASH streams, while on Windows (Vista+) and recent Firefox they are serving encrypted ones. I'm not a Google Chrome fan, but out of curiosity I did try GC on my setup (mind you, the last version of Google Chrome that would run on Vista SP2 is 50); being consistent with desktop FirefoxESR v52.2.1, on desktop Google Chrome v50.0.2661.102 I am still being served MPEG-DASH streams from the ITV Hub player:

When the MPD is inspected, it definitely references ENCRYPTED streams:

(snip)
xmlns:cenc="urn:mpeg:cenc:2013"
(snip)
<!-- Common Encryption -->
      <ContentProtection
        schemeIdUri="urn:mpeg:dash:mp4protection:2011"
        value="cenc"
        cenc:default_KID="0F59445A-C6B9-404C-92F4-EE55C7762F5D">
      </ContentProtection>
      <!-- ClearKey -->
      <ContentProtection
        schemeIdUri="urn:uuid:1077EFEC-C0B2-4D02-ACE3-3C1E52E2FB4B">
      </ContentProtection>
      <!-- Widevine -->
      <ContentProtection
        schemeIdUri="urn:uuid:EDEF8BA9-79D6-4ACE-A3C8-27DCD51D21ED">
      </ContentProtection>
      <!-- Marlin -->
      <ContentProtection
        schemeIdUri="urn:uuid:5E629AF5-38DA-4063-8977-97FFBD9902D4">
<mas:MarlinContentIds>
  <mas:MarlinContentId>urn:marlin:kid:0f59445ac6b9404c92f4ee55c7762f5d</mas:MarlinContentId>
</mas:MarlinContentIds>
      </ContentProtection>

Furthermore, I tried downloading that MPD (while its token was still valid) with youtube-dl, but the tool FAILED to initiate the download :-( . So I'm asking again: Are you absolutely positive you're being served plain unencrypted DASH streams in your setup?

chrome allowed me to dump the network traffic.

Can you please elaborate? Was it a playable unencrypted file you ended up with? FWIW, on their help pages they apparently only support the following OS+browser combos:

The ITV Hub actively supports the latest versions of most major browsers
- Internet Explorer 11 - Windows 8 and higher
- Edge - Windows 10 and higher
- Chrome - Windows 7 and higher, OSX 10.10* and higher
- Safari - OSX 10.10* and higher, iOS 10 and higher
- Firefox - Windows 7 and higher, OSX 10,10* and higher

Itv.pm has been updated.

As ever, greatly appreciated :+1:; pending a new _combined-get_flashvideos release, I had to patch the previous release (combined-get_flash_videos-1.25.92) manually with a text editor as per commit https://github.com/monsieurvideo/get-flash-videos/commit/e55ddb9 (or use the "dev" edition extracted into a folder). Things seem to work well again:

Using method 'itv' plugin version 0.09 for https://www.itv.com/hub/amazing-anima
l-births/2a3939a0004
Downloading https://www.itv.com/hub/amazing-animal-births/2a3939a0004
Subtitle Fetching
Subtitle URL http://subtitles.secure.content.itv.com/CATCHUP/7fbc1169/dd19/467b/
9520/5f42ed643646/2-3939-0004-003_AmazingAnimalBirths_TX030717.xml?__gda__=15006
13411_167bf3e959c7b6cc247c403e04cc1f2b&fileExt=.xml
Saved subtitles to 2-3939-0004-003_Amazing-Animal-Births-Episode-4_PC011800_16X9
.srt
swf type = CWS version = 10 size = 990750
RTMPDump v2.4-102-ga3a600d-get_iplayer
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Metadata:
INFO:   duration              1311.92
INFO:   moovPosition          32.00
INFO:   width                 896.00
INFO:   height                504.00
INFO:   videocodecid          avc1
INFO:   audiocodecid          mp4a
INFO:   avcprofile            100.00
INFO:   avclevel              31.00
INFO:   aacaot                2.00
INFO:   videoframerate        25.00
INFO:   audiosamplerate       48000.00
INFO:   audiochannels         2.00
INFO: trackinfo:
INFO:   length                32798.00
INFO:   timescale             25.00
INFO:   language              und
INFO: sampledescription:
INFO:   sampletype            avc1
INFO:   length                62970880.00
INFO:   timescale             48000.00
INFO:   language              und
INFO: sampledescription:
INFO:   sampletype            mp4a
2-3939-0004-003_Amazing-Animal-...800_16X9.flv: 99% (247028.14 / 248519.26 KiB)
Done. Saved 253249315 bytes to 2-3939-0004-003_Amazing-Animal-Births-Episode-4_P
C011800_16X9.flv

Seeing that they have now practically decommissioned their RTMP (Flash) streams, is there a contingency plan in the works for when/if they completely shut down their RTMP VOD infrastructure?

Finally, and I don't want to sound ungrateful or greedy, I do want to request an enhancement to the Itv.pm, possibly echoing many overseas users: The ITV Hub team are notorious for blacklisting

1. the majority of the Tor Network UK exit nodes,
2. most working open UK HTTP(S) proxies (that a Google search might yield),
3. many commercial UK Proxy/VPN/VPS providers (thankfully not all of them...).

As I posted previously, the RTMP streams themselves are not geo-blocked at CDN level, only the request to their playlist API must come from a whitelisted UK IP address. Using a UK VPN for an ITV fetch needlessly slows down speeds, because the RTMP streams too are routed via the VPN server.

I am currently using a whitelisted UK HTTP proxy and the --proxy "http://proxyhost:proxyport" option in GFV, so the Flash streams are downloading DIRECT, but;

1. When we'll have to resort to HTTP(S) protocol streams (e.g. HLS/DASH?), I fear the --proxy switch would apply to the streams themselves, too; this won't be needed if the streams are not blocked (currently they're not); so maybe a --partial-proxy switch (a la get_iplayer) would have to be implemented...
2. As things stand at the time of writing, their geo-location API can be "fooled" by simply including an "X-Forwarded-For=" header in the playlist request; no need for a proper UK proxy or a full-blown UK VPN solution; the streamlink devs have implemented an "X-Forwarded-For=*" switch for many geo-filtered media providers, while youtube-dl is far more versatile with regards to that:

   Geo Restriction:
   --geo-verification-proxy URL     Use this proxy to verify the IP address for
                                    some geo-restricted sites. The default
                                    proxy specified by --proxy (or none, if the
                                    options is not present) is used for the
                                    actual downloading.
   --geo-bypass                     Bypass geographic restriction via faking X
                                    -Forwarded-For HTTP header (experimental)
   --no-geo-bypass                  Do not bypass geographic restriction via
                                    faking X-Forwarded-For HTTP header
                                    (experimental)
   --geo-bypass-country CODE        Force bypass geographic restriction with
                                    explicitly provided two-letter ISO 3166-2
                                    country code (experimental)

   The above is an excerpt from --help; in the case of many geo-restricted plugins (itv.py included), there's a built-in mechanism that kicks in automatically, see the Python code here...

I fully understand that currently GFV is quite understaffed, but could something like that be implemented for Itv.pm too, so that no need for UK proxy/VPN when fetching from overseas? A randomly generated UK IP address is less prone to becoming blacklisted!

With the hope my suggestion be considered in the future, many thanks again for all the hard work you have put thus far towards maintaining (often single-handedly) the ITV plugin :+1:

[njtaylor]

I will create a release with the download files shortly.

Gone back and checked and was wrong dash is being encrypted, there is a but here, OpenBSD chrome I use is built from the source, which means somewhere in the source is the means to decrypt the dash, or a supplied binary that does it (maybe a jar file).

A partial proxy I can look at, however being located in the UK it may prove difficult to test. Adding a header such as X-Forwarded-For is simpler. May consider a mean of adding own headers.

There is no code to handle any dash in GFV yet.

[Vangelis66]

I will create a release with the download files shortly.

Yes, extremely thankful for that! Both combined-get_flash_videos-1.25.93 and get_flash_videos-1.25.93 work fine on my setup for ITV Hub VOD!

and was wrong, dash is being encrypted

Thought as much... Thanks for re-checking and verifying the bad news...

A partial proxy I can look at, however being located in the UK it may prove difficult to test

There are many free resources to mask your actual physical location; being in the UK is a plus in your case, since you simply have to choose one of the many overseas VPN locations; e.g. if OpenVPN is available in OpenBSD (or you can compile from source), then take a look here...

There is no code to handle any dash in GFV yet.

Unfortunately, this is a more generalised problem ;-( . Apart from get_iplayer 3.01, which specialises only in BBC iPlayer TV+Radio DASH streams, and youtube-dl, which can only fetch "static" - non-live - MPDs, there's a lack of free tools/scripts to download MPEG-DASH streams; which is kind of an odd thing, because MPEG-DASH is an open source implementation. VLC 3.0.0-git Nightly version has some preliminary support for playing back (and recording) certain dash streams, but that's it for the moment!

Adding a header such as X-Forwarded-For is simpler. May consider a mean of adding own headers.

... Will be keeping an eye in this repo for this... Take the best of care!

[chancemimi]

where to get combined-get_flash_videos-1.25.93 and get_flash_videos-1.25.93 ?

[njtaylor]

You get from here, click on code 2-3rd line down at top, should see appear below, commits, branches, releases, contributors. Select releases - combined-get_flash_videos-1.25.93 and get_flash_videos-1.25.93 should appear under the Downloads. If not, can use link below...

Releases

[chancemimi]

THANKS, njtaylor! I remember in around the year 2014, I was able to use get_flash_videos to download very wonderful shows from hulu. But the hulu.pm did not work any more. Is there any ways to get videos from hulu now ?

[njtaylor]

The hulu.pm was a separate project. 2014 hulu switched to newer flash, and became subscription only. I only maintain the itv plugin, other site plugins are supplied by others, I can create releases that include that they supply. Alternatives like youtube-dl, and kodi don't support hulu any more.

[Vangelis66]

In addition to the above, all hulu streams, at least on desktop browsers, are heavily encrypted, which is a no go by itself; unless you're a member of a highly expert hacker team (like the one that hacked HBO), use a screen recording software, if you can...

[idealitystarts]

Allavsoft still works great to download ITV videos.

[njtaylor]

Allavsoft still works great to download ITV videos.

Keep to the topic ITV downloads work for most content, only content with additional encryption fails to download, comments aren't for advertising paid for software that only work on limited platforms.

[Vangelis66]

comments aren't for advertising paid for software that only work on limited platforms.

@njtaylor: The user above joined GitHub less than a month ago, with sole agenda apparently to spam repositories related to media content with plugs for paid-for third party Windows software; proof:

https://github.com/jellyfin/jellyfin-kodi/issues/20#issuecomment-533364350

https://github.com/croneter/PlexKodiConnect/issues/825#issuecomment-533366483

https://github.com/r0oth3x49/udemy-dl/issues/303#issuecomment-533935258

https://github.com/streamlink/streamlink/issues/1046#issuecomment-533985584

https://github.com/sannies/mp4parser/issues/231#issuecomment-536237423

I would permanently block and report to GitHub (... but sadly those spammers tend to come back under a different alias) ...

Regards