Some recent ITVHub additions fail to fetch

[Vangelis66]

Hi @njtaylor, I do hope you're well 😃

This issue title might sound as a dupe of #221, but my gut feeling is the underlying root cause is different... :confused:

At least two recent ITVHub offerings,

https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034
https://www.itv.com/hub/harry-hills-alien-fun-capsule/2a4728a0009

fail to download; both were first broadcast last Saturday, Apr 21st 2018. Testing platform is:

OS: Windows Vista SP2 32-bit, + MS updates for WS2008SP2 (same kernel version, in extended support until Jan 2020) Perl version: 5.24.0.1-32bit-portable + GFV dependencies, installed manually via CPAN GFV version: Latest combined script from the releases section of this repo (combined-get_flash_videos-1.25.99.01) renamed for brevity to c-gfv-1_25_99_01 Command issued:

perl scripts/c-gfv-1_25_99_01 "https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034" --debug

Cmd Prompt Window output:

Using method 'itv' plugin version 0.09.03 for https://www.itv.com/hub/james-mart
ins-saturday-morning/2a5159a0034
Downloading https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034
-> GET https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034
<- 200 text/html; charset=UTF-8 15889 (82784): UTF8 on, ASCII, 82784 characters
82784 bytes
~0034#001

~s:Envelope>
Error: Unable to find <Video> in XML at scripts/c-gfv-1_25_99_01 line 20585.

Couldn't extract Flash movie URL. This site may need specific support adding,
or fixing.

Please confirm that the URL is valid and that you are running the latest
version of get-flash-videos.

If the latest version does not support this site please create a bug
report at https://github.com/monsieurvideo/get-flash-videos making
sure you include the output with --debug enabled. Alternatively,
fix it yourself and send us a pull request.
Couldn't download any videos.

... and same goes for the second programme 😠

Issue was first reported here, as a result I created a related issue in the yt-dl tracker, where you can read more details... :wink: Downloadable HLS streams do exist for these shows, it's probably a recent change in ITVHub's side that makes them undetectable currently 😢 ; I hope your coding wizardry can assist...

Note to overseas users: Since mid-March 2018, ITVHub are geo-fencing HLS streams at CDN level, hence you now need a geolocation circumvention solution to download those streams (more here).

Best regards

[njtaylor]

I encountered the same myself with James Martin, using itv search selecting the first episode from the list.

Browsers when viewed with firefox, it failed, viewing with chrome, it worked, looking at the traffic chrome was using a dash download. However I had found ITV doing some additional browser detection. switch chrome to ipad it stops working suggesting no hls is available.

From looking at Lethal Weapon, support on ITV, it states only available from certain devices like Windows 10.

It seems ITV will stream as flash, hls, or dash. Using dash for restricted content. That would be fine but they seem to apply the rules with no consistency, Lethal Weapon some weeks can be downloaded as flash some as hls, when it should be dash only content.

Then as with James Martin, they get it wrong and is only available as protected content. But other new content was available as flash downloads. Sometimes they fix a few days later so I was waiting to see if changes.

Only the very early start for implementing dash, and not sure if I can, limited time available to work on this at the moment...

$ perl -Ilib get_flash_videos -r 1200 -t dash itv James Martin James martins saturday morning 5 episodes available to watch Search for 'itv James Martin' found these results: [ 1] James Martin's Saturday Morning - Series 1 episode Episode 34 - Saturday 21 Apr 9.25am (itv) Cookery show. Singer Lemar and chef Daniel Galmiche join James for a morning of good food. [ 2] James Martin's Saturday Morning - Series 1 episode Episode 33 - Saturday 14 Apr 9.25am (itv) Cookery show. Anton du Beke and Brian Turner join James for fish finger sandwiches. [ 3] James Martin's Saturday Morning - Series 1 episode Episode 32 - Saturday 7 Apr 9.30am (itv) Cookery show. Comedian Chris Ramsey and chef Kenny Atkinson join James Martin. [ 4] James Martin's Saturday Morning - Series 1 episode Episode 31 - Saturday 31 Mar 9.30am (itv) Cooking show. Music producer Naughty Boy and chef Lenny Carr-Roberts join James Martin. [ 5] James Martin's Saturday Morning - Series 1 episode Episode 30 - Saturday 24 Mar 9.30am (itv) Corrie actor Sean Wilson and chef Tony Tobin join James Martin for a morning of good food. Enter the number(s) or range (e.g. 1-3) of the videos to download (separate multiple with comma or space): 1 Using method 'itv' plugin version 0.09.04 for https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034 Downloading https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034 DASH Content Not implemented yet

[Vangelis66]

Thanks for your swift reply; just let me get some facts right, for the benefit of both you and others following this... 😉 😄

Browsers when viewed with firefox, it failed, viewing with chrome, it worked, looking at the traffic chrome was using a dash download

This part was again discussed back in https://github.com/monsieurvideo/get-flash-videos/issues/211#issuecomment-316590795 and below; TL:DR:

On desktop browsers, notably Mozilla Firefox and Google Chrome (and its forks), ITVHub serves encrypted (i.e. DRMed) MPEG-DASH streams. The viewing of these streams locally require the browser is built with EME (Encrypted Media Extensions) support, EME being closed-source; EME handles the on-the-fly acquisition of decryption keys, while the decryption of streams is delegated to a CDM (Content Decryption Module), itself closed-source; Firefox 51.0+ on Windows has support for the Google-owned Widevine CDM, which has to be explicitly enabled in the browser settings. I re-iterate, I have no issue whatsoever watching those DRMed streams on latest FirefoxESR 52.7.3 (32-bit) with DRM enabled: about:preferences#content => DRM selected and about:addons => Plugins => Widevine CDM provided by Google v1.4.8.903: Always enabled My Firefox Windows binary has the following inside about:buildconfig:

--enable-eme --with-mozilla-api-keyfile=c:/builds/mozilla-desktop-geoloc-api.key --with-google-api-keyfile=c:/builds/gapi.data

I can only speculate your Firefox binary on OpenBSD has been compiled without DRM support (e.g. like IceCat/IceWeasel, which oppose the blackboxed nature of DRM), hence the encrypted streams fail there... Google Chrome has native support for DRM (and Widevine?), so that's why it's able to play these streams.

Only the very early start for implementing dash, and not sure if I can, limited time available to work on this at the moment...

If you want to add MPEG-DASH downloading capabilities to GFV, that would be tremendous, as many sites are switching to this method (which is open-source, not proprietary like AdobeHDS/AppleHLS); recently the FFmpeg project has added preliminary support for DASH streams, in the form of a dash-demuxer, so you may consider that option, too...; but if you want to implement MPEG-DASH for the sake of ITVHub, don't waste your time at all with that project! Unless, of course, you have devised a way to break Common Encryption (CENC) used in those streams, which would be a ground-breaking development, one that would cause a major sensation web-wide (but will be instantly patched by the powers that be...)!

From looking at Lethal Weapon support on ITV, it states only available for certain devices like Windows 10.

This is because the on-line rights for the Lethal Weapon TV series negotiated by ITVHub only permit them to use a special kind of DRM protection, called PlayReady; this is a proprietary protection scheme owned by Microsoft, as such it can only run in Win10's Internet Explorer 11 (possibly also in Win8.1's IE11, not quite sure about that OS...).

It seems ITV will stream as flash, hls, or dash

Flash is a legacy infrastructure, I don't think they are normally deploying RTMPE streams for VOD content (maybe on some mobile devices?) currently, most certainly not on desktop browsers; only the live streams of ITV1+ITV4 use Flash at the moment, while ITV2+ITV3 use DRMed MPEG-DASH. As you say, the production of RTMPE streams is very erratic; some shows immediately have them, others only several days after DASH/HLS streams have been produced, in many (especially recent episodes) cases, no RTMPE streams will be provided for the duration the shows are available on-line...

switch chrome to ipad it stops working suggesting no hls is available.

Both Firefox and Chrome lack native support for HLS streams; there exist extensions (Fx: 1, Chrome: 2) that mitigate this; however, ITVHub will only serve HLS streams on their approved clients/platforms!

I don't use Chrome myself, but with Mozilla Firefox, if I spoof its user agent string to an iPhone one,

general.useragent.override;Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3

navigate to

https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034

sign-in to my ITVHub account, bring up Web Console (CTRL+SHIFT+K) and hit play button, I can sniff HLS playlists served to me:

https://itvpnpmobile-a.akamaihd.net/2-5159-0034-001/6/1/VAR003/2-5159-0034-001_6_1_VAR003.ism/.m3u8?hdnea=st=1524525929~exp=1524547529~acl=/*~data=nohubplus~hmac=34e894d979a530d26690ac61e9812b4e2806c1214f3775e3d085567c5db56822

Of course Firefox won't be able to play the stream, but if I am swift enough to feed that playlist to FFmpeg, youtube-dl, streamlink etc., I can dump that stream to file:

youtube-dl --console-title --proxy "socks5://127.0.0.1:1080" "https://itvpnpmobile-a.akamaihd.net/2-5159-0034-001/6/1/VAR003/2-5159-0034-001_6_1_VAR003.ism/.m3u8?hdnea=st=1524525929~exp=1524547529~acl=/*~data=nohubplus~hmac=34e894d979a530d26690ac61e9812b4e2806c1214f3775e3d085567c5db56822" --hls-prefer-native --hls-use-mpegts --no-part -o "James Martin's Saturday Morning - Se01Ep34[2a5159a0034].ts" --fixup warn

[generic] *~data=nohubplus~hmac=34e894d979a530d26690ac61e9812b4e2806c1214f3775e3
d085567c5db56822: Requesting header
[redirect] Following redirect to https://itvpnpmobile-a.akamaihd.net/2-5159-0034
-001/6/1/VAR003/2-5159-0034-001_6_1_VAR003.ism/.m3u8?hdnea=st=1524525929~exp=152
4547529~acl=/*~data=nohubplus~hmac=34e894d979a530d26690ac61e9812b4e2806c1214f377
5e3d085567c5db56822&redirect=1
[generic] *~data=nohubplus~hmac=34e894d979a530d26690ac61e9812b4e2806c1214f3775e3
d085567c5db56822&redirect=1: Requesting header
[generic] *~data=nohubplus~hmac=34e894d979a530d26690ac61e9812b4e2806c1214f3775e3
d085567c5db56822&redirect=1: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 896
[download] Destination: James Martin's Saturday Morning - Se01Ep34[2a5159a0034].
ts
[download]   5.9% of ~1.04GiB at 146.18KiB/s ETA 29:26

Again, as I have demonstrated, AppleHLS streams do exist for that episode, it is my sense they have only changed some strings in the playlist response file; hopefully you can fine-tune some regex to identify the HLS streams anew...

Don't give up on this!

Best wishes

[njtaylor]

As you say looking and hls is supported. A different fault code is being returned, for the xml request this change in Itv.pm should fix the issue

   if ( $hls_dl && (! $rtmp_dl || ($rtmp_dl && 
-        $browser->content =~ m%<faultcode>InvalidEntity</faultcode>%) ) ) {
+        $browser->content =~ m%<faultcode>Invalid[A-z]*</faultcode>%) ) ) {

However this is the response from trying the first part of a rtmp download. You can force hls only by using get_flash_videos -t hls ...... This downloads the James Martin

$ get_flash_videos -t hls -r 1200 itv james martin
James martins saturday morning 4 episodes available to watch 
Search for 'itv james martin' found these results:
[ 1] James Martin's Saturday Morning - Series 1 episode Episode 34  -  Saturday 21 Apr 9.25am (itv)
      Cookery show. Singer Lemar and chef Daniel Galmiche join James for a morning of good food. 
[ 2] James Martin's Saturday Morning - Series 1 episode Episode 33  -  Saturday 14 Apr 9.25am (itv)
      Cookery show. Anton du Beke and Brian Turner join James for fish finger sandwiches. 
[ 3] James Martin's Saturday Morning - Series 1 episode Episode 32  -  Saturday 7 Apr 9.30am (itv)
      Cookery show. Comedian Chris Ramsey and chef Kenny Atkinson join James Martin. 
[ 4] James Martin's Saturday Morning - Series 1 episode Episode 31  -  Saturday 31 Mar 9.30am (itv)
      Cooking show. Music producer Naughty Boy and chef Lenny Carr-Roberts join James Martin. 
Enter the number(s) or range (e.g. 1-3) of the videos to download (separate multiple with comma or space): 1
Using method 'itv' plugin version 0.09.04 for https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034
Downloading https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034
Downloading 448 segments
100% [=============================================================================================================================================================================================================]
Done. Saved 692428503 bytes to 2-5159-0034-001_James_Martin_s_Saturday_Morning_-_Series_1_-_Episode_34_hls.mp4

Try for now with -t hls, too late to finish off things tonight.

[njtaylor]

1.25.99.02 release should fix the issues, run on OpenBSD. Other platforms not tested.

[Vangelis66]

Apologies for tardy response, many thanks for your dedication! 👍

Another recent programme which falls into this issue is https://www.itv.com/hub/american-dad/2a4263a0229

As you say looking and hls is supported. (snip) However this is the response from trying the first part of a rtmp download. You can force hls only by using get_flash_videos -t hls ... This downloads the James Martin

Confirmed; I had to first install LWP::Protocol::socks perl module via CPAN to be able to use my (local) SOCKS5 proxy (dynamic forwarding of SSH tunnel to the UK), but then an incantation similar to

perl scripts/c-gfv-1_25_99_01 --proxy "127.0.0.1:1080" "https://www.itv.com/hub/james-martins-saturday-morning/2a5159a0034" -t hls

successfully started a download...

1.25.99.02 release should fix the issues

I can confirm, at least with my setup (see first post):

perl scripts/c-gfv-1_25_99_02 --proxy "127.0.0.1:1080" "https://www.itv.com/hub/american-dad/2a4263a0229" =>

Using method 'itv' plugin version 0.09.04 for https://www.itv.com/hub/american-d
ad/2a4263a0229
Downloading https://www.itv.com/hub/american-dad/2a4263a0229
No rtmp download found
Downloading 102 segments
  9% [=======                                                                 ]

So, immensely obliged, many thanks for the timely fix :1st_place_medal: :+1:

Closing as "Solved"

OT: If/when your free time permits, could you be so kind as to concoct a patch for get_iplayer that implements SOCKS(4/5) support? Both get_flash_videos (perl) and youtube-dl (python) support such an option, but GiP only supports HTTP(s) proxies; I have to apply proxy-chaining currently (HTTP proxy => SOCKS5 proxy => SSH tunnel) to be able to use GiP, native support for socks would eliminate the need for HTTP proxy... BTW, the current maintainer would never accept such a feature request, especially coming from me... Apologies for bringing this up here, I saw no harm in just asking (LOL, you can simply refuse! 😉 )

Best wishes

[njtaylor]

Thanks for closing.

On the subject of get_iplayer, not tried this but if you have LWP::Protocol::Socks installed then, gfv defaults to --proxy to socks by appending socks://, in fact --proxy 127.0.0.1 should be enough on the command line, gets expanded in the code to socks://127.0.0.1:1080 in gfv.

So for get_iplayer --proxy socks://127.0.0.1:1080 would have to be used.

$ get_iplayer --proxy socks://127.0.0.1:1080 
get_iplayer v3.13, Copyright (C) 2008-2010 Phil Lewis
  This program comes with ABSOLUTELY NO WARRANTY; for details use --warranty.
  This is free software, and you are welcome to redistribute it under certain
  conditions; use --conditions for details.

INFO: Using proxy: socks://127.0.0.1:1080

Haven't tried this it may work. You do need to have LWP::Protocol::Socks installed, gfv does a check to see if installed if proxy start socks: once expanded, get_iplayer doesn't check.

[Vangelis66]

Thanks for your input on my OT query ❤️

but if you have LWP::Protocol::Socks installed then gfv defaults to --proxy to socks by appending socks://, in fact --proxy 127.0.0.1 should be enough on the command line, gets expanded in the code to socks://127.0.0.1:1080 in gfv.

Yes, I figured that part by reading GFV's --help output: perl Scripts\c-gfv-1_25_99_02 -h > Help-1_25_99_02.txt 2>&1 =>

Options:
(snip)
     --proxy      Proxy to use, use host:port for SOCKS, or URL for HTTP.

You do need to have LWP::Protocol::Socks installed, gfv does a check to see if installed if proxy start socks: once expanded

Yes, after I included --proxy 127.0.0.1:1080 in my command, I was instructed to install LWP::Protocol::Socks

So for get_iplayer --proxy socks://127.0.0.1:1080 would have to be used. (snip) You do need to have LWP::Protocol::Socks installed (snip) get_iplayer doesn't check.

First problem with GiP on Windows is that the official Windows installer doesn't have LWP::Protocol::Socks installed/included in the minimal Strawberry Perl 5.26.1.1 package that comes with it; nor is there a way to install additional modules to that package 😞 So I have to try the GiP perl script on my full-blown SP 5.24.0.1-32bit-portable installation (470+ MiB!).

Trying a small video clip download (pid=p062qcym),

perl Scripts/get_iplayer-313w.pl -p socks://127.0.0.1:1080 --type=tv --pid=p062qcym --tvmode=hvfworst -w --force =>

get_iplayer 3.13.0, Copyright (C) 2008-2010 Phil Lewis
  This program comes with ABSOLUTELY NO WARRANTY; for details use --warranty.
  This is free software, and you are welcome to redistribute it under certain
  conditions; use --conditions for details.

INFO: Using proxy: socks://127.0.0.1:1080
Episodes:
The Sky at Night - Starguide: April 2018, BBC iPlayer, p062qcym
INFO: 1 total programmes

WARNING: A UK TV licence is required to access BBC iPlayer TV content legally
INFO: Downloading tv: 'The Sky at Night - Starguide: April 2018 (p062qcym) [orig
inal]'
 42.4% of ~16.47 MB @   3.1 Mb/s ETA: 00:00:25 (hvflow1/bi) [audio+video]
(after a short while)
INFO: Downloaded: 7.07 MB (00:03:40) @ 2.09 Mb/s (hvflow1/bi) [audio+video]
INFO: Converting to MP4
ERROR: Thumbnail download failed
ERROR: Use --thumbnail-only to re-download
WARNING: Required AtomicParsley utility not found - cannot tag MP4 file

which proves your recommendation does indeed work :clap: (but my SP portable installation isn't able to find AP binary for tagging...; I think I know how to fix that...)!

Of course, next challenge for me was to somehow install LWP::Protocol::Socks on the "official" Windows perl package that is bundled with the installer; my aim was to selectively copy files from my proper (full-blown) SP portable install to the mini-package; I had to first identify what was installed by cpan LWP::Protocol::Socks, where it ended up inside portable perl's file structure
and move that over to the mini-package... It took me well over an hour of trial-and-error, but finally managed it!

".\strawberry-perl-5.24.0.1-32bit-portable\perl\site\lib\LWP\Protocol\socks.pm" => "C:\Program Files\get_iplayer\perl\lib\LWP\Protocol\socks.pm"

".\strawberry-perl-5.24.0.1-32bit-portable\perl\site\lib\LWP\Protocol\socks4.pm" => "C:\Program Files\get_iplayer\perl\lib\LWP\Protocol\socks4.pm"

".\strawberry-perl-5.24.0.1-32bit-portable\perl\site\lib\URI\socks.pm" => "C:\Program Files\get_iplayer\perl\lib\URI\socks.pm"

".\strawberry-perl-5.24.0.1-32bit-portable\perl\site\lib\URI\socks4.pm" => "C:\Program Files\get_iplayer\perl\lib\URI\socks4.pm"

... and the hardest to figure out:

".\strawberry-perl-5.24.0.1-32bit-portable\perl\vendor\lib\IO\Socket\Socks.pm" => "C:\Program Files\get_iplayer\perl\lib\IO\Socket\Socks.pm"

Though I understand this is highly OT for this tracker, I have documented my experiments here for the benefit of other Windows users of GiP (with the need to use a SOCKS proxy 😀 within the standard install of GiP on Windows); my OS is 32-bit, on 64-bit WinOSes the default installation path is "C:\Program Files (x86)\get_iplayer"

Thank you again, Nigel, for providing direction towards the resolution of a long-standing issue of mine; you're a true gem! 👍 👍 :+1: