ITVPlayer site changes break itv.pm module

GoogleCodeExporter commented 8 years ago

The ITV site has changed to use SEO-friendly URLs instead of specifying the 
video to watch with the ?Filter=,,, parameter.  Consequently, the itv.pm module 
fails before even trying to download anything as the regexp
  my($id) = $browser->uri =~ /Filter=(\d+)/;
doesn't match the new URLs.

What steps will reproduce the problem?
1. get_flash_videos 
https://www.itv.com/itvplayer/pokemon/series-12/guarding-the-guardian-of-the-mou
ntain-

What is the expected output? What do you see instead?

Expected output would be downloading the episode.  Instead what I get is:

Using method 'itv' for 
https://www.itv.com/itvplayer/pokemon/series-12/guarding-the-guardian-of-the-mou
ntain-
Downloading 
https://www.itv.com/itvplayer/pokemon/series-12/guarding-the-guardian-of-the-mou
ntain-
Error: No id (filter) found in URL

Couldn't extract Flash movie URL. This site may need specific support adding,
or fixing.

Please confirm the site is using Flash video and if you have Flash available
check that the URL really works(!).

What version of the product are you using? On what operating system?
Recent git version, on Ubuntu 12.04

Please provide any additional information below.

Original issue reported on code.google.com by insomnia...@googlemail.com on 5 Dec 2012 at 5:45

GoogleCodeExporter commented 8 years ago

I tried your URL in a browser and it doesn't exist. https is strange not 
normally needed, ending in a "-" doesn't look right as if there should be more 
to the URL. I have downloaded programmes from ITV recently and not had any 
problems, in fact doing so now for a programme aired today for the first time. 
Always the URL I see they take the form of
http://www.itv.com/itvplayer/Video/?Filter=327434

So how have you got the URL your using, I can't find it, or how you got it, 
navigated to it? I tried looking at the ITV site and I don't get any new form 
of URL. I can't find a matching episode description for the one your 
downloading on ITV, searching the ones I found are Series 14 Episode 78 and 
Series 15 Episode 30. I can find what looks to be series 15 episodes 31-34 on 
the ITV site, episode 30 is possibly too old from the dates to still be on the 
ITV site over the 30 day limit.

Original comment by njtaylor...@gmail.com on 5 Dec 2012 at 10:22

GoogleCodeExporter commented 8 years ago

ITV are trialling a new version of the itvplayer (accessible from a link on 
their 
homepage http://www.itv.com/itvplayer/)

When you click on that it changes over to https and the programme links change 
to
https://www.itv.com/itvplayer/agatha-christie  format.

You can get back to the old version (for now anyway) by clearing your cookies 
for itv and hitting their page again.

Original comment by richard....@gmail.com on 7 Dec 2012 at 12:37

GoogleCodeExporter commented 8 years ago

That helps, the https url only works if the cookie itv_player_switch is set. 
Otherwise it returns it location has moved. We can look into making the itv 
player beta version work. For now downloads still work using the current 
version.

The beta ITV iplayer also allows rentals.

Original comment by njtaylor...@gmail.com on 7 Dec 2012 at 5:35

GoogleCodeExporter commented 8 years ago

Thanks for the tip about the cookies, I can now see the old URLs again.  I'm 
not aware of choosing the beta anywhere, guess maybe I was selected by some A-B 
testing or a search or channel link switched me over.  Just rotten luck it 
happened as I was trying to download an episode as it had only a couple of 
hours left and my son hadn't seen that one yet...  Oh well, at least you got a 
bit of warning of the changes before things broke completely :-)

Original comment by insomnia...@googlemail.com on 8 Dec 2012 at 4:34

GoogleCodeExporter commented 8 years ago

I have the same problem but deleting the cookies does not help, nor do they 
have a selectable beta option on http://www.itv.com/itvplayer; have they have 
they made this change permanent now?

Original comment by dwhelan...@gmail.com on 20 Dec 2012 at 2:02

GoogleCodeExporter commented 8 years ago

It appears ITV have updated the site making the change permanent. The filter 
format still does work if you can find the number. For example these should 
work..

Coronation Street Mon 10 Dec 2012 
http://www.itv.com/itvplayer/Video/?Filter=328761
Coronation Street Fri 14 Dec 2012 
http://www.itv.com/itvplayer/Video/?Filter=328782
Coronation Street Mon 17 Dec 2012 
http://www.itv.com/itvplayer/Video/?Filter=328953
Coronation Street Mon 17 Dec 2012 
http://www.itv.com/itvplayer/Video/?Filter=328954
Coronation Street Wed 19 Dec 2012 
http://www.itv.com/itvplayer/Video/?Filter=328975

Original comment by njtaylor...@gmail.com on 20 Dec 2012 at 2:48

GoogleCodeExporter commented 8 years ago

Thanks, but how did you find these numbers?

Original comment by dwhelan...@gmail.com on 20 Dec 2012 at 7:06

GoogleCodeExporter commented 8 years ago

I used this - 
get_iplayer --type itv --listformat "<name> <episode> <web>" Coro

I have a plugin for ITV I updated from the get_iplayer original, only downloads 
the available programmes, doesn't do actual downloads, never got around to 
updating that part. There is another one available which I haven't tried.

I combine this with get_flash_videos in a script which keep a history of 
downloads to avoid trying previous downloads. Allows me to search by name, 
rather than visiting ITV, website for the Filter number.

The other get_iplayer ITV plugin is here, don't know if this works, and may 
work differently..
http://code.google.com/p/get-iplayer-automator

Original comment by njtaylor...@gmail.com on 20 Dec 2012 at 8:24

GoogleCodeExporter commented 8 years ago

My get_iplayer version (2.80) doesn't allow an itv type: which version are you 
using?

Original comment by dwhelan...@gmail.com on 20 Dec 2012 at 9:23

GoogleCodeExporter commented 8 years ago

I use OpenBSD, with get_iplayer v2.82 which includes some upstream patches.

Place the file itv.plugin file in ~/.get_iplayer/plugins

For all users the plug-ins can also be placed into
/usr/local/share/get_iplayer/plugins might use slightly different path
on Linux.

I looked back and the plugins feature was in version 2.76 of get_iplayer.

Original comment by njtaylor...@gmail.com on 21 Dec 2012 at 1:11

GoogleCodeExporter commented 8 years ago

Please excuse my lack of knowledge, I am learning but it's slow going.

I can't seem to find an itv plugin in version 2.80, nor in 2.82, and version 
2.76 is no longer available at ftp://ftp.infradead.org/pub/get_iplayer/. Can 
you let me know where I can find a copy of itv.plugin?

Also, I have been trying to get the other get_iplayer ITV plugin you mention at 
http://code.google.com/p/get-iplayer-automator to work; this plugin seems to be 
a template for writing your own plugins; I am progressing but having trouble 
figuring out how to turn it into an itv plugin.

Original comment by dwhelan...@gmail.com on 21 Dec 2012 at 4:54

GoogleCodeExporter commented 8 years ago

This is not a template...
http://code.google.com/p/get-iplayer-automator/source/browse/trunk/plugins/itv.p
lugin

This only has templates.
http://git.infradead.org/get_iplayer.git/tree/HEAD:/plugins

Attached is my copy... The old repository where this came from is no
longer available.

Original comment by njtaylor...@gmail.com on 21 Dec 2012 at 6:28

GoogleCodeExporter commented 8 years ago

That's great thanks. I couldn't find your attachment but plugin at the "not a 
template" link works well.

Original comment by dwhelan...@gmail.com on 21 Dec 2012 at 10:27

GoogleCodeExporter commented 8 years ago

There is an update here

https://github.com/njtaylor/get-flash-videos/raw/master/lib/FlashVideo/Site/Itv.
pm

For anyone want to test this, download and copy into 
~/.get_flash_videos/plugins 
(once Itv.pm is released the downloaded version should be removed).

This is still being tested, the following works...

$ get_flash_videos 
https://www.itv.com/itvplayer/pokemon/series-12/episode-48-meloetta-and-the-unde
rsea-temple-
Using method 'itv' for 
https://www.itv.com/itvplayer/pokemon/series-12/episode-48-meloetta-and-the-unde
rsea-temple-
Downloading 
https://www.itv.com/itvplayer/pokemon/series-12/episode-48-meloetta-and-the-unde
rsea-temple-
Production ID 1\/6988\/0235#001
RTMPDump 2.4pl2
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Connected...
Starting download at: 0.000 kB
INFO: Metadata:
INFO:   duration              1280.02
INFO:   moovPosition          32.00
INFO:   audiocodecid          mp4a
INFO:   width                 704.00
INFO:   height                396.00
INFO:   videocodecid          avc1
INFO:   avcprofile            77.00
INFO:   avclevel              31.00
INFO:   aacaot                2.00
INFO:   audiosamplerate       48000.00
INFO:   audiochannels         2.00
INFO:   videoframerate        25.00
INFO: trackinfo:
INFO:   length                61441024.00
INFO:   timescale             48000.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            mp4a
INFO:   length                31998000.00
INFO:   timescale             25000.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            avc1
1-6988-0235-001_pokemon--bw---rival-destinies-(series-15)_tx311212_itv1200_16x9-
reg.mp4: 1% (3522.97 / 195720.39 KiB)^C

Tested on openBSD amd64 current.

Original comment by njtaylor...@gmail.com on 3 Jan 2013 at 8:46

Changed state: Started

GoogleCodeExporter commented 8 years ago

This works for me (Ubuntu 12.04 LTS, Intel® Pentium(R) M processor 1.73GHz), 
as does  the version at your "not a template" address. That brings up a 
question: how do you figure out which version of a plugin you have installed?

Original comment by dwhelan...@gmail.com on 3 Jan 2013 at 10:49

GoogleCodeExporter commented 8 years ago

"how do you figure out which version of a plugin you have installed?" 

Normally the plugins are supplied as a complete set of source files, or using a 
package, so the version is assigned by the packager to the whole 
get_flash_videos installation, plugins don't have individual versions.

Giving plugins their own version is something I have been considering. Having a 
version in the plugins would help to spot failures because of out of date 
plugins.

Only the Hulu plugin gets updated, and this works using the date/time of the 
plugin not any version set within the plugin.

The method of placing the Itv.pm as a plugin into ~/.get_flash_videos/plugins 
is only for testing. Having a version in the plugin would help spot if a test 
plugin had not been removed.

Currently the only method is to compare file sizes/date/time of plugins, or 
look for specific changes.

Original comment by njtaylor...@gmail.com on 4 Jan 2013 at 3:01

GoogleCodeExporter commented 8 years ago

The updated version looks good to me:

$ get_flash_videos https://www.itv.com/itvplayer/dci-banks/series-1/episode-2
Using method 'itv' for 
https://www.itv.com/itvplayer/dci-banks/series-1/episode-2
Downloading https://www.itv.com/itvplayer/dci-banks/series-1/episode-2
...
Done. Saved 417092958 bytes to 
1-9089-0002-001_dci-banks--aftermath_tx211212_itv1200_16x9-reg.mp4

Original comment by insomnia...@googlemail.com on 5 Jan 2013 at 11:42

GoogleCodeExporter commented 8 years ago

That's funny, this:
get_flash_videos https://www.itv.com/itvplayer/dci-banks/series-1/episode-2
doesn't work for me but:
get_flash_videos http://www.itv.com/itvplayer/video/?Filter=177485
does. 
Same video but different link.

Original comment by dwhelan...@gmail.com on 5 Jan 2013 at 4:25

GoogleCodeExporter commented 8 years ago

Odd, I'm also running Ubuntu 12.04 and I get the following error when 
attempting the same programme. I used the plugin saved under  

https://github.com/njtaylor/get-flash-videos/raw/master/lib/FlashVideo/Site/Itv.
pm

My guess is it is opting for using method 'generic' rather than itv. Yet how to 
solve this.

My output is:

get_flash_videos 
https://www.itv.com/itvplayer/pokemon/series-12/episode-48-meloetta-and-the-unde
rsea-temple- --debug
1 plugin installed:
- Itv.pm
Downloading 
https://www.itv.com/itvplayer/pokemon/series-12/episode-48-meloetta-and-the-unde
rsea-temple-
-> GET 
https://www.itv.com/itvplayer/pokemon/series-12/episode-48-meloetta-and-the-unde
rsea-temple-
<- 200 text/html (129291)
Trying to open plugin ~/.get_flash_videos/plugins/Www.pm
Trying to open plugin ~/.get_flash_videos/plugins/Itv.pm
Trying to open plugin ~/.get_flash_videos/plugins/Com.pm
Trying to open plugin ~/.get_flash_videos/plugins/Twitter.pm
Trying to open plugin ~/.get_flash_videos/plugins/Com.pm
Trying to open plugin ~/.get_flash_videos/plugins/Brightcove.pm
Trying to open plugin ~/.get_flash_videos/plugins/Mtvnservices.pm
Trying to open plugin ~/.get_flash_videos/plugins/Gawker.pm
Using method 'generic' for 
https://www.itv.com/itvplayer/pokemon/series-12/episode-48-meloetta-and-the-unde
rsea-temple-
Found \/itvplayer\/
Guessed https://www.itv.com/itvplayer/pokemon/series-12/%5C/itvplayer%5C/ 
trying...
Exists on https://www.itv.com/itvplayer/pokemon/series-12/%5C/itvplayer%5C/: 301
Redirected to 
https://www.itv.com/itvplayer/pokemon/series-12/%25255C/itvplayer%25255C (404)
Found \/itvplayer\/jcarousel\/ajax\/views
Guessed 
https://www.itv.com/itvplayer/pokemon/series-12/%5C/itvplayer%5C/jcarousel%5C/aj
ax%5C/views trying...
Exists on 
https://www.itv.com/itvplayer/pokemon/series-12/%5C/itvplayer%5C/jcarousel%5C/aj
ax%5C/views: 404
Found lserver
Guessed https://www.itv.com/itvplayer/pokemon/series-12/lserver trying...
Exists on https://www.itv.com/itvplayer/pokemon/series-12/lserver: 404
Found node%2F33622
Guessed https://www.itv.com/itvplayer/pokemon/series-12/node%2F33622 trying...
Exists on https://www.itv.com/itvplayer/pokemon/series-12/node%2F33622: 404
Found lserver
Guessed https://www.itv.com/itvplayer/pokemon/series-12/lserver trying...
Exists on https://www.itv.com/itvplayer/pokemon/series-12/lserver: 404
Found {
Guessed https://www.itv.com/itvplayer/pokemon/series-12/%7B trying...
Exists on https://www.itv.com/itvplayer/pokemon/series-12/%7B: 404
Found itv_sitesection3
Guessed https://www.itv.com/itvplayer/pokemon/series-12/itv_sitesection3 
trying...
Exists on https://www.itv.com/itvplayer/pokemon/series-12/itv_sitesection3: 404
Error: No URLs found at /usr/share/perl5/FlashVideo/Generic.pm line 128.

Couldn't extract Flash movie URL. This site may need specific support adding,
or fixing.

Please confirm the site is using Flash video and if you have Flash available
check that the URL really works(!).

Check for updates by running: /usr/bin/get_flash_videos --update

If the latest version does not support this please open a bug (or
contribute a patch!) at http://code.google.com/p/get-flash-videos/
make sure you include the output with --debug enabled.
Couldn't download any videos.

Thanks in advance

Original comment by allen.sh...@googlemail.com on 5 Jan 2013 at 4:31

GoogleCodeExporter commented 8 years ago

Not to worry. Made a slight spelling error. Thanks for the plugin.

Original comment by allen.sh...@googlemail.com on 5 Jan 2013 at 4:34

GoogleCodeExporter commented 8 years ago

I downloaded the plugin again and reinstalled it and now both:
get_flash_videos https://www.itv.com/itvplayer/dci-banks/series-1/episode-2
and
get_flash_videos http://www.itv.com/itvplayer/video/?Filter=177485
work. 
I must have corrupted your plugin somehow. 
Thanks for the update.

Original comment by dwhelan...@gmail.com on 5 Jan 2013 at 5:16

GoogleCodeExporter commented 8 years ago

And how about Windows version? I can not get it working

W:\>c:\perl\perl\bin\perl.exe c:\perl\perl\bin\get_flash_videos -r high 
https://www.itv.com/itvplayer/horrid-henry/serie
s-3/episode-13-horrid-henry-grown-up --debug
~1~ ~p~l~u~g~i~n~ ~i~n~s~t~a~l~l~e~d~:~
~~-~ ~I~t~v~.~p~m~
~Subroutine FlashVideo::Utils::GetACP redefined at (eval 58) line 2.
Subroutine FlashVideo::Utils::GetACP redefined at (eval 60) line 2.
Trying to open plugin 
C:\Users\John\AppData\Roaming/get_flash_videos/plugins/Www.pm
Using method 'itv' for 
https://www.itv.com/itvplayer/horrid-henry/series-3/episode-13-horrid-henry-grow
n-up
Downloading 
https://www.itv.com/itvplayer/horrid-henry/series-3/episode-13-horrid-henry-grow
n-up
-> GET 
https://www.itv.com/itvplayer/horrid-henry/series-3/episode-13-horrid-henry-grow
n-up
<- 200 text/html (83717): UTF8 on, non-ASCII, 83717 characters 83718 bytes
Error: No id (filter) found in URL

Couldn't extract Flash movie URL. This site may need specific support adding,
or fixing.

Please confirm the site is using Flash video and if you have Flash available
check that the URL really works(!).

Check for updates by running: c:\perl\perl\bin\get_flash_videos --update

If the latest version does not support this please open a bug
at http://code.google.com/p/get-flash-videos/ making sure you include
the output with --debug enabled. Alternatively, fix it yourself and send us
a pull request on Github: https://github.com/monsieurvideo/get-flash-videos
Couldn't download any videos.

Original comment by scer...@gmail.com on 13 Jan 2013 at 9:07

GoogleCodeExporter commented 8 years ago

In fact it works, I had to remove itv section from:

https://github.com/downloads/njtaylor/get-flash-videos/get_flash_videos-1.24-git
-20121106

and the plugin was used instead & it downloads OK !!!

sebus

Original comment by scer...@gmail.com on 13 Jan 2013 at 9:29

GoogleCodeExporter commented 8 years ago

I will look at making a new download available, github no longer allows 
download files to be uploaded, going to have to find a new location.

Original comment by njtaylor...@gmail.com on 13 Jan 2013 at 10:29

GoogleCodeExporter commented 8 years ago

The changes have been request to be pulled from my github repository. This 
includes the fixes for this problem. There are download files available on this 
site, with the fix included.

Original comment by njtaylor...@gmail.com on 22 Feb 2013 at 2:50

Changed state: Fixed

GoogleCodeExporter commented 8 years ago

Does this still work for people? I have tried in both Windows with Strawberry 
Perl and now in Ubuntu. The result is the same. I think I am using the GIT 
version of get_flash_videos and also the GIT version of RTMPDUMP. Please see 
the results below when I turn debugging on. 

Any help greatly appreciated.

I get it when trying to download this file:

https://www.itv.com/itvplayer/foyle-s-war/series-7/episode-1-the-eternity-ring

Using method 'itv' plugin version 0.03 for 
https://www.itv.com/itvplayer/foyle-s-war/series-7/episode-1-the-eternity-ring
Downloading 
https://www.itv.com/itvplayer/foyle-s-war/series-7/episode-1-the-eternity-ring
RTMPDump v2.4
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Failed to open file! 
2-1515-0001-001_foyle-s-war-(series-8)_tx240313_itv1200_16x9-reg.flv
Download failed, no valid file downloaded
Couldn't download any videos.

Debugging showed this further information:

DEBUG: swfUrl   : http://www.itv.com/mercury/Mercury_VideoPlayer.swf
DEBUG: app      : 
ondemand?auth=eaEaac.bbaydTaabmdbb_ajcBd7dJcJdxdXcoaubyaQaHcBdJanc.cgbWdcb2awclc
5-bruBxu-eS-4zo2DArmArwq&aifp=v002&slist=production/
DEBUG: live     : no
DEBUG: timeout  : 30 sec
DEBUG: SWFSHA256:
DEBUG: b6 c8 96 6d a3 f4 96 10 be 71 78 b0 1c a3 3d 04
DEBUG: 6b bf 91 5e 29 08 d9 da fe 11 e4 b0 42 d8 ee ea
DEBUG: SWFSize  : 990750
Failed to open file! 
2-1515-0001-001_foyle-s-war-(series-8)_tx240313_itv1200_16x9-reg.flv
Download failed, no valid file downloaded
Couldn't download any videos.

Original comment by l...@farraway.co.uk on 26 Mar 2013 at 2:57

GoogleCodeExporter commented 8 years ago

Does this still work for people? I have tried in both Windows with Strawberry 
Perl and now in Ubuntu. The result is the same. I think I am using the GIT 
version of get_flash_videos and also the GIT version of RTMPDUMP. Please see 
the results below when I turn debugging on. 

Any help greatly appreciated.

I get it when trying to download this file:

https://www.itv.com/itvplayer/foyle-s-war/series-7/episode-1-the-eternity-ring

Using method 'itv' plugin version 0.03 for 
https://www.itv.com/itvplayer/foyle-s-war/series-7/episode-1-the-eternity-ring
Downloading 
https://www.itv.com/itvplayer/foyle-s-war/series-7/episode-1-the-eternity-ring
RTMPDump v2.4
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Failed to open file! 
2-1515-0001-001_foyle-s-war-(series-8)_tx240313_itv1200_16x9-reg.flv
Download failed, no valid file downloaded
Couldn't download any videos.

Debugging showed this further information:

DEBUG: swfUrl   : http://www.itv.com/mercury/Mercury_VideoPlayer.swf
DEBUG: app      : 
ondemand?auth=eaEaac.bbaydTaabmdbb_ajcBd7dJcJdxdXcoaubyaQaHcBdJanc.cgbWdcb2awclc
5-bruBxu-eS-4zo2DArmArwq&aifp=v002&slist=production/
DEBUG: live     : no
DEBUG: timeout  : 30 sec
DEBUG: SWFSHA256:
DEBUG: b6 c8 96 6d a3 f4 96 10 be 71 78 b0 1c a3 3d 04
DEBUG: 6b bf 91 5e 29 08 d9 da fe 11 e4 b0 42 d8 ee ea
DEBUG: SWFSize  : 990750
Failed to open file! 
2-1515-0001-001_foyle-s-war-(series-8)_tx240313_itv1200_16x9-reg.flv
Download failed, no valid file downloaded
Couldn't download any videos.

Original comment by l...@farraway.co.uk on 27 Mar 2013 at 4:54

rt2yrru / get-flash-videos

ITVPlayer site changes break itv.pm module #459