ustream returns 302 error, Downloader::download does not handle redirect

[GoogleCodeExporter]

What steps will reproduce the problem?
1. get_flash_videos -d http://www.ustream.tv/recorded/28594019

What is the expected output? What do you see instead?

expected: File NASA_Replay__Dark_energy_in_the_universe.flv has been fully 
downloaded.

actual

No plugins installed
Downloading http://www.ustream.tv/recorded/28594019
-> GET http://www.ustream.tv/recorded/28594019
<- 200 text/html; charset=UTF-8 (38379): UTF8 on, non-ASCII, 38379 characters 
38425 bytes
Trying to open plugin ~/.get_flash_videos/plugins/Www.pm
Using method 'ustream' for http://www.ustream.tv/recorded/28594019
Downloading http://tcdn.ustream.tv/video/28594019...
-> GET http://tcdn.ustream.tv/video/28594019
<- 302 text/html (0): UTF8 off, ASCII, 0 characters 0 bytes
Couldn't download http://tcdn.ustream.tv/video/28594019: 302 Found
Couldn't download any videos.

What version of the product are you using? On what operating system?

Latest git (1.25?)

Please provide any additional information below.

Using the perl debugger, the Ustream.pm returns 
http://tcdn.ustream.tv/video/28594019 to download 
(FlashVideo::Downloader::download), but download does not handle the 302 
re-direct. If I use wget on http://tcdn.ustream.tv/video/28594019, it does 
handle the re-direct and download the video.

Examining the download get with the perl debugger I can see the redirect 
response below in the rc => 302 response. I can try to fix it, but what should 
Ustream.pm do to handle this redirect?

Should not FlashVideo::Downloader::download handle the redirect (ala wget)?

  DB<32> n
FlashVideo::Mechanize::get(Download/get-flash-videos/get_flash_videos-1.25:3874)
:
3874:     if($App::get_flash_videos::opt{debug}) {
  DB<32> p $r
HTTP::Response=HASH(0xa64c368)
  DB<33> x $r
0  HTTP::Response=HASH(0xa64c368)
   '_content' => ''
   '_headers' => HTTP::Headers=HASH(0xa64c488)
      'client-date' => 'Sat, 02 Mar 2013 20:00:33 GMT'
      'client-peer' => '199.66.238.50:80'
      'client-response-num' => 1
      'connection' => 'close'
      'content-length' => 0
      'content-type' => 'text/html'
      'date' => 'Sat, 02 Mar 2013 20:00:32 GMT'
      'keep-alive' => 'timeout=15, max=198'
      'location' => 'http://upmv07.xo.upmv.ustream.tv/0/1/28/28594/28594019/1_5808990_28594019.flv?tracking=c4bb05_43_1_1_0&src=ro'
      'server' => 'Apache'
      'x-rsrc' => 'sjc-web006'
   '_msg' => 'Found'
   '_protocol' => 'HTTP/1.1'
   '_rc' => 302
   '_request' => HTTP::Request=HASH(0xa5d46a0)
      '_content' => ''
      '_headers' => HTTP::Headers=HASH(0xa5d4690)
         'accept-encoding' => 'gzip'
         'referer' => 'http://rgw.ustream.tv/gateway.php'
         'user-agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6'
      '_method' => 'GET'
      '_uri' => URI::http=SCALAR(0xa635510)
         -> 'http://tcdn.ustream.tv/video/28594019'
      '_uri_canonical' => URI::http=SCALAR(0xa635510)
         -> REUSED_ADDRESS

Original issue reported on code.google.com by pcwal...@comcast.net on 2 Mar 2013 at 9:54

[GoogleCodeExporter]

I can reproduce the problem. Before saying the 302 response should be handled. 
This is the url extracted by Ustream.pm, it maybe the site has been updated and 
using an old method for extracting the url, resulting in the needed to redicted 
to a new url. This needs some more investigation of the problem.

Original comment by njtaylor...@gmail.com on 3 Mar 2013 at 2:16

• Changed state: Accepted

[GoogleCodeExporter]

I did hack redirect handling logic into FlashVideo::Downloader::download and it 
appears to work so far. I tried non-ustream sites and they appear not to be 
affected.

I have attached the hacked Downloader.pm as a reference, in case any project 
member is interested in leveraging the logic. I am not proud of the logic. It 
is basically a test for a 302 redirect return from the get, if the get returns 
a redirect, the redirect location is assigned into $url and a goto is executed 
to retry the get.

Original comment by pcwal...@comcast.net on 3 Mar 2013 at 3:21

Attachments:

• Downloader.pm

[GoogleCodeExporter]

I had a look at this, and found allow-redirects. You might like to try this 
slightly simpler change. Worked for me.

--- /usr/local/libdata/perl5/site_perl/FlashVideo/Site/Ustream.pm       Thu Feb 
21 23:02:57 2013
+++ Ustream.pm  Sun Mar  3 18:48:24 2013
@@ -5,6 +5,9 @@
 use FlashVideo::Utils;
 use MIME::Base64;

+our $VERSION = '0.01';
+sub Version() { $VERSION };
+
 sub find_video {
   my ($self, $browser, $embed_url) = @_;

@@ -41,6 +44,8 @@

   # Data::AMF fails to understand this response, so just parse ourselves.
   my($flv_url) = $browser->content =~ /flv.{3,5}(http:[^\0]+)/;
+
+  $browser->allow_redirects;

   return $flv_url, title_to_filename($title);
 }

Place the attached Ustream.pm file in ~/.get_flash_videos/plugins (Unix/linux 
os) or replace Ustream.pm to try.

Original comment by njtaylor...@gmail.com on 4 Mar 2013 at 3:29

Attachments:

• Ustream.pm

[GoogleCodeExporter]

Thanks njtaylor, yes adding the $browser->allow_redirects to Ustream.pm works.

I was under the impression that redirects was ON, already and was not being 
handled for some other reason.

If you want to submit the Ustream.pm patch to the main trunk, I'd appreciate it.

Original comment by pcwal...@comcast.net on 4 Mar 2013 at 5:20

[GoogleCodeExporter]

I will add this to my repository, will go in with my next pull request. I will 
close this issue.

Original comment by njtaylor...@gmail.com on 4 Mar 2013 at 6:13

• Changed state: Fixed

[GoogleCodeExporter]

This is now in the master repository, downloads files will be created shortly.

Original comment by njtaylor...@gmail.com on 5 Mar 2013 at 1:00