Downloads repeatedly fail after most of show downloaded

[willson556]

From m...@rowatt.com on January 02, 2013 09:30:50

What steps will reproduce the problem? 1. add multiple shows to queue

2. download
3. some shows download (completely or very close to completely), but then instead of processing and saving the download, the downloaded file is deleted and the download starts again. This repeats multiple times. Occasionally the show does successfully download on a subsequent attempt, but mostly it just fails.
4. after all downloads have failed, restarting the download process usually results in successful download of previously failed shows. What is the expected output? What do you see instead? Downloads should not normally fail, and when they do, they shouldn't normally fail after downloading most or all of the show. What version of the product are you using? On what operating system? I have experienced this on 1.5 and the latest prerelease build (1.5.1-pre8). Please provide any additional information below. * log from a few downloads attached, the first few failed, that last one did not
   • I am running MacOSX 10.8.2
   • I first started noticing more than the occasional failure about 2-3 weeks ago
   • I'm not aware of any significant changes in my Mac setup in that timeframe
   • I've only noticed this on BBC iPlayer shows (though I rarely download anything else)

Attachment: download_log.txt

Original issue: http://code.google.com/p/get-iplayer-automator/issues/detail?id=152

[willson556]

From dinkypumpkin on January 02, 2013 06:12:56

rtmpdump (utility used for RTMP streaming) can run afoul of any number of problems: overloaded system, flakey router, upstream network congestion, misbehaving media servers, etc. Bottom line: there is nothing GiA can do to help. FWIW, your log indicates problems at the BBC end, something which happens fairly often. The iPlayer site is occasionally afflicted by this as well. As you pointed out yourself, you just have to keep trying.

As for partial downloads, there is no guarantee that once a download starts it will finish. Even if rtmpdump reconnects and attempts to resume a download, the partially-downloaded file is often not in a state to support resuming, so GiA can only delete it and start over.

Status: Invalid

[willson556]

From m...@rowatt.com on January 02, 2013 06:36:18

Thanks very much for the info.

FWIW, in most cases I think that the RTMP resume is working OK - it's not deleting and restarting from scratch - it continues adding to the already received file.

Maybe is more of a feature request than a bug report but would it be possible for GiA to handle the errors differently? As it stands, it can be downloading near enough 100% of a show maybe 6 times before finally giving up on a show which is eating bandwidth and delaying downloads of other files.

Would it be possible, for example, to have a setting (perhaps hidden) which says something to the effect of:-

don't automatically retry failed downloads if more than X% of a file has downloaded

You already have the 'retry failed downloads' setting in the main prefs (which I have set to no), but in this case downloading all/nearly all of a show in, say, flashvhigh, failing for whatever reason, and then trying again in flashhigh etc seems to be unnecessary overkill.

I'd rather it just failed after one nearly complete attempt and then let me decide (manually or with the retry main pref) if/when to retry.

[willson556]

From dinkypumpkin on January 02, 2013 08:35:32

BBC downloads work somewhat differently since they are handled by a separate application (get_iplayer). As you can see from your log, get_iplayer automatically falls back to lower quality streams if they are requested. The thing for you to do (at least temporarily) is to remove all undesired BBC formats in GiA Preferences. Perhaps for the moment limit yourself to a single format (Flash - Very High) for testing. For BBC TV shows, you may still see 2 attempts per show for each format since each show is usually available from 2 BBC CDNs. get_iplayer will fall back to the second CDN if downloading from the first fails.

[willson556]

From m...@rowatt.com on January 02, 2013 08:46:37

I'll remove some of the quality options from prefs, so that should help, thanks. Presumably, though, if I request say flashvhigh only, and the stream doesn't exist in that format, then the download will just fail, and always fail until I change the settings to something else?

I saw in the log that it was trying each quality twice, but didn't appreciate it was trying different CDNs. It would be good if get_iplayer didn't do that for failures after most of the show has downloaded. I see the original developer is no longer maintaining get_iplayer... are you using a version maintained by someone else? If so, I could try posting a request there if that would be any help.

[willson556]

From dinkypumpkin on January 02, 2013 12:07:05

Re: using flashvhigh only - that's true, but most BBC TV programmes should be available in a flashvhigh version.

Re: get_iplayer and CDNs. There is a way to limit get_iplayer to use a single CDN. The problem is that you don't know precisely which CDN will be used since the values returned in the metadata for a BBC programme vary with time for load balancing purposes, so not terribly useful. The most common problem is that one CDN is rejecting connections, so you almost always want get_iplayer to roll over to the other CDN. You would also want this to happen even if the file was mostly downloaded from the first CDN since there is always a chance the download might resume successfully. If it can't resume from the second CDN, rtmdump will croak and you won't waste that much extra bandwidth. AFAICT, your issue is simply that this process is repeated for each format, which you now know how to remedy.

get_iplayer is still alive ( http://www.infradead.org/get_iplayer ). I'm the primary maintainer these days, and I'm not very likely to implement what you ask for since I don't know a reliable way to do it. However, get_iplayer is just a giant Perl script, so feel free to have a go. Get it from the main repo ( http://git.infradead.org/get_iplayer.git ) or fork it on GitHub ( https://github.com/dinkypumpkin/get_iplayer ). If you just want to try get_iplayer separate from GiA, do yourself a favour and create a shell script to launch it (assumes GiA in /Applications):

!/bin/bash

GIA app bundle directory

APP_DIR="/Applications/Get iPlayer Automator.app"

location of GIA binaries

BIN_DIR="$APP_DIR/Contents/Resources"

run get_iplayer with GIA binaries

DYLD_LIBRARY_PATH needed to locate librtmp

DYLD_LIBRARY_PATH="$BIN_DIR" \ /usr/bin/perl "$BIN_DIR/get_iplayer.pl" \ --atomicparsley "$BIN_DIR/AtomicParsley" \ --ffmpeg "$BIN_DIR/ffmpeg" \ --flvstreamer "$BIN_DIR/rtmpdump-2.4" \ --lame "$BIN_DIR/lame" \ --mplayer "$BIN_DIR/mplayer" \ "$@"

If you want to go all in, you can install the whole get_iplayer ecosystem with Homebrew as well: https://github.com/dinkypumpkin/homebrew-get_iplayer/wiki

[willson556]

From m...@rowatt.com on January 02, 2013 14:58:53

Thank you.

Re: CDNs. I could be wrong, but I think I'm seeing the download nearly complete on the first attempt, fail, and then go to the second CDN starting from the beginning, though it's possible that every time I saw a failure 'in progress' it was rolling over to a different quality. Either way, even if it downloads twice, that's a big improvement on before.

My perl isn't up to much, so I doubt I'll be hacking the core script, though I might give some simple bash scripts a try - so thanks for that info.

Thank you for your help, and thank you for all the effort you put into GiA and get_iplayer.

[willson556]

From m...@rowatt.com on January 02, 2013 15:23:09

In case it adds any useful info to the picture, I just all but caught a failure in the act. Show had downloaded to at least 95% last time I checked, on 'flashvhigh2' according to log. It's now downloading from 'flashvhigh1' from the beginning. So if the CDN failover is meant to just pick up from where it left off, it's not doing that in this case.

Partial log attached (the second download is still in progress).

Attachment: log.txt

[willson556]

From dinkypumpkin on January 02, 2013 17:02:24

Without Verbose set in Preferences, there isn't enough information in the log to even guess at what is going on. AFAIK, get_iplayer won't delete a partial download > 100KiB, and I don't think GiA interferes with that. If you want to replicate your test with the bash script I included above (saved as, e.g., get_iplayer.sh), the command would be:

get_iplayer.sh --verbose --modes=flashvhigh --force --get "Dances With Wolves" > log.txt 2>&1

Be sure to delete the partial download before each attempt.

[willson556]

From m...@rowatt.com on January 03, 2013 12:23:39

Thanks - I've been doing a few downloads today using the script with verbose logging, and so far have failed to replicate the problem. Will keep trying and post back results when I have something definitive.

[willson556]

From m...@rowatt.com on January 04, 2013 00:15:26

Now that I have verbose logging, the incidence of the issue is much less than it was :-~

However, I did see one in the log which has the same error:- WARNING: Failed to stream file XXX

I don't know if get_iplayer actually downloaded most of the file before failing though.

It did happen immediately after this, though:- DEBUG: Invoking deleteStream rtmpdump-2.4(90408) malloc: * error for object 0x1f3983000: pointer being freed was not allocated * set a breakpoint in malloc_error_break to debug WARNING: Command executed but died with signal 6 INFO: Command exit code 2 (raw code = 6)

Maybe that's indicative of the problem?

Full log of the failed download attached.

Attachment: debug_log.txt

[willson556]

From dinkypumpkin on January 04, 2013 07:14:29

Well, if (as in this case) rtmpdump crashes, I would say that is a problem. You log shows that rtmpdump apparently received some garbage data, so it's no surprise it crashed. How fast is your internet connection?

BTW, I left some things out of the get_iplayer command line. The example command above should have read:

get_iplayer.sh --verbose --attempts=5 --force --modes=flashvhigh --get "Dances With Wolves" > log.txt 2>&1

The --attempts option (default=50) limits the number of reconnection attempts if rtmpdump times out. Also add the --overwrite option for command line testing if you want to repeat tests without first deleting the output files from previous tests.

[willson556]

From m...@rowatt.com on January 04, 2013 15:03:53

I only have an 'up to 8mbps' connection, and typical download speeds are about 4mpbs as measured by speedtest.net. The connection is pretty stable though and I haven't noticed it being any different from normal.

Is it possible for debug purposes to hack get_iplayer (or GiA) so that options are tuned for debugging?

I'll try some more downloads from the commandline with attempts=5 and see what happens.

[willson556]

From dinkypumpkin on January 04, 2013 15:12:18

I wondered about the speed because someone recently reported similar symptoms with rtmpdump on a very fast (30Mb/s+) connection. Yours is only 2x-3x realtime, so shouldn't be a problem.

There isn't really such a thing as tuning get_iplayer options for debug. You basically just have the choice of turning on the --verbose option or not. There is a --debug option, but that invokes the rtmpdump --debug option, which produces a punishingly large spew of output, too much to be useful.

[willson556]

From m...@rowatt.com on January 04, 2013 15:50:04

Ah, so being on a relatively slow connection is an advantage for once ;-)

Re tuning for debug, I wasn't very clear... I just meant can I hack the code or otherwise hard wire verbose logging (or other helpful debug options) into GiA so that as/when errors occur in normal use I'll have all the log data available, rather than having to run get_iplayer separately with specific debug options.

[willson556]

From dinkypumpkin on January 04, 2013 15:54:37

Check Verbose in GiA preferences, which passes the --verbose option to get_iplayer. You shouldn't need it, but if you want more than that for BBC downloads, you'll have to hack it into get_iplayer. The get_iplayer script used by GiA is located here:

/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer.pl

Adjust as need for a different installation directory.

[willson556]

From m...@rowatt.com on January 07, 2013 00:42:36

Ah... I'd missed the verbose option in GiA. I'll run things with that on and post back in a few days once I have some more data.

[willson556]

From m...@rowatt.com on January 15, 2013 03:24:44

I've done some more testing. I can send more logs if helpful for other failed downloads, but see attached for one specific failure. In addtion to whatever is in the log, I observed:-

• same behaviour as before, namely file appears to download 100%, something fails, then whole download restarts
• the raw flv video file was left and, although GiA reported that the download failed, the flv file is complete and plays fine... it just needs converting to non-flash mp4, and the ".partial." removed from the filename

Hope that gives some clues as to what is going on.

Attachment: failed_log.txt.zip

[willson556]

From dinkypumpkin on January 15, 2013 06:06:28

According to the log, the file is downloading fine, so no surprise the FLV works. The problem appears to come when get_iplayer checks the downloaded file after rtmpdump finishes. For whatever reason, it thinks the file is either missing or < 100 KiB in size, so it rolls over to a new source and starts the download again. Are you downloading to a NAS or some other kind of network storage? If so, try downloading to your internal disk and see if you get a different result.

[willson556]

From m...@rowatt.com on January 16, 2013 01:05:26

Thanks. I'm not downloading to a NAS, but I do have a script which moves the files after download. It shouldn't be doing anything until the files have been unmodified for at least 1 hour, but it sounds like that may be the problem. I'll investigate further... if it is anything other than 'script stupidity' (or at least a bug somewhere else), I'll let you know, but if the file is downloading fully and GiA thinks it's missing, then it seems highly likely that is what's going on.

Thank you for all your help.