New RTMPDUMP: Resume issues

[willson556]

From thomas.h...@gmail.com on January 17, 2014 08:28:32

What steps will reproduce the problem? 1. Start to download a new show. It starts downloading a few MB normally.

2. Go drink a cup of coffee and come back.
3. The download is now stuck initialising, regardless of how many more coffees you drink patiently for the download to actually resume downloading.

(It might work with tea as well) What is the expected output? What do you see instead? Initialising is expected to conclude and downloading to resume. What version of the product are you using? On what operating system? Version 1.6.1 (645) on OS X Version 10.9.1 Please provide any additional information below. I am using a reliable VPN from China. The internet connection is good but may sometimes suffer hiccups. This problem is recent (a week or two) after a period of bliss downloading shows without problems. It affects almost all attempts at downloading shows. It might be related to my starting using version >= 1.6.0.

I reported the problem using the GiA bug reporting facility with a log attached but I am not sure you get these.

Thanks.

Original issue: http://code.google.com/p/get-iplayer-automator/issues/detail?id=336

[willson556]

From dinkypumpkin on January 17, 2014 02:58:52

Spend less time drinking coffee and more time creating a log of your failed download. You haven't provided any useful information otherwise. See wiki: VerboseLog

[willson556]

From tobyr...@gmail.com on January 17, 2014 04:12:52

Hi Dinkypumpkin,

I think I am seeing the same issue. I have attached a verbose log from my 10.7.5 machine (father brown failed.debug.rtf) .

I see the download go to initialising, and never move on. The network stats show I am downloading at my maximum DSL rate. What I have also noticed is that after a few hours I run out of memory, the hard disk fills up with swap and the machine crashes. I can't pin the memory loss to getiplayer automator definitively , as when the machine gets to no free memory it is too unstable to debug, but I did see that RTMPDUMP was using 1.5GB memory before I hit the power switch.

I just tried this on my 10.8.1 machine (same download) - no problem.

I went back to the 10.7 machine an noticed it had a partial download for that program. Deleted the part ion and retried the download, and it started fine (log attached father brown ok.debug.rtf. Could this be related to the restarting partial download improvements made?

Thomas - check for a partial download, delete it and try again.

Attachment: Father Brown failed.debug.rtf father brown ok.debug.rtf

[willson556]

From dinkypumpkin on January 17, 2014 04:23:04

1: In case it wasn't clear, any submissions via bug reporting facility go GiA's author. I have no idea what he does with them, but they don't appear here, so please submit a log as instructed.

[willson556]

From dinkypumpkin on January 17, 2014 04:41:38

2: Thanks for the logs. As I mentioned in issue #335 , runaway rtmpdump is a known problem on all platforms, not just OSX and not just with GiA. I don't think switching versions of OSX is a general solution. Reports in issue #335 indicate that rtmpdump problems can occur on 10.8 as well.

I suspect you are correct that the "improvements" in resume functionality may not be improvements after all. I don't yet have any way to confirm that, but I don't recent reports of rtmpdump problems are purely a coincidence. For the moment, roll back to 1.5.9 (w/old rtmpdump) and see if things improve. I'll consider making a fresh GiA build with the old version of rtmpdump, but as yet I don't have enough evidence that it would help. The new rtmpdump is obviously working fine for the vast majority of users.

[willson556]

From tobyr...@gmail.com on January 17, 2014 06:59:42

I just tried this again on my 10.8 machine. I started a download (the same father brown episode) and waited for it to initialise and the download to commence. I then stopped the download, exited GIA, restarted GIA and tried restarting the download - it fails. Log attached. I am pretty sure the "improvements" to the restart code are causing issues.

Attachment: Father brown 10.8.failed.txt

[willson556]

From dinkypumpkin on January 17, 2014 07:20:52

5: Resuming downloads can fail at the best of times. To make a complete test, you need to go on to then run 1.5.9 (i.e., old rtmpdump) to see if it can resume the download. If enough people find that to be case, it militates for rolling back to the old rtmpdump. Frequent failures during resuming would be preferable to rtmpdump killing your machine.

Your latest log shows that the first resume attempt was unable to initialise, and the second (prior to the one that failed) was rejected with a "StreamNotFound" error. Both are suggestive of upstream server problems. I also see you're using some kind of local proxy with your 10.8 machine. That shouldn't matter, but it makes me wonder if you're testing the two systems under the same conditions.

[willson556]

From tobyr...@gmail.com on January 17, 2014 07:55:48

Fair points :-)

Trying to resume that download using 1.5.9 on my 10.8 machine (via local proxy) gives a helpful error message about an unresumeable download, and stops. (Father brown 1.5.9 10.8.failed-resume.txt)

I then deleted the partial again, and restarted again - successful clean start. (Father brown 1.5.9 10.8.clean-start-success.txt)

I then stopped, restarted GIA and restarted the download. Same helpful failure message as before about an unrestartable download. (Father brown 1.5.9 10.8.failed-resume fom 1.5.9.txt)

I will re-test all with 10.7.5 and update here.

Attachment: Father brown 1.5.9 10.8.clean-start-success.txt Father brown 1.5.9 10.8.failed-resume.txt Father brown 1.5.9 10.8.failed-resume fom 1.5.9.txt

[willson556]

From tobyr...@gmail.com on January 17, 2014 08:43:16

Adding 10.7 test results - no proxy.

Clean 1.5.9 install with no resumable file. The download starts with no problem. (Father brown 10.7.5 1.5.9 clean start.txt)

Stop the download restart GIA and restart the same download. Download fails and GIA gives a friendly message. (Father brown 10.7.5 1.5.9 failer resume.txt)

Attachment: Father brown 10.7.5 1.5.9 clean start.txt Father brown 10.7.5 1.5.9 failer resume.txt

[willson556]

From thomas.h...@gmail.com on January 17, 2014 15:45:11

To confirm that GiA 1.5.9 works better for me too.

Last attempt at downloading a new show with GiA 1.6.1 : started correctly but a few hours later, the computer had been restarted due to a problem, most probably GiA eating up ressources (no log available).

Attached:

• log of failed attempt resuming the same show with GiA 1.6.1 (stuck initialising).
• log of successful resuming the same show with GiA 1.5.9.

Attachment: Sherlock resuming failure with GiA 1.6.1.rtf Sherlock resuming success with GiA 1.5.9.rtf

[willson556]

From dinkypumpkin on January 17, 2014 16:37:53

8: Thanks for the additional logs. They make sense given the scenario you tested. From the scant evidence posted here, it seems that for a few people the new rtmpdump either tries to resume downloads when it shouldn't, or else fails to resume and goes haywire.

9: Thanks for the logs. They look consistent with other reports.

There are basically two types of resume errors appearing. One is when an attempt to resume can't initialise, indicate by messages like "Failed to get last keyframe" or "Command exit code 1 (raw code = 256)". The other appears to be when rtmpdump determines it can resume, but in actuality it can't and it starts chewing up resources.

The problem is compounded because it appears that GiA isn't showing all the error messages generated by rtmpdump. GiA screens messages from get_iplayer, but it looks like that could do with some adjustments. I think rtmpdump may be spewing a bunch of additional error messages while it's chewing up memory, but they're not being handled by GiA.

If you want to look into this - and you're bored - I have attached a bash script that you can use to run get_iplayer outside GiA (though it will use your GiA output directory and XBMC naming preferences). If you don't know about bash scripts or working at the command prompt, ignore this.

1. Install GiA 1.6.1 (we already know what 1.5.9 is doing)
2. The script assumes that GiA 1.6.1 is installed in /Applications. If not, edit the value of APP_DIR near the top of the script. Be sure to save it as plain text file.
3. Make script executable:

chmod +x ./get_iplayer-gia.sh

1. Run this command to download episode 10 of Father Brown (should be available for another week or so), HD format only, with verbose logging

./get_iplayer-gia.sh --verbose --modes=flashhd2,flashhd1 --pid=b03q5vlj

1. Kill the download part-way through with Ctrl-C, then re-run the command
2. Look for a spew of errors, things like "Couldn't find the seeked keyframe in this chunk". Kill the download with Ctrl-C before it goes haywire.

I'm not interested in the whole log, just if you see a bunch of errors after:

WARNING: Stream does not start with requested frame, ignoring data...

As an alternative to the shell script you can also create a separate installation of get_iplayer as described here: http://squarepenguin.co.uk/guides/mac-os-x-quick-install-guide/ That setup uses yet another build of rtmpdump, but it would be useful to know if it handles resuming without going haywire. In this case, the test command becomes:

get_iplayer --verbose --modes=flashhd2,flashhd1 --pid=b03q5vlj

Attachment: get_iplayer-gia.sh

[willson556]

From thomas.h...@gmail.com on January 18, 2014 14:25:11

10: Thanks for the detailed explanation. I have just tried the script you provided. Indeed, as you expected, I killed it at that point after trying to resume downloading:

... WARNING: Stream does not start with requested frame, ignoring data... WARNING: Stream does not start with requested frame, ignoring data... DEBUG: Checked keyframe successfully! DEBUG: ignoring too small video packet: size: 2 ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! etc.

[willson556]

From thomas.h...@gmail.com on January 18, 2014 15:29:56

10 (& follow up to #11): I have also tried the alternative to the shell script you mentioned. Attached is the log of the entire session trying the same command twice: the first fresh download succeeds but the second fails (it makes 50 attempts at resuming and quits). The error messages seem more verbose and different from #11, but I cannot really tell.

Attachment: get_iplayer_Father_Brown_10_failure_resuming_terminal_session.rtf

[willson556]

From dinkypumpkin on January 18, 2014 17:33:12

11: Thanks. I can only generate these errors artificially, so I wanted confirmation the same thing was happening in field conditions.

12: Sorry - I forgot about the default number of attempts being 50 in get_iplayer. If you should try it again, use --attempts=5 in the command line. That's what GiA does. As to the log, everything looks as I would expect, but again, it's good to have confirmation.

Clearly, the new rtmpdump doesn't work any better than the old version in terms of resuming downloads. The results are arguably worse - a runaway rtmpdump instead of a "corrupt file" message and quit. I can amend GiA to handle the additional error messages, but it seems like a bad idea to have a flawed rtmpdump deployed. I'll make a new GiA build this week, probably with the latest "standard" version of rtmpdump. It probably won't do any better at resuming downloads, but at least it should quit cleanly and not screw up your machine. Until then, keep using 1.5.9.

[willson556]

From thomas.w...@me.com on January 24, 2014 16:47:52

Yeah, I'm seeing the same thing on my end after having used it for a while. I think reverting back to the latest standard version of RTMPDUMP is a good call. The only thing it seems to resume better is when a download cancelled and then resumed. Other than that, there seem to be a lot of issues. At some point when I get chance I may try and figure out where KSV/Anonymous' patches are falling down. In the meantime, the old "reliable" version is the best bet. I'll make that change this weekend.

As for bug reports, I originally was able to respond to almost all of them but as usage has increased, I can no longer keep up. I get about 25-50/day and as a college student with an already large workload, it's tough to keep up. So at this point, I tend to use them to discern major trends rather than as a means of communicating with individual users.

Summary: New RTMPDUMP: Resume issues (was: Most downloads get stuck initialising for ever.)
Status: Accepted

[willson556]

From thomas.w...@me.com on January 24, 2014 16:49:28

Labels: -Priority-Medium Priority-Critical Milestone-Next

[willson556]

From dinkypumpkin on January 24, 2014 17:56:17

Thomas - New rtmpdump build has been pushed. I was sitting on it until the weekend, but if - as it sounds - you are going to cut another release, rtmpdump is ready to go.

As to why the KSV patch doesn't work, it seems to be this: The patched code expects the keyframe where resuming should begin to be duplicated in the first packet of flash video data after the keyframe. But at least for iPlayer streams, the calculation it uses to find the duplicate keyframe doesn't work because the timestamp of the first keyframe is zero where a non-zero value is expected to be used in calculating an offset to identify the duplicate keyframe. The result is that rtmpdump will seek to the end of the stream looking for the duplicate keyframe, but never finds it and thus never resumes downloading. In theory, this approach would make for better download resuming if the arithmetic could be made to work for iPlayer streams. For now the only improvement in the KSV version is that it does better at initiating live streams, which is irrelevant to GiA.

[willson556]

From dinkypumpkin on January 27, 2014 11:18:56

Standard version of rtmpdump restored in 1.6.2-pre.1. See wiki: PreReleaseBuilds

Status: Fixed