Closed willson556 closed 10 years ago
From dinkypumpkin on January 17, 2014 02:58:52
Spend less time drinking coffee and more time creating a log of your failed download. You haven't provided any useful information otherwise. See wiki: VerboseLog
From tobyr...@gmail.com on January 17, 2014 04:12:52
Hi Dinkypumpkin,
I think I am seeing the same issue. I have attached a verbose log from my 10.7.5 machine (father brown failed.debug.rtf) .
I see the download go to initialising, and never move on. The network stats show I am downloading at my maximum DSL rate. What I have also noticed is that after a few hours I run out of memory, the hard disk fills up with swap and the machine crashes. I can't pin the memory loss to getiplayer automator definitively , as when the machine gets to no free memory it is too unstable to debug, but I did see that RTMPDUMP was using 1.5GB memory before I hit the power switch.
I just tried this on my 10.8.1 machine (same download) - no problem.
I went back to the 10.7 machine an noticed it had a partial download for that program. Deleted the part ion and retried the download, and it started fine (log attached father brown ok.debug.rtf. Could this be related to the restarting partial download improvements made?
Thomas - check for a partial download, delete it and try again.
Attachment: Father Brown failed.debug.rtf father brown ok.debug.rtf
From dinkypumpkin on January 17, 2014 04:23:04
From dinkypumpkin on January 17, 2014 04:41:38
I suspect you are correct that the "improvements" in resume functionality may not be improvements after all. I don't yet have any way to confirm that, but I don't recent reports of rtmpdump problems are purely a coincidence. For the moment, roll back to 1.5.9 (w/old rtmpdump) and see if things improve. I'll consider making a fresh GiA build with the old version of rtmpdump, but as yet I don't have enough evidence that it would help. The new rtmpdump is obviously working fine for the vast majority of users.
From tobyr...@gmail.com on January 17, 2014 06:59:42
I just tried this again on my 10.8 machine. I started a download (the same father brown episode) and waited for it to initialise and the download to commence. I then stopped the download, exited GIA, restarted GIA and tried restarting the download - it fails. Log attached. I am pretty sure the "improvements" to the restart code are causing issues.
Attachment: Father brown 10.8.failed.txt
From dinkypumpkin on January 17, 2014 07:20:52
Your latest log shows that the first resume attempt was unable to initialise, and the second (prior to the one that failed) was rejected with a "StreamNotFound" error. Both are suggestive of upstream server problems. I also see you're using some kind of local proxy with your 10.8 machine. That shouldn't matter, but it makes me wonder if you're testing the two systems under the same conditions.
From tobyr...@gmail.com on January 17, 2014 07:55:48
Fair points :-)
Trying to resume that download using 1.5.9 on my 10.8 machine (via local proxy) gives a helpful error message about an unresumeable download, and stops. (Father brown 1.5.9 10.8.failed-resume.txt)
I then deleted the partial again, and restarted again - successful clean start. (Father brown 1.5.9 10.8.clean-start-success.txt)
I then stopped, restarted GIA and restarted the download. Same helpful failure message as before about an unrestartable download. (Father brown 1.5.9 10.8.failed-resume fom 1.5.9.txt)
I will re-test all with 10.7.5 and update here.
From tobyr...@gmail.com on January 17, 2014 08:43:16
Adding 10.7 test results - no proxy.
Clean 1.5.9 install with no resumable file. The download starts with no problem. (Father brown 10.7.5 1.5.9 clean start.txt)
Stop the download restart GIA and restart the same download. Download fails and GIA gives a friendly message. (Father brown 10.7.5 1.5.9 failer resume.txt)
Attachment: Father brown 10.7.5 1.5.9 clean start.txt Father brown 10.7.5 1.5.9 failer resume.txt
From thomas.h...@gmail.com on January 17, 2014 15:45:11
To confirm that GiA 1.5.9 works better for me too.
Last attempt at downloading a new show with GiA 1.6.1 : started correctly but a few hours later, the computer had been restarted due to a problem, most probably GiA eating up ressources (no log available).
Attached:
Attachment: Sherlock resuming failure with GiA 1.6.1.rtf Sherlock resuming success with GiA 1.5.9.rtf
From dinkypumpkin on January 17, 2014 16:37:53
There are basically two types of resume errors appearing. One is when an attempt to resume can't initialise, indicate by messages like "Failed to get last keyframe" or "Command exit code 1 (raw code = 256)". The other appears to be when rtmpdump determines it can resume, but in actuality it can't and it starts chewing up resources.
The problem is compounded because it appears that GiA isn't showing all the error messages generated by rtmpdump. GiA screens messages from get_iplayer, but it looks like that could do with some adjustments. I think rtmpdump may be spewing a bunch of additional error messages while it's chewing up memory, but they're not being handled by GiA.
If you want to look into this - and you're bored - I have attached a bash script that you can use to run get_iplayer outside GiA (though it will use your GiA output directory and XBMC naming preferences). If you don't know about bash scripts or working at the command prompt, ignore this.
chmod +x ./get_iplayer-gia.sh
./get_iplayer-gia.sh --verbose --modes=flashhd2,flashhd1 --pid=b03q5vlj
I'm not interested in the whole log, just if you see a bunch of errors after:
WARNING: Stream does not start with requested frame, ignoring data...
As an alternative to the shell script you can also create a separate installation of get_iplayer as described here: http://squarepenguin.co.uk/guides/mac-os-x-quick-install-guide/ That setup uses yet another build of rtmpdump, but it would be useful to know if it handles resuming without going haywire. In this case, the test command becomes:
get_iplayer --verbose --modes=flashhd2,flashhd1 --pid=b03q5vlj
Attachment: get_iplayer-gia.sh
From thomas.h...@gmail.com on January 18, 2014 14:25:11
... WARNING: Stream does not start with requested frame, ignoring data... WARNING: Stream does not start with requested frame, ignoring data... DEBUG: Checked keyframe successfully! DEBUG: ignoring too small video packet: size: 2 ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! ERROR: Couldn't find the seeked keyframe in this chunk! etc.
From thomas.h...@gmail.com on January 18, 2014 15:29:56
Attachment: get_iplayer_Father_Brown_10_failure_resuming_terminal_session.rtf
From dinkypumpkin on January 18, 2014 17:33:12
Clearly, the new rtmpdump doesn't work any better than the old version in terms of resuming downloads. The results are arguably worse - a runaway rtmpdump instead of a "corrupt file" message and quit. I can amend GiA to handle the additional error messages, but it seems like a bad idea to have a flawed rtmpdump deployed. I'll make a new GiA build this week, probably with the latest "standard" version of rtmpdump. It probably won't do any better at resuming downloads, but at least it should quit cleanly and not screw up your machine. Until then, keep using 1.5.9.
From thomas.w...@me.com on January 24, 2014 16:47:52
Yeah, I'm seeing the same thing on my end after having used it for a while. I think reverting back to the latest standard version of RTMPDUMP is a good call. The only thing it seems to resume better is when a download cancelled and then resumed. Other than that, there seem to be a lot of issues. At some point when I get chance I may try and figure out where KSV/Anonymous' patches are falling down. In the meantime, the old "reliable" version is the best bet. I'll make that change this weekend.
As for bug reports, I originally was able to respond to almost all of them but as usage has increased, I can no longer keep up. I get about 25-50/day and as a college student with an already large workload, it's tough to keep up. So at this point, I tend to use them to discern major trends rather than as a means of communicating with individual users.
Summary: New RTMPDUMP: Resume issues (was: Most downloads get stuck initialising for ever.)
Status: Accepted
From thomas.w...@me.com on January 24, 2014 16:49:28
Labels: -Priority-Medium Priority-Critical Milestone-Next
From dinkypumpkin on January 24, 2014 17:56:17
Thomas - New rtmpdump build has been pushed. I was sitting on it until the weekend, but if - as it sounds - you are going to cut another release, rtmpdump is ready to go.
As to why the KSV patch doesn't work, it seems to be this: The patched code expects the keyframe where resuming should begin to be duplicated in the first packet of flash video data after the keyframe. But at least for iPlayer streams, the calculation it uses to find the duplicate keyframe doesn't work because the timestamp of the first keyframe is zero where a non-zero value is expected to be used in calculating an offset to identify the duplicate keyframe. The result is that rtmpdump will seek to the end of the stream looking for the duplicate keyframe, but never finds it and thus never resumes downloading. In theory, this approach would make for better download resuming if the arithmetic could be made to work for iPlayer streams. For now the only improvement in the KSV version is that it does better at initiating live streams, which is irrelevant to GiA.
From dinkypumpkin on January 27, 2014 11:18:56
Standard version of rtmpdump restored in 1.6.2-pre.1. See wiki: PreReleaseBuilds
Status: Fixed
From thomas.h...@gmail.com on January 17, 2014 08:28:32
What steps will reproduce the problem? 1. Start to download a new show. It starts downloading a few MB normally.
(It might work with tea as well) What is the expected output? What do you see instead? Initialising is expected to conclude and downloading to resume. What version of the product are you using? On what operating system? Version 1.6.1 (645) on OS X Version 10.9.1 Please provide any additional information below. I am using a reliable VPN from China. The internet connection is good but may sometimes suffer hiccups. This problem is recent (a week or two) after a period of bliss downloading shows without problems. It affects almost all attempts at downloading shows. It might be related to my starting using version >= 1.6.0.
I reported the problem using the GiA bug reporting facility with a log attached but I am not sure you get these.
Thanks.
Original issue: http://code.google.com/p/get-iplayer-automator/issues/detail?id=336