enhance remaining time indication

[GoogleCodeExporter]

The remaining time indicator on downloads varies wildly with network speed,
making overall reliability quite low.

Would it not be a better idea to keep track of how large a part of the file
has been downloaded, how long that has taken and work out the ETA from
that? In this fashion network speed variations are ironed out and the
remaining time indicator would get more reliable further into the download.

Original issue reported on code.google.com by ultrana...@gmail.com on 7 Oct 2009 at 10:08

[GoogleCodeExporter]

I agree the current speed indicator might be a bit too real-time, but only 
looking at
the entire file OTOH is a too big window I think. Marked as todo: have a look 
at what
indication strategies other applications implement. If you got an idea, please 
mention.

Original comment by tim.besard@gmail.com on 10 Oct 2009 at 4:13

• Changed state: Accepted

[GoogleCodeExporter]

Well I'm a digital filter designer so this is daily work :)

Keeping a moving average works quite well for filtering out transient errors. It
would simply be the average of say, the last 10 measured speeds (if available). 
More
sophisticated would be (if the reported speed update is a known time interval) 
to
define a speed averaging time in the config file and work out from there how 
many
past speeds to average. Applying weight factors and noise shaping is too much 
work
for such a relatively simple application.

I think you're right that averaging the entire file may be erroneous as well, 
for
instance when I'm starting a parallel download and my bandwidth drops 
systematically
rather than incidentally.

I'd say an averaging window approximating 30 seconds of activity would be a nice
balance between ironing out random fluctuations whilst accurately reporting 
remaining
time even under time-varying pipe widths.

Original comment by ultrana...@gmail.com on 13 Oct 2009 at 2:50

[GoogleCodeExporter]

Thanks for your hints, and sorry for the late reaction. Things have been busier 
than
I had hoped. I'll be releasing without a fix for now, but it'll get addressed 
soon.

Original comment by tim.besard@gmail.com on 26 Oct 2009 at 5:41

[GoogleCodeExporter]

This issue was closed by revision r379.

Original comment by tim.besard@gmail.com on 6 Nov 2009 at 9:20

• Changed state: Fixed

[GoogleCodeExporter]

Implemented in r379. I used a time-window based moving average, window size is
configurable using the "speed_window" key in the configuration file. Do comment 
if
you have any criticism on the implementation though.

Original comment by tim.besard@gmail.com on 6 Nov 2009 at 9:23

[GoogleCodeExporter]

Ok, I ran tests on this and it sure is a lot better. I have played with the 
window
setting from 10 to 3600 seconds. I'm getting much better average speed readings.
However I can still not very much rely on the remaining time indicator.

I ran a --debug download and created the graph attached from parsing the log. 
In this
graph I'm plotting in blue the remaining time indicated versus in red the real
remaining time. The good news is that towards the end the ideal and measured 
line
converge and that overall the indication error decreases with time. The bad 
news is
that as you can see slimrat is still very jumpy (window was 3600 for this!) and
always somewhat too optimistic. I have no explanation for the peakiness / 
strange
short transients, it seems the window resets now and then?

I think this has to do with Rapidshare (again!). When it's busy as it was today 
RS
often simply stops responding for a while (between 5 and 360 seconds). During 
this
time slimrat holds its last d/l speed value while in actuality it is 0! This 
skews
the rate and leads to the optimistic value. It also explains why the errors are 
very
large in the beginning (dead time trumps the live time there). I haven't looked 
into
the code to see if this is in the slimrat code or in WWW:mechanize and I can't 
easily
suggest a fix either.

I'll have a look in the code later. This project is sucking me in :)

Original comment by ultrana...@gmail.com on 8 Nov 2009 at 2:28

Attachments:

• slimrat_remaining_indicator.tiff

[GoogleCodeExporter]

Additional plot generated from a Megaupload download to demonstrate the problem 
with
the rapidshare indication. Notice the differences, this one has no dead time and
therefore much smoother behaviour (although nowhere near as smooth as e.g. 
wget).

The two peaks are from when I downloaded another file in the backgound, such 
that the
Megaupload speed dropped for a moment. This is expected behaviour.

My window_length is 3600 (I'm not sure this registers, however!).

Original comment by ultrana...@gmail.com on 15 Nov 2009 at 2:31

[GoogleCodeExporter]

Thanks for having a detailed analytical look at the problem, your conclusion is
correct indeed. Rapidshare doesn't throttle smoothly: it spits out high-speed 
chunks
of data and idles after it in order to get the global transfer speed down to the
level acceptable for non-premium users. This fools slimrat, as the global 
progress
indication code reads the speeds calculated by every download thread, which get
calculated upon each received chunk of data. As the speed calculation thus 
happens
_right after_ having received such a high-speed data chunk, the per-thread speed
value is set at a way too optimistic value, and re-used by the global speed 
indicator
for quite a long time.

The fix I have in mind is not to calculate the thread speed after chunk 
reception,
but only to propagate the (currently internally used) chunk data. This'll 
enable the
global speed indicator to accurately re-calculate the download speed in the 
interval
in which Rapdidshare idles and the speed wouldn't get updated normally.

I'll have a look at it once I get some time, but having narrowed the problem 
down
pretty closely it shouldn't take too long. Still, patches are always welcome ;)

About the window_length: you're awfully right, pushing this change right now.

Original comment by tim.besard@gmail.com on 15 Nov 2009 at 9:44

• Changed state: Accepted