search

leftist / warrick

Automatically exported from code.google.com/p/warrick
0 stars 0 forks source link

Warrick Sleeping 2 #19

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
Please describe your feature requests here.

Once you start timing how long mcurl is taking (too see if it has hung or not), 
then you can adjust how long Warrick sleeps between requests to maximize 
recovery speed.  

Right now, Warrick is sleeping for 7 seconds between requests, but if the 
Internet is unclogged, and archive.org is returning the page to mcurl in 1 
second, then why not sleep for 1 second since no one else is using the 
bandwidth and archive.org?  If archive.org returns a result in 10 seconds 
because it's loaded down, THEN Warrick should sleep for 10 seconds to ease up.  
A 30 second response would be matched with a long 30 second sleep.

Simply put, Warrick should sleep the number of seconds it takes to pull the 
last page.  This is how the ftp algorithm maximizes bandwidth, without killing 
the internet.

Original issue reported on code.google.com by jbrunell...@gmail.com on 5 Sep 2012 at 11:22

GoogleCodeExporter commented 8 years ago 
The command line options handles this, now. It's also probably overkill for 
most options.

Original comment by jbrunell...@gmail.com on 28 May 2013 at 6:44