Closed GoogleCodeExporter closed 8 years ago
Currently in designing the algo for this...
Must have the initial implementation in 2 days.
Original comment by abpil...@gmail.com
on 16 Jun 2008 at 1:27
Fixed the issue. Changes in connector.py .
Original comment by abpil...@gmail.com
on 25 Jun 2008 at 8:58
Apparently, this needs rework. Quoting from mail by Lucas,
> I just tested the speed and for some reason I get:
>
> <maxbandwidth value="5"/>
>
> 4120 links scanned in 1 server .
> [01:12:54] 69 files written.
> [01:12:54] 1450624 bytes received at the rate of 8.12 KB/sec .
> [01:12:54] 9498335 bytes were written to disk.
>
>
> try crawling:
> http://www.automotive.com/used-cars/index.html
>
> to test this.
> Thanks.
> Lucas
Re-opening the bug for a better solution.
Original comment by abpil...@gmail.com
on 28 Jun 2008 at 2:24
Hope this one helps too.
http://www.ibm.com/developerworks/aix/library/au-threadingpython/
or
http://www.velocityreviews.com/forums/t583705-urllib2-rate-limiting.html
Lucas
Original comment by szybal...@gmail.com
on 29 Jun 2008 at 4:45
Fixed this by implementing the logic on the controller thread on all data
downloaded
so far, instead on a per connector basis.
Lucas, please test this. Thanks!
Original comment by abpil...@gmail.com
on 1 Jul 2008 at 9:36
Original issue reported on code.google.com by
abpil...@gmail.com
on 16 Jun 2008 at 8:41