search

bibanon / BASC-Archiver

Python-based Imageboard (4chan) complete thread archiver.
https://pypi.python.org/pypi/BASC-Archiver/
135 stars 18 forks source link

Add Delay to Image Downloading, and Handle Connection Refused #8

Closed antonizoon closed 9 years ago

antonizoon commented 9 years ago

After scaling up the 4chan archiver for heavy duty scraping, I think we should really add a delay setting to image/file downloading (not just thread inquiry). 4chan starts to give us Connection Refused errors.

Also we need to handle the Connection Refused error in some safe manner.

DanielOaks commented 9 years ago

Hmm, something like waiting X ms between each image download, or are you thinking something different?

antonizoon commented 9 years ago

Yes, make a configurable delay specifically for file downloads. I think in the download() function.

DanielOaks commented 9 years ago

Sounds good, should be pretty simple

antonizoon commented 9 years ago

Alright, I added the option to delay image/thumbnail download.