Get painting list requests get throttled by wikiart

LorenzoCianciaruso commented 5 years ago

When running genre-scraper.py using the currently harcoded values for randomization

time.sleep(3.0*random.random())  # random sleep to decrease concurrence of requests

requests get throttled by Wikiart that returns

[Errno 104] Connection reset by peer

I think there are 2 improvements:

Throttled a bit more the request from the script side by having a fixed latency summed up to the randomised one.
when you run the script only failures are printed out in the console. And because of that initially I wrongly assumed the the script wasn't working at all. In reality all the successful requests are not logged, so might be good to add some more logs.

I'm happy to open a PR for this.

ghost commented 3 years ago

I'm getting the same error but there doesn't seem to be any successful downloads at all.

failed to scrape URL [Errno 54] Connection reset by peer

I'm using the following command

python genre-scraper.py --genre abstract --output_dir abstract

spasmann commented 3 years ago

Am having the same issue. But only when I'm running through a remote computer cluster. If I run on laptop I get 100% of the images, no throttling.

Tried removing the random.random() and it actually scraped about 500 less images.

Depending on the style/genre it sometimes doesn't download any at all. With the larger sets it usually gets somewhere 900-1500 downloads.

robbiebarrat / art-DCGAN