search

lovasoa / dezoomify-rs

Zoomable image downloader for Google Arts & Culture, Zoomify, IIIF, and others
https://dezoomify-rs.ophir.dev
GNU General Public License v3.0
739 stars 66 forks source link

Slown down the resquests to prevent fllooding servers #24

Open XRolland opened 4 years ago

XRolland commented 4 years ago

Hi,

Is possible to get a parameter to slow down the requests when you download at big zoom level (it's what's most of us are targeting, isn't it?) to prevent flooding the server and getting some time-out or 404 error?

Thanks for this good utility!

URL example : https://www.siv.archives-nationales.culture.gouv.fr/mm/media/download/FRAN_0196_1183_L/ImageProperties.xml?t1582631960253n0

lovasoa commented 4 years ago

Hello, and thank you for getting in touch. Dezoomify does not have an option to make it explicitly sleep between successful requests, but it does have a way to control the concurrency and retry policy, which should allow you to fit in most servers' fair usage policies. See the list of options in our README. The most important option is parallelism, which controls the number of requests that can be in flight at the same time. A conservative configuration could be:

./dezoomify-rs --parallelism 1 --retries 8 --retry-delay 30s --timeout 5min

However, you mention two things: timeouts, and 404s. I tried the image you give a link to, and I get only 404s, (meaning that tiles are not present on the server), and no other error. There is nothing dezoomify can do if the server is missing tiles, such as in your example.

dezoomify works just fine with the tiles that are present :

dezoomified

The remaining tiles simply do not exist :

https://www.siv.archives-nationales.culture.gouv.fr/mm/media/download/FRAN_0196_1183_L/TileGroup2/6-14-11.jpg ... https://www.siv.archives-nationales.culture.gouv.fr/mm/media/download/FRAN_0196_1183_L/TileGroup2/6-4-14.jpg

(I tried from several different IPs, so it's not just a misconfigured server returning 404 instead of 401. These are legitimate 404, the tiles do not exist)