search

mapproxy / mapproxy

MapProxy is a tile cache and WMS proxy
http://mapproxy.org
Apache License 2.0
553 stars 253 forks source link

Seed throttle #89

Open ramunasd opened 11 years ago

ramunasd commented 11 years ago

Often imagery source is very slow and cannot handle many requests at same time. So when someone wants to warm up it's own cache, he cannot control download speed and imagery source can crash, especially this happens with ESRI products. That's why mapproxy-seed needs some throttling mechanism. It could be based on request rate per second or downloaded data size per second, or maybe time pauses between requests could help. Throttling would be very useful option for mapproxy-seed.

olt commented 11 years ago

req/sec and kb/sec would be tricky to implement in combination with --concurrency, because the seeders run in separate processes and can't share a single progress meter. A delay should be easier to implement.

ramunasd commented 11 years ago

Delay between requests should be fine.

miloskroulik commented 10 years ago

This seems to be very important for us, because we need to seed source, that warns against sanction in cases of overuse (which is not defined precisely).

steamboatid commented 7 years ago

please provide a feature to set the minimum delay when the mapproxy-seed failed to fetch. by default it set as 2 seconds with the same incremental delays.

it's good we can set base minimum greater than 2 seconds and its incremental delays more than 2 seconds. let's say 10 second for the base minimum and 15-20 seconds for its incremental. apply logaritmic incremental also beneficial.