SlideRuleEarth / sliderule

Server and client framework for on-demand science data processing in the cloud
https://slideruleearth.io
Other
26 stars 11 forks source link

enabled gdal multirange, needs ulimit -n 8192 #345

Closed elidwa closed 9 months ago

elidwa commented 9 months ago

GDAL_HTTP_MULTIRANGE has been enabled but it requires larger than default per process max file descriptor limit. On ubuntu add entry to /etc/security/limits.conf

Do something similar for docker images.

With multirange enabled GDAL curl driver will open several tcp sockets and request ranges data over multiple connections. This can speed up downloads up to 25% to 30% for some data sets (ESA worldcover).

With multirange enabled and default nofile set to 1024 both selftest and subset_perf_test fail. They should pass after nofile limit has been increased.