search

leap-stc / wavewatch3_feedstock

Apache License 2.0
0 stars 0 forks source link

Caching issues #3

Open jbusecke opened 6 months ago

jbusecke commented 6 months ago

Classic case of a moody server here that shuts requests down if we have more than 1 worker running. Each time I observed a failure on the server side, the files were downloadable to my laptop without issue. This is particularly problematic since this also requires ridiculous amounts of ram on the worker (#2 ) and thus having these expensive workers idling around for downloads exacerbates the problem.

jbusecke commented 6 months ago

This might be somewhat alleviated by specifying: OpenWithFsspec(..., max_concurrency=...) but together with #2 this still means that one or two expensive workers are just idling around...