Open observingClouds opened 1 year ago
Again, this is related to my comment in #10 . I think
A few more insights. The different is likely caused by dask and how the dask task graph looks like. Depending on the available resources the task graph is created differently. If more resources are available then more data is requested.
open-dataset
requests and is thereby independent of the available resources.With #21 being merged the recommended way to retrieve files is to use the ds.slk.stage()
command which operates independent of the available resources.
This issue unfortunately seems to remain. Dask still schedules the retrievals depending on the available resources to the cluster.
xarray seems to request different amounts of files concurrently depending on the cluster configuration:
levante interactive
levante compute