search

eurec4a / eurec4a-intake

Intake catalogue for EUREC4A field campaign datasets
17 stars 19 forks source link

OPeNDAP: change chunk from {} to null #60

Closed d70-t closed 3 years ago

d70-t commented 3 years ago

There's an issue with the interworkings of dask and OPeNDAP: If dask decides to do chunks on a dataset, it will request the whole chunk from the backend storage, even if only a part of it is needed in the end. However for OPeNDAP it is best to request only the required parts from the backend. This change disables initial dask-chunking for all datasets which previously used only a single chunk.

While this likely will improve performance for a lot of cases, it may not always be the best solution. If multiple datasets are to be concatenated, dask has to be used. However, in those cases, the user maybe knows about chunking an will define those manually.

RobertPincus commented 3 years ago

@d70-t Many tests are failing - do we know why?

d70-t commented 3 years ago

Yes, we do:

Bildschirmfoto 2021-03-18 um 16 56 14

The OPeNDAP-Server is responding with 503 Service Unavailable. The plot shows the availability during the last 14 days.

RobertPincus commented 3 years ago

schade

d70-t commented 3 years ago

@RobertPincus now it passed 🎉