R&D Opendap

[meomancer]

Problem

• R&D for opendap
• How to use it (querying the nextCDF file with attribute+spatial query)
• How to integrate it to infrastructure

[danangmassandy]

Update on this ticket: I'm now using xArray instead of OpenDAP+pyDap. There is improvement when reading the files from s3 (minio). Previously using OpenDAP+pyDap to read 10 files of CBAM NetCDF, the API took 6-10seconds. Now, the API only take 1seconds. The other benefit is we can now fetch multiple variables at once.

The downside that I noticed is that the xArray will download parts of NetCDF files bigger than OpenDAP with DMR++. Note: I cannot generate DMR++ file for Salient dataset, so the previous API will have to download the file first into Hyrax file cache.

Other downside when using OpenDAP:

• need more configuration to read files from S3 without generating DMR++ files
• not supporting zarr format