Hack Wrapup

[jbusecke]

Great day hacking on using virtualizarr to produce ref files on ESGF with @sashakames.

We successfully produced a few ref files, exposed them via HTTp, and were able to access (and compute on them) in some environments.

Bugs:

• [x] The metadata is lost in the combination step (@jbusecke will provide fix). (fixed in https://github.com/jbusecke/esgf-virtual-zarr-data-access/commit/3baa6a9cb7c325fd9109500c4557796aa7bb94d7)

Optional:

• [ ] @jbusecke parallelize virtual dataset generation

Plan to wrap up proof of concept:

• [ ] Produce a few more demo references
• [ ] Write instructions on how to access them
• [ ] Send out to users to test (@jbusecke was not able to compute on the data via Google Cloud)
  • [ ] Give @sashakames access to the LEAP hub for testing.
• [ ] Write up for report.

We should also wait until https://github.com/TomNicholas/VirtualiZarr/pull/126 is fully tested and merged until we produce a lot of references. I believe there is currently a bug in the PR, but it is easy enough to circumvent.

[jbusecke]

@sashakames I just cleaned up the code a little bit. Lets use https://github.com/jbusecke/esgf-virtual-zarr-data-access/blob/main/virtual-zarr-script.py and https://github.com/jbusecke/esgf-virtual-zarr-data-access/blob/main/requirements.txt to produce the next files.

I will add dependencies and code there to parallelize virtual data and fix the bugs above.

[TomNicholas]

The metadata is lost in the combination step

Do you mean the metadata stored in the xarray .attrs or something else?

Yes that is right. In my initial attempt the metadata was lost during the concat step due to the default behavior of xarray to just drop all of it. I have now set this option to only drop conflicts