List and evaluate tools/technologies for dataset transformation

[briannapagan]

This is really exciting work! I just wanted to plug leveraging pangeo-forge if converting datasets to zarr: https://pangeo-forge.org/ https://github.com/pangeo-forge

[aldenks]

Hi @briannapagan, thank you for your excitement and suggestion! The code in this repo is a part 1 of trying out a few technologies/libraries to do this work, the others on the top of our list are (this list is expanding as more suggestions come in)

• pangeo-forge
• xarray-beam
• virtualizarr
• google weather-tools
• potentially some combination of the above

I'm going to rename this issue and leave it open in case others come across it and have other suggestions of tools to evaluate!

[dikwickley]

If you'd like to download data from ECMWF, you can also take a look at https://github.com/google/weather-tools/tree/main/weather_dl

[aldenks]

oh yes, thank you for the reminder @dikwickley!