Closed jbusecke closed 1 year ago
We (at Google) would be excited to be able to play around with the ARCO version of ClimSim. I think this could make for an interesting training dataset for some of our ML weather models :)
@shoyer (Sorry for the dumb question.) What is "ARCO"?
"Analysis-Ready, Cloud Optimized". See this paper for a longer definition.
So basically:
That's the goal here.
@rabernat Thanks for the clarification. One challenge would be the gigantic size of the high-res dataset, ~41TB. (BTW, we already have the 'Zarr'-ed real-go low-res dataset in LEAP cloud storage.)
One challenge would be the gigantic size of the high-res dataset
This is not a problem in any way. On the contrary, this is exactly the type of dataset that benefits most from the ARCO approach. 🚀
For comparison, Google's ARCO ERA5 is hundreds of TB.
Yes, absolutely, Zarr's the way to go for these huge datasets. However, what I meant was processing netcdf files (>400k files totaling 41TB) to a single Zarr file. It took quite a long time just for a low res dataset (same file number but much smaller size at several 100s GB). Maybe there's a more efficient way of doing this, I guess.
Maybe there's a more efficient way of doing this, I guess.
@sungdukyu 🤝 Meet Pangeo Forge!
@rabernat Thanks! Seems like my bakery was suboptimal. (https://pangeo-forge.org/pangeo-forge-diagram.png)
Since we have the right audience here, would it make sense to explore uploading a zarr store directly to huggingface?
Let's not try to mix in huggingface-related development work. Let's just try to ship the ClimSim dataset as fast as possible by whatever means necessary.
I can try to spend some time on this next week! In case somebody wants to start before that, starting a recipe PR in our LEAP feedstock would be the way to go.
@jbusecke could you push the work we began together on this as a PR?
I have some time I can devote to this and would love to pick up where we left off.
See #33, sorry for the delay.
Thanks to a lot of work by @sungdukyu @jerrylin96 and many others, the E3SM-MMF(ne4) aka ClimSim dataset is now available on huggingface.
I would like to reingest it via pangeo-forge to make this workflow entirely reproducible!