search

pangeo-data / pangeo-cmip6-cloud

Documentation for Pangeo CMIP6 data stored in GCP/AWS cloud
https://pangeo-data.github.io/pangeo-cmip6-cloud/
17 stars 9 forks source link

Issues opening certain stores #32

Open jbusecke opened 2 years ago

jbusecke commented 2 years ago

Many thanks to @Yadidya5 for bringing an issue with a certain store to my attention here.

We concluded over there that the problem is likely related to the zarr store itself (since we cannot even open it properly), and thus here is the better place to keep track of that.

I propose that any refactor as in #31 should probably have some rudimentary test of each store, ensuring that it can be opened with e.g. xr.open_zarr(store).

yadidya-b commented 2 years ago

Any update on this issue? @jbusecke @yuvipanda @rabernat

jbusecke commented 2 years ago

Hi @Yadidya5, I am just back from vacation and will dedicate quite a bit of time to the CMIP6 data in the coming weeks. Ill add this issue to the list, but might still need some time to get caught up on things that have piled up during my time away. Thanks for the ping.

yadidya-b commented 2 years ago

Thank you for your reply. I hope you had a great vacation @jbusecke!

The main reason I've brought this up is because I've (along with some of my peers) faced the same issue with multiple datasets, which is sort of crippling the main purpose of using Pangeo and cloud compution. I can understand that it's going to be really difficult to completely resolve this issue but I hope that there is a definte solution to this. Thank you once again & cheers to the amazing work that you are doing!

jbusecke commented 2 years ago

I am starting to survey how prevalent of a problem this is, and if this is the only type of error that is encountered when opening any store. For now I have used a simple for loop, but that is unacceptably slow for this purpse (takes about 2 days to crawl all stores). I will report back once I have a full view of the issue, and then we can discuss how to go about fixing this.