search

leap-stc / data-management

Collection of code to manually populate the persistent cloud bucket with data
https://catalog.leap.columbia.edu/
Apache License 2.0
0 stars 5 forks source link

Add CASM recipe #22

Closed jbusecke closed 1 year ago

jbusecke commented 1 year ago

Closing #17

jbusecke commented 1 year ago

This is again taking forever at the rechunk step. Ill abort the current run and submit with only the first/last half of years for debugging.

jbusecke commented 1 year ago

First half (2002-2012) went through in 18 min. Now trying (2012-...)

jbusecke commented 1 year ago

2012-2020 worked nicely. trying to extend the start back now.

jbusecke commented 1 year ago

YES! It successfully ran and the (for now) final location is:

import xarray as xr
ds = xr.open_dataset('gs://leap-persistent/data-library/casm-595733423-4997696883-1/CASM.zarr', engine='zarr', chunks={}) 
ds