search

nasa / zarr-eosdis-store

Zarr data store for efficiently accessing NetCDF4 data from NASA’s Earth observing system in the cloud using the Zarr Python library
Other
18 stars 6 forks source link

Handle CompressionType = "deflate shuffle" #5

Closed ashiklom closed 3 years ago

ashiklom commented 3 years ago

I have a NetCDF file that produced a dmrpp file with the following compressionType:

<dmrpp:chunks compressionType="deflate shuffle" byteOrder="LE">

This results in EosdisStore throwing an Unrecognized CompressionType error. Full traceback in "Details" below, but it's pretty clear and self-explanatory. Looks like you already handle the deflate case, but dmrpp.py needs to be revised to also handle the deflate + shuffle case.

The source file is the first tavg1_2d_flx file from here (direct file link), with the .dmrpp created via the mkdmrpp script in this repo.

``` in ----> 1 zfile = EosdisStore(netcdf, dmr_url=dmrpp) ~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/stores.py in __init__(self, data_url, dmr_url) 256 dmrpp = requests.get(dmr_url).text 257 tree = ElementTree.fromstring(dmrpp) --> 258 meta_store = to_zarr(tree) 259 super(EosdisStore, self).__init__(meta_store, data_url) ~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/dmrpp.py in to_zarr(root) 277 zarr = {} 278 dims = get_dimensions(root) --> 279 zarr = group_to_zarr(root, dims) 280 return zarr 281 ~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/dmrpp.py in group_to_zarr(node, dims, prefix) 245 # if this is an array, convert to zarr array 246 if tag in TYPE_INFO: --> 247 zarr_array = array_to_zarr(child, dims, prefix=prefix) 248 zarr.update(zarr_array) 249 # otherwise, if this is group or a Container Attribute - this has not been tested ~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/dmrpp.py in array_to_zarr(node, dims, prefix) 204 zarray['compressor'] = None 205 else: --> 206 raise Exception('Unrecognized compressionType: ' + compression) 207 chunks = chunks_to_zarr(child) 208 zarray.update(chunks['zarray']) Exception: Unrecognized compressionType: deflate shuffle ```
bilts commented 3 years ago

This is fixed in v0.1.2, which was just published to pypi. Thanks for the feedback!