Open aaronspring opened 2 years ago
Some further interesting reading for me:
unlimited_dims=[]
in to_netcdf()
What I do not want is that the compression impacts the read performance too much, i.e. reading the compressed data should not take much more (subjective I know) time that compared to before (or is this the tradeoff anyways).
Many lossless compressors have a decompression speed that's approximately independent of the compression level. So read performance is rarely something to worry about with this method. E.g.
(from https://www.nature.com/articles/s43588-021-00156-2)
compress and save with xarray in xbitinfo
: https://xbitinfo.readthedocs.io/en/latest/api.html#save-compressed
I want to use
BitInformation
andBitRound
on NetCDF output. Anyone with experience withxarray
?What I do: see also https://github.com/milankl/BitInformation.jl/issues/25#issuecomment-1060889412
My routine:
plot_bitinformation.jl
and save keepbits between 0 and 23 to adict
withjson
BitRound
for each variableds_rounded.to_netcdf(path, encoding, engine='h5netcdf')
I use
engine="h5netcdf"
because I can use{"compression": True}
in encoding. I do this because some compression is needed forBitRound
to be effective on disk storage. (I have only very little knowledge of compression and netcdf4 basics.) But probably there is a better way. What I do not want is that the compression impacts the read performance too much, i.e. reading the compressed data should not take much more (subjective I know) time that compared to before (or is this the tradeoff anyways).https://xarray.pydata.org/en/latest/generated/xarray.Dataset.to_netcdf.html
EDIT:
Result: