search

stcorp / harp

Data harmonization toolset for scientific earth observation data
http://stcorp.github.io/harp/doc/html/index.html
BSD 3-Clause "New" or "Revised" License
55 stars 19 forks source link

include numpy int64 mapping to harp integer in python interface #253

Closed StevenCompernolle closed 3 years ago

StevenCompernolle commented 3 years ago

When exporting from python to a harp netcdf file, variables with dtype int64 were converted to a float in the netcdf file. Within python, a print of the Variable gave "Variable type=double" although the numbers within the array were integer.

int64 is missing in the table at https://stcorp.github.io/harp/doc/html/python.html, which probably explains things.

I could prevent the conversion to float in the netcdf file by first converting to dtype int32 within python. Then one gets "Variable type=long". Still, it seems better if HARP would take this into account automatically.

svniemeijer commented 3 years ago

We can't use int64, since we need to be able to store data in both HDF5(/netCDF4), HDF4, and netcdf3 format. And HDF4 and netcdf3 do not support int64 data types. So support for int64 will never be an option for HARP.

Also, I am not sure why you would need int64. It provides no size benefit over double (both are 8 bytes, and you probably want to use float anyway if you don't need that many significant digits). And quality flags with 64 bits are overkill. If you need that many bits, you should probably rethink your flagging and simplify things.