Support for envi float16

[petermoore14]

Possibly barking up the wrong tree but have to start somewhere.

I have a specific use case where support for np.float16 in envi cubes would be invaluable. According to the envi specs at

https://www.nv5geospatialsoftware.com/docs/ENVIHeaderFiles.html

... there does not exist a 'data type' for float16. Given the variety of types supported, however, this omission seems to stick out, and in theory there's no reason why float16 COULDN'T be supported. Wondering if there is any specific reason why it isn't, or if it's on the agenda somewhere.

Apologies if this is the wrong venue for this; would greatly appreciate being pointed to the proper authorities.

[tboggs]

You could test this by cloning the spectral repo and editing the dtype_map variable, defined in spectral/io/envi.py.

[petermoore14]

@tboggs Thanks; yeah making our own fork for this is something I could attempt, but hoping to get insight from the experts first. As mentioned the absence of float16 sticks out, didn't want to start working on it only to discover some hours later that there's an obvious stumbling block with that particular data type I missed when flipping through the source code.

[tboggs]

I'm not aware of anything obvious that would prevent it from working. I assume I just stuck with the officially supported ENVI data types (vice using all numpy data types) for the sake of compatibility.

For a quick & dirty test, you don't actually need to fork the repo. You can just clone the repo, edit the file, and install the folder via pip (or edit PYTHONPATH). Just note that you should edit the file, rather than changing the variable after import the module, because a few other dependent variables are defined by it when the module loads.

[petermoore14]

Right that makes sense @tboggs; obviously you'd want to match the spec as closely as possible, which is what the library is doing now. The envi specification supporting additional types would be the only reason to modify this lib.

I definitely exaggerated the time to experiment with this a bit; certainly shouldn't take hours to confirm. My use case would ultimately require a fork so that we can host the custom module for others to use when reading our cubes, which would be the much more time consuming part I was hoping could be avoided.

Anyway thanks for the advice; consider this answered.

[tboggs]

No problem.

If it is confirmed to work and there is a reasonable use case for it, it could certainly be added to the module. There are numerous things the module supports that ENVI doesn't. As long as it doesn't generally break ENVI compatibility, it could still be added. We would likely just require that one explicitly indicates float16 when saving the image (e.g., via keyword) and maybe issue a warning that the saved file is not compatible with ENVI.

[petermoore14]

Ah, well that would be splendid! Assuming the experiment works I'd be happy to make the PR for that. Totally understand not wanting to break compatibility and using a keyword/tag of some sort would be fine by me. So 'data type' stays the same and follows the spec as before, but when the keyword is present the spectral library will happily read/write as float16s, with some extra warnings as you mentioned.

The specific use case is storing reflectance. Float32 produces needlessly large cubes for our precision requirements. Can obviously 'shoehorn' lower precision data into the existing types with some custom pre-write/post-read code, but was shooting for a more elegant/robust solution if at all possible

[tboggs]

fwiw, I believe most people up to now handle that by multiplying floating point reflectance by a scale factor and converting the result to int16. See "reflectance scale factor" here. The spectral module handles that conversion automatically.

[petermoore14]

Facepalming right now for not noticing that earlier. We're doing something similar at the moment, but it's ugly because we can't really share/exchange our data without providing the secret sauce to 'decrypt' our uint16 cubes. Given that this is a standard hdr field defined by the envi spec, that issue is resolved if we change our setup to use 'reflectance scale factor' instead.

Would obviously still be a bit nicer/more succinct if one could simply envi.load reflectance without having to do any scaling, but ultimately not the end of the world and probably not worth the effort of supporting float16 in spectral. Most important thing for us is the ability to communicate that reflectance using a smaller data type, without any customized 'magic spells' to translate it.

Thanks for the help Thomas, much appreciated.