The current metadata format is subject to single bit errors. I thought it was purely a "Q" format float, but it looks like it contains to float quantization as well.
For example, writing -1925348.096 read back as -1925347.968, a difference of 0.128. This demonstrates a "exponent/mantissa" rounding error and not just a 0.001 error as previously thought.
Some proposals:
Try to represent as float binary data (">f") instead of Q format int (">i" * 1000). There are enough issues with this that it doesn't seem to work well
Keep existing Q format, but do some tricks around the mantissa. Seems too fragile
Move from 9 bytes per WCS to 8 bytes per WCS, giving a few more bits of lenience per coordinate. Probably simplest and 8 bytes is probably fine anyway
Most likely will do the last scheme, with the only question being how to pack the bits.
The current metadata format is subject to single bit errors. I thought it was purely a "Q" format float, but it looks like it contains to float quantization as well.
For example, writing -1925348.096 read back as -1925347.968, a difference of 0.128. This demonstrates a "exponent/mantissa" rounding error and not just a 0.001 error as previously thought.
Some proposals:
Most likely will do the last scheme, with the only question being how to pack the bits.