Incorrect mapping of values whose exponent bits are either all-zero or all-one

[zackw]

In IEEE floating point format, the maximum possible value of the biased-exponent field (all bits 1) is used for the special ±Inf and NaN values. In all four VAX floating point formats, it's the other way around: "reserved operands" have the minimum possible value of the biased-exponent field (all bits zero) and also sign bit 1. (If the biased-exponent field is all-zero and the sign is also zero, the overall number is treated as zero regardless of its mantissa bits.)

rms-vax doesn't seem to be taking this into account, meaning that there's a chunk of the space of normal VAX floats that are being mapped to IEEE NaNs, and conversely the VAX reserved-operand values are not being mapped to NaNs. For example

>>> vax.from_vax32(b'\x80\x7f\x00\x00')  # should produce 8.507059e+37
np.float32(inf)
>>> vax.from_vax32(b'\x80\x7f\x55\x55')  # should produce 8.529213e+37
np.float32(nan)
>>> vax.from_vax32(b'\x00\x80\x55\x55')  # should produce NaN
np.float32(-7.652e-42)
>>> vax.from_vax32(b'\x00\x80\x00\x00')  # should produce NaN
np.float32(-0.0)
>>> vax.from_vax32(b'\x7f\x00\xff\xff')  # should produce zero
np.float32(2.938736e-39)

(As you know, VAX floats do not have ±Inf, negative zero, or subnormals.)

I believe that all the values whose exponent bits are either all-zero or all-one are currently being misinterpreted, in both directions.

[zackw]

Note: I do not have actual VAX hardware to test on. I'm going by https://dl.acm.org/doi/pdf/10.1145/641845.641849, in particular the beginning of section 3.0 (page 2 of the PDF, numbered as page 23).