search

equinor / segyio

Fast Python library for SEGY files.
Other
490 stars 218 forks source link

IBM360Converter converts to IEEE NaN for some values #555

Open lpartington opened 1 year ago

lpartington commented 1 year ago

IBM360Converter compares the IBM float value (as an integer) with IEMAXIB, returning IEEEMAX (+Inf) if the value is larger than IEMAXIB. IEMAXIB is defined as: public static final int IEMAXIB = 0x611FFFFF; The IBM value 0x61100000 corresponds to IEEE +Inf (0x7F800000, signifying exponent FF and mantissa 0). IBM values between 0x61100001 up to 0x611FFFFF get converted to IEEE values in the range 0x7F800008 - 0x7FFFFFF8. These are NaN values, where exponent = FF and mantissa is not 0. Then IBM values from 0x61200000 upwards are converted to +Inf again (due to the IEMAXIB test). It would appear that IEMAXIB should be 0x61100000 rather than 0x611FFFFF so that all values above 0x61100000 are converted to +Inf.

lpartington commented 1 year ago

If 0x61100000 is used, then the comparison could be changed to >= IEMAXIB for clarity (although since val == IEMAXIB yields +inf anyway, it doesn't really matter). Alternatively, > 0x610FFFFF could be used.