Incorrect conversion of double precision constants to IEEE-754 format

[alvoskov]

OCC 6.73.8 incorrectly transforms 2^53-111 to IEEE-754 double format. It gives a rounding error for an exactly representable number. The rounding error is small but it may be critically important by some pseudorandom number generators like RANMAR and https://doi.org/10.1016/j.spl.2003.11.001.

The code for bug demonstration:

#include <stdio.h>
#include <stdint.h>

typedef union {
    double x;
    uint64_t i1;
} dblint;

int main()
{
    // OCC gives 0x433FFFFFFFFFFF92 (i.e. 9007199254740882.0) 
    // GNU Octave and GCC give 433fffffffffff91
    // Test for GNU Octave/MATLAB
    // >> format hex; (2^53-111)/2^53
    cc.x = 9007199254740881.0; // (2^53 - 111)
    printf("% .18f | 0x%llX\n", cc.x, cc.i1);
}

0x43_3f_ff_ff_ff_ff_ff_91: Sign bit is 0 Exponent is 0x433 (i.e. 1075 - 1023 = 52) Mantissa is [1]f_ff_ff_ff_ff_ff_91 = 2**52 + 0xf_ff_ff_ff_ff_ff_91 (9007199254740881)

[LADSoft]

I verified the compiler issue, but it doesn't seem to happen in the runtime as well.

Will fix it soon....

[alvoskov]

It is not in runtime, because it reproduces even in cpp source in constexpr. Disassembling shows that the binary constant inside EXE file is already calculated (from decimal record) wrong.

[LADSoft]

ok thanks. I made a fix, the code that was truncating things to fit in 53 bits was broken, probably whenever the high and low bits of the 53 bit number were set it would round up... should be good whenever it builds...

[GitMensch]

@LADSoft as I've seen no related commit - did you add that to the tests or is this piece missing to be able to mark that as closed-solved?

[LADSoft]

sorry i forgot to tag the commits with the issue number when i made them. Here are the commits for this issue:

e0addaa92a6621e7f70c032908e7b46b53857fb6 ( fix issue)

c860f473a294308f0d3cb84efd2ecdfd72967a31 (add tests)