nearlyEqual function incorrect for values on opposite side of 0

[Stannieman]

When we have for example the values -0.000001 and 0.000001, nearlyEqual always returns false for an epsilon < 0. This is because on the last line "diff / Math.min((absA + absB), Float.MAX_VALUE)" will always return 1. Some extra care should be taken when a and b are on opposite sides of 0. I'll try to work something out, but I think that in this case an absolute comparison of diff against epsilon is valid.

[brazzy]

I really think this is really a special case where no single implementation will satisfy everyone - for some applications the difference between those two numbers is gigantic and they should absolutely not be considered "nearly equal". I'm actually more concerned about the fact that something like -0.000001 and 0.000001000001 could be considered "nearly equal" when it shouldn't be.

I definitely agree that there needs to be a special different treatment of values on opposite sides of 0, but I'd tend to considere them different by default.

[brazzy]

After debugging through my test code (and fixing some errors in the tests), I've realized that they actually cover the case I was worried about and the comparison code effectively does treat numbers on opposite sides of zero the way I want it to.

Basically, the idea is that numbers should only be considered "nearly equal" if the difference between them could conceivably be the result of rounding or base conversion. And that is not really the case for -0.000001 and 0.000001 because there is still a huge range of smaller values between the two due to the scaling the format supports through the exponent.

One could argue that the code is still not right because making the epsilon a paramenter implies that if you choose it large enough, more or less any given values should be considered equal. But I suspect that requirement would cause a lot of problems.

[Stannieman]

Basically, the idea is that numbers should only be considered "nearly equal" if the difference between them could conceivably be the result of rounding or base conversion.

Ok wow you're absolutely right, I was so deep into looking at this that I completely forgot about why we actually need such method at all...

@brazzy what's your opinion when 1 of the values is exactly zero? I find this rather contradicting: assertFalse(nearlyEqual(0.00000001f, 0.0f)); assertTrue(nearlyEqual(Float.MIN_VALUE, 0));

IMO if one value is 0 then either:

• it should return false every time the other value is not zero (if such case cannot occur due to rounding just like opposite sides of zero then I think this is more correct).
• it should return true every time the diff < epsilon, every time so no exception for FLOAT_MIN.

I also have to say that I'm porting this to C++ and it does seem to behave more consistent for these edge cases (this contradiction does not exist there). On x86 Windows that is, I still need to verify on Linux and ARM.

[brazzy]

@Stannieman I wouldn't be so sure that rounding errors cannot change the value from 0 to Float.MIN_VALUE, which is the smallest nonzero value a float can hold - that's a very different case than the original -0.000001 and 0.000001 where there are billions of different values in between them.

I also find it a bit hard to believe that there would be much different behaviour between languages and architectures, I'd expect them all to follow the IEEE standard.