tools4j / decimal4j

Java library for fast fixed-point arithmetic based on longs with support for up to 18 decimal places.
decimal4j.org
MIT License
156 stars 17 forks source link

Conversion from float to long rounds 0.99999992f incorrectly to 2 with rounding mode CEILING or UP #8

Closed terzerm closed 8 years ago

terzerm commented 8 years ago

Failure cases: 1) Positive Float values rounded incorrectly: 99999992f to 99999997f Double values rounded incorrectly: 0.99999999999999984 to 0.99999999999999994 Rounding modes: CEILING or UP Expected: 1 Actual: 2

2) Negative Float values rounded incorrectly: -99999992f to -99999997f Double values rounded incorrectly: -0.99999999999999984 to -0.99999999999999994 Rounding modes: FLOOR or UP Expected: -1 Actual: -2

Remarks: The reason is that values between 0 and 1 have higher fraction precision than values between 1 and 2. The code adds 1.0 to the value x, and double arithmetic uses HALF_EVEN rounding by default. This leads to unexpected results for edge cases when HALF_EVEN triggers rounding UP to 2.

Proposed FIX: The FIX is to cast the float/double value x to a int/long before the addition of 1.0: Current: x + 1.0 Fixed: (long)x + 1.0

Bug occurs in code copied from Guava. An issue has been raised and a pull request submitted to the guava project:

terzerm commented 8 years ago

Fixed as of commit 6f13496f9cac96ff99ca3f20b03e1f667e226946.