precision loss in Fp mul

[9il]

Fp Multiplication result should be normalized for the extended 128-bit mantissa representation before round-to-even rounding. Otherwise, it may lose a bit of precision if both arguments are normalized and more bits if they weren't.

https://github.com/rust-lang/rust/blob/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/num/diy_float.rs#L24

Affects dec2float conversion. https://github.com/rust-lang/rust/blob/7c34d8d6629506a596215886e5fc4bb2b04b00ae/src/libcore/num/dec2flt/algorithm.rs#L156

[hanna-kruppe]

Does this actually lead to incorrect results in dec2float? I've long since paged this stuff out, but from a cursory glance both big_to_fp(f) and power_of_ten(e) seem to give normalized values. (Edit: nevermind this part, I didn't read your description carefully.)

[9il]

In very rare cases, yes (in theory). The stuff uses Clinger's algorithm and the bug violates the condition of how the slope variable is used. It was used in the context of actual 64-bit precision while the actual number is 63 bit.

[hanna-kruppe]

Right, I understand how this might cause problems in theory, but I was wondering if you could give a concrete example where it causes an "end-to-end" error. Both to ensure the theory actually holds up, and because it would be useful to have regression test.

[9il]

unfortunately, I don't have a test

[9il]

The bug report comes from the review, I found the code quite good to understand the Clinger's paper better (papers functional style is hard to get). Plus the python script to generate powers of 10 is awesome as well.