search

cmpute / dashu

A library set of arbitrary precision numbers implemented in Rust.
Apache License 2.0
74 stars 9 forks source link

incorrect precision in `RBig::to_float` #45

Closed Shoeboxam closed 7 months ago

Shoeboxam commented 7 months ago

The following assertion panics:

assert_eq!(RBig::ZERO.to_float::<Up, 2>(20).value().precision(), 20);

In Repr::to_float there is an edge-case that returns FBig::ZERO (with precision zero) without updating the precision.

EDIT: This seems to be a common issue throughout the library: precision is not preserved when any edge-cases are encountered (see ln_internal, exp, powf). This makes me think the behavior might be intentional? If so, feel free to disregard.

cmpute commented 7 months ago

Sorry that I missed this ticket. I think this is a bug, whenever a precision is explicitly defined in the arguments, the output should always have that precision. However, the value FBig::ZERO is intentionally designed to have a precision zero (unlimited precision), because it's a exact value and it's exposed as a common constant.

I haven't checked this principle through out the project, but I will try to make it consistent when I complete the guide of the project, and these corner cases will be fixed! Thanks for your report!

cmpute commented 7 months ago

The fix has been pushed. But I think it's too small a change to create a new version. Please kindly wait for the next release :)

Shoeboxam commented 7 months ago

Agreed, and there's easy workarounds. Thanks!