search

printfn / fend

Arbitrary-precision unit-aware calculator
https://printfn.github.io/fend
MIT License
694 stars 53 forks source link

Incorrect significant figure rounding #308

Open Markos-Th09 opened 2 months ago

Markos-Th09 commented 2 months ago

When rounding using significant figures, the number is truncated instead of being rounded. For example

12.9 to 2 sf

outputs 12 instead of 13

printfn commented 2 months ago

Thanks for the report, this would definitely be good to fix.

Markos-Th09 commented 2 months ago

also to <n> dp is also truncating but I don't what is the intended behavior of that

BoolPurist commented 2 months ago

@printfn Do you intent to fix it by yourself in the near future ?

If not. I would be willing to fix it. However, after looking into the code for that, I came to the conclusion that making sf rounding work will take me a while to get it right.

What are thoughts about it ?

Markos-Th09 commented 2 months ago

@printfn also should regular rounding be preferred over to truncation for the to dp rounding mode? If so, should a truncation rounding mode be introduced?

printfn commented 2 months ago

I think both to dp and to sf should round correctly. Supporting optional truncation would be nice too, but less important than getting the rounding right.