Closed jessealama closed 2 months ago
Could you give an example of the bug, and what the output is before and after this fix?
Could you give an example of the bug, and what the output is before and after this fix?
Here are a couple of examples of where we did the wrong thing before and how we do the right thing now:
ToRawFixed
resp. ToRawPrecision
AOs, which require a non-negative Intl mathematical value.AFAIK ToRawFixed and ToRawPrecision do the right thing with Decimal128 values.
The
FormatNumericToString
AO delegates its work to theToRawFixed
andToRawPrecision
AOs after checking the sign of its argument and, for negative arguments, taking the absolute value. There were a couple of bits where Decimal128 values were not getting properly handled inFormatNumericToString
; this PR fixes that. We also make clear that, when Decimal128 values are given to theToRaw{Fixed,Precision}
AOs, the solution to the equation to be solved is the cohort and quantum, so essentially no work needs to be done.In other words, given this change, checking the correctness of
FormatNumericToString
on decimal arguments is reduced to checking that theToRawFixed
andToRawPrecision
AOs correctly handle decimals. Those are correct, AFAICS. If there are any bugs, though, I'm happy to work on them in this PR.CC @sffc