cmpute
commented
10 months ago I'm not very familiar with this standard, but it seems that the precision of that function is defined in a "fixed point" manner, i.e. it's always defined with regard to the least significant digit in the integral part. However, the "precision" in dashu is defined as the "effective digits" in scientific representation. For example 8.5e3 has a precision of 2, while 8.452e3 has a precision of 4. As I understand in the xpath standard, 8.5e3 has a precision of 2 (because the least significant digit in on the 10^2 position), while 8.452e3 has a precision of 0 (because the LSD in on the 10^0 position).
Therefore if you want to achieve the behavior of the function defined in that standard, you might need to calculate the precision based on the dashu precision of the number. Essentially xpath precision = -dashu exponent.
Context: https://www.w3.org/TR/xpath-functions-31/#func-round
See also #36
This defines a precision different from Dashu's. I think for precision indicators in the spec >= 0, turning it into a Dashu precision requires adding 1.
But the spec also defines negative precision indicators. So you can do:
fn:round(8452, -2) returns 8500.I can implement this by doing
10.pow(precision.abs()), and dividing by this before application of the round, and then after rounding multiplying by that again. But I was wondering whether there is a more clever way to do this with Dashu, given Dashu's general support for precision already.