Floating point calculation errors

[pbowen-oc]

We're seeing unexpected results with decimal values. This was first observed due to incorrect results from the mod operator, but further investigation reveals that the underlying issue is that decimal arithmetic is producing incorrect results.

Note that this can be observed with both the old XPath evaluator and the new one.

Typical examples are: 10.04 10 or 10.04 100

10.13 10 or 10.13 100

The results from these are expected to be integers or numbers with exactly 1 decimal digit.

Test form: decimal test.xml.txt

[MartijnR]

Oh gee, that looks like some Javascript quirkiness (type 10.13 * 10 in javascript console). Same in Chrome and Firefox though.

[MartijnR]

https://www.exploringbinary.com/floating-point-questions-are-endless-on-stackoverflow-com/

[MartijnR]

• [x] pick @alxndrsn brain about any ways of dealing with this

[alxndrsn]

This looks like normal floating point behaviour to me. In XPath 1.0, all numbers are floating points:

A number represents a floating-point number. -- https://www.w3.org/TR/1999/REC-xpath-19991116/#numbers

Javascript is similar:

The JavaScript Number type is a double-precision 64-bit binary format IEEE 754 value, like double in Java or C#. This means it can represent fractional values, but there are some limits to what it can store.

• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number

With the common browser built-in XPath evaluators you'll see similar behaviour:

Firefox:

Chrome:

Simple workaround would be to use round(), e.g. if you know the number of decimal places you want to display:

round(0.1 + 0.1 + 0.1, 1)

And for a dynamic number of decimal places, something like:

round(0.1 + 0.1 + 0.1, string-length(substring-after('0.x', '.')))

'0.x' could be replaced with a path to a reference node, e.g. /some-user-input, and should also work with proper number values, e.g. string-length(substring-after(12.345, '.')) returns 3.

Be aware though that you're still using floating points under the hood, and so rounding errors are always a possibility.

[MartijnR]

Thanks Alex!