MikeMcl / decimal.js

An arbitrary-precision Decimal type for JavaScript
http://mikemcl.github.io/decimal.js
MIT License
6.35k stars 480 forks source link

Irregular length of precision for at least .sub method #235

Closed SeaOfCrystalGreens closed 4 months ago

SeaOfCrystalGreens commented 4 months ago

.sub method seems to calculate to precision one less than precision set for Decimal object

Decimal.set({ precision: 3, rounding: 5 })

let n1 = new Decimal("1")
let n2 = new Decimal("2")
let n3 = new Decimal("3")

console.log(n1.div(n3).toString()) // 0.333
console.log(n2.sub(n1.div(n3)).toString()) // 1.67
console.log(n2.sub(n2.sub(n1.div(n3))).toString()) // 0.33

let a = n1.div(n3)
let b = n2.sub(n2.sub(n1.div(n3)))
console.log(a.equals(b)) //false

after changing line https://github.com/MikeMcl/decimal.js/blob/7f01abd83d5632005d5f97ddbea41e069ee69a74/decimal.js#L1296 to

pr = Ctor.precision + 1;

in https://github.com/MikeMcl/decimal.js/blob/7f01abd83d5632005d5f97ddbea41e069ee69a74/decimal.js#L1264

it seems to start calculating correctly

Decimal.set({ precision: 3, rounding: 5 })

let n1 = new Decimal("1")
let n2 = new Decimal("2")
let n3 = new Decimal("3")

console.log(n1.div(n3).toString()) // 0.333
console.log(n2.sub(n1.div(n3)).toString()) // 1.667
console.log(n2.sub(n2.sub(n1.div(n3))).toString()) // 0.333

let a = n1.div(n3)
let b = n2.sub(n2.sub(n1.div(n3)))
console.log(a.equals(b)) //true

Although maybe it is intended behavior.

This is ephemery fix. Intended only to highlight issue.

If unintended, what parts of code might cause shortening of precision? To what extend it might be in other parts of code?

MikeMcl commented 4 months ago

Sorry, which calculation do you think is incorrect?

You are not going to find any errors here in basic arithmetic operations. This is a reliable, time-tested library.

Note that precision refers to number of significant digits not decimal places.

SeaOfCrystalGreens commented 4 months ago

Comparison of : (2-(2-(1/3)) === (1/3)

Decimal.set({ precision: 3, rounding: 5 })

let n1 = new Decimal("1")
let n2 = new Decimal("2")
let n3 = new Decimal("3")

let a = n1.div(n3)
let b = n2.sub(n2.sub(n1.div(n3)))
console.log(a.equals(b))

Before code change, results in false. After equality holds.

It looks like it holds after decimal places are of equal length (erroneously increased precision).

If all works as intended then I do apologize for opening issue. I'll try to understand it more carefully.

MikeMcl commented 4 months ago

1 / 3 to a precision of 3 significant digits is 0.333.

2 - 0.333 is exactly 1.667, but rounded to a precision of 3 significant digits it is 1.67.

2 - 1.67 = 0.33 exactly.

0.333 != 0.33, as expected.

Normally, a much greater precision is used for calculations (for example, the default precision is 20) and then toSignificantDigits or toDecimalPlaces or toFixed etc. is used to round the result to the desired final precision.