Can't convert nautical miles to meters correctly

[r-hede]

I'm trying to set up a converter between the units m and Nm (meters -> nautical miles) like this:

console.log(Qty("55560 m").to("naut-mile").scalar); // 30.000000000000007

A nautical mile is equal to 1852 meters, so the following calculation must be performed during this conversion:

55560 / 1852 = 30

So there shouldn't be so many zeros after the decimal point.

However, this is not the case with the following conversion, for example:

console.log(Qty("53708 m").to("naut-mile").scalar); // 29

The problem only occurs with certain values.

Is this problem already known?

[glepretre]

This is a JavaScript issue : https://stackoverflow.com/questions/47634766/algorithm-to-correct-the-floating-point-precision-error-in-javascript

You can use toPrec method or custom formatting to prevent it :wink: