Loss of precision when converting types

[camjo]

Converting a type to another type can affect the precision of the value.

e.g.

Grams(5).in(Kilograms)
// squants.mass.Mass = 0.005 kg

Grams(5.1).in(Kilograms)
// squants.mass.Mass = 0.0050999999999999995 kg
(rather than 0.0051 kg)

The example simply changes the units and doesn't seem like it should affect the precision. If the same calculation was done directly with BigDecimal you would not see this loss of precision.

e.g.

BigDecimal(5.1) / 1000
// 0.0051

[camjo]

After a bit of digging it looks like this method might be the culprit:

protected def converterTo: Double ⇒ Double = value ⇒ value / conversionFactor

in the trait UnitConverter.

A possible fix could be to adjust this to the following:

protected def converterTo: Double ⇒ Double = value ⇒ (value / BigDecimal(conversionFactor)).toDouble

This would fix it for all implementations however it would have unnecessary overhead of creating the BigDecimal object for types where Double is sufficient.

Another option might be to overwrite UnitConverter with a more accurate implementation with classes where it matters?

[derekmorr]

This is a known issue. Currently, because almost all of squants is based on Double you have all the normal issues with floating-point math. For version 2, we plan to redesign the library around a numeric type class, so users can use BigDecimal or the spire types, or any other type they want. This is a lot of work, and will break compatibility with existing users, so we're trying to take our time with it.

[camjo]

Ok makes sense. Any rough idea on timelines for 2.x at this point?

[derekmorr]

Not currently. We just released 1.0 in December 2016. We've already identified a few issues we'd like to change but can't because of backwards-compatibility.