search

crossfilter / reductio

Reductio: Crossfilter grouping
Other
248 stars 42 forks source link

d3 format '.2f' sum precision #37

Closed mukherjeea closed 8 years ago

mukherjeea commented 8 years ago

Hi,

I have an attribute amount which I'm converting into d3.format('.2f'). In the reductio reducer the sum accessor is function(d) { return +d.amount; } I am expecting a sum amount that is also in '.2f' format but in one of the cases I see more than 2 decimal places. Object { =4, sum=396.54999999999995}

The summed amounts are as follows 346.5, 15.4, 19.25, 15.4

I was expecting a 396.55

What is the reason behind this behaviour?

Thanks

esjewett commented 8 years ago

d3.format returns strings. The sum accessor then converts these strings back into numbers to sum them and I assume they suffer from floating point precision errors. The solution is to use d3.format before displaying the results, not on your original data.

If you go into your Javascript console and run the following you'll see the same thing Reductio puts out:

+"346.5" + +"15.4" + +"19.25" + +"15.4"
mukherjeea commented 8 years ago

Thanks!