search

nathanpjones / DecimalMath

The math support for Decimal that Microsoft forgot and more.
MIT License
47 stars 13 forks source link

Log10 returning imprecise values #5

Closed finsterdexter closed 4 years ago

finsterdexter commented 4 years ago

For certain values, the Log10 function is returning some strange results.

Call Result Expected (same as Math.Log10)
DecimalEx.Log10(0.1m) -1.0000000000000000000000000001 -1
DecimalEx.Log10(0.01m) -1.999999999999999999999999999 -2
DecimalEx.Log10(0.001m) -2.9999999999999999999999999924 -3

Ideally, these results should be the same as the Math.Log10 function, as by definition, the Log10 of 0.1 is the integer -1.

nathanpjones commented 4 years ago

Good point. The way I calculate Log10 is by using the natural log divided by a constant value representing the natural log of 10. It simplifies calculations but it doesn't work quite as expected in situations like this.

I think what I'll have to do is add special cases for these but I'll have to think about how to do that in a performant way.

nathanpjones commented 4 years ago

@finsterdexter Thanks for the issue report. I've added special-case handling for all powers of 10 in v1.0.2.