statistics libraries have unexpected precision behavior

usethesource / rascal

The implementation of the Rascal meta-programming language (including interpreter, type checker, parser generator, compiler and JVM based run-time system)

http://www.rascal-mpl.org

Other

402 stars 77 forks source link

statistics libraries have unexpected precision behavior #669

Open jurgenvinju opened 9 years ago

jurgenvinju commented 9 years ago

For example from analysis::statistics::Descriptive:

rascal> median([1.0, 0.99, 0.7])
num: 0.9900000095367432

This is probably due to an internal mapping to Java's floats or doubles.

DavyLandman commented 9 years ago

isn't this the same as #641 ?

jurgenvinju commented 9 years ago

No but it is an effect of the same core issue we have with reusing Apache's libraries: marshaling between different slightly incompatible data-types.

DavyLandman commented 9 years ago

my bad, that's what I meant, same problem, different manifestation.

jurgenvinju commented 9 years ago

yes, same cause different problems :-P