search

thomasoa / andrews-deal

Automatically exported from code.google.com/p/andrews-deal
Other
10 stars 6 forks source link

Standard deviation tries to take square root of negative, returns -NaN #20

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Build deal
2. Run: deal -x issue.tcl

With issue.tcl attached.

Expect output of a floating point number very close to zero.

Instead get -NaN.

The problem is that the standard deviation is the square root of a floating
point number, and, due to standard rounding errors, the square root is
taken of a negative number, rather than 0.

The hack solution is to return 0 for any negative value.  I've implemented
that solution already.

Original issue reported on code.google.com by thomasoa on 29 Mar 2010 at 5:54

Attachments: