Problem when sigma is 0.

[GoogleCodeExporter]

(concerning step1.py and release 0.3.0) awh142857 in Issue 42 reports that:

"Also, in 'sigma' if the code is fed all equal values, and sigma = 0, the 
machine error can produce 
a negative number, resulting in sqrt(negative), which causes an error."

Original issue reported on code.google.com by d...@ravenbrook.com on 12 Feb 2010 at 8:44

[GoogleCodeExporter]

It would seem, due to floating point round-off, that "sos/count < mean*mean" 
could be true in the Python 
computation even though it is mathematically impossible.

Either we are being lucky, or the input data somehow guarantees it can't happen 
(for example, all the inputs 
are the (rounded) ratio of two fairly modest integers), or a proper floating 
point analysis would show it can't 
happen.

Since math.sqrt will raise a ValueError when fed a negative it will be totally 
obvious if it ever happens.

Let me know if you find a test case where it fails.

Original comment by d...@ravenbrook.com on 12 Feb 2010 at 9:22

[GoogleCodeExporter]

I've now woken up a bit more and found (by random trial) a test case that fails:

>>> from code import step1
>>> step1.sigma([1.4592859018312442e+63, 1.4592859018312442e+63, 
1.4592859018312442e+63])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "code/step1.py", line 459, in sigma
    return math.sqrt(sos / count - mean * mean)
ValueError: math domain error

I suspect that the nature of actual input data, ratios of modest integers (each 
less < 1e6, say) means that we 
never lose enough precision for it to be a problem in practice.

Will fix.

Original comment by d...@ravenbrook.com on 12 Feb 2010 at 9:29

• Changed state: Accepted

[GoogleCodeExporter]

This issue was closed by revision r291.

Original comment by d...@pobox.com on 12 Feb 2010 at 6:13

• Changed state: Fixed