search

jchester / spc-kit

A toolkit for statistical process control using SQL
Apache License 2.0
58 stars 3 forks source link

Limit decimal precision #13

Closed jchester closed 3 months ago

jchester commented 3 months ago

We don't need dozens of significant digits after the point. For example this doozy from the Montgomery EWMA example:

74.0029202389164420846316248048718811340627492898768264678435292188558800966131466701064192937553180708356868392767078607750008710856