fslaborg / FSharp.Stats

statistical testing, linear algebra, machine learning, fitting and signal processing in F#
https://fslab.org/FSharp.Stats/
Other
205 stars 54 forks source link

[Discussion] Consider using checked operators or use consistent overflow prevention #214

Open kMutagene opened 2 years ago

kMutagene commented 2 years ago

the built in arithmetic operators are unchecked, meaning they can produce overflows (numbers larger/smaller than the max/min value that fits into the target data structure)

Example:

178! is

6235135397241908741680674639927586558582878361231153877729215864681715113389074517106770270394081618113953882713761626619212667044889245663364667862568799843977208366195524719896472807475423880975816549024219691598995799655217698833954618998814300242650289839922492033762220111298560000000000000000000000000000000000000000000

this does not fit into float, and the result of SpecialFunctions.Factorial.factorial is infinity. This can produce unexpected results in functions that use this result for further arithmetic.

I would suggest to either use checked arithmetic operators, which throw an error when an overflow happens (https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/symbol-and-operator-reference/arithmetic-operators#summary-of-unary-arithmetic-operators), or discuss another appropriate way of handling these issues.

The fact that this was not really encountered or reported before leads me to the assumption that it should be safe to use checked operators.