JohnAD / decimal128

A Decimal128 IEEE 754 2008 number library for the Nim programming language.
MIT License
25 stars 2 forks source link

Universally add optional "precision" and "scale" parameters to newDecimal128 #3

Closed JohnAD closed 4 years ago

JohnAD commented 4 years ago

This issue came from the discussion in issue #1

For each of the newDecimal128([T]) procs, add two optional parameters:

precision to set the absolute precision of the decimal, regardless of what is seen in the incoming number. That is, this will artificially set the number of significant digits. Any value greater than 34 is treated as 34.

scale to set the precision of the decimal, regardless of what is seen in the incoming number, to enable a fixed number of decimal places of the fractional part. This can be a negative number, though that is not a common use of it.

Example:

assert $newDecimal128("43.2") == "43.2"
assert newDecimal128("43.2").getPrecision == 3
assert newDecimal128("43.2").getScale == 1

assert $newDecimal128("43.2", precision=5) == "43.200"
assert newDecimal128("43.2", precision=5).getPrecision == 5
assert newDecimal128("43.2", precision=5).getScale == 3

assert $newDecimal128("43.2", scale=2) == "43.20"
assert newDecimal128("43.2", scale=2).getPrecision == 4
assert newDecimal128("43.2", scale=2).getScale == 2

assert $newDecimal128("43.2", scale=-1) == "4E+1"
assert newDecimal128("43.2", scale=-1).getPrecision == 1
assert newDecimal128("43.2", scale=-1).getScale == -1

Any rounding is based on bankers rounding aka "round-up-to-even".

JohnAD commented 4 years ago

If both precision and scale are set, what should I do?

Opinions?

AS400JPLPC commented 4 years ago

it is a mistake: look, if i have to switch to a DB field (8.2) there is an error if you put a 10.2 field it is a mistake: if the field leaves its definition there is an error

you just need an indicator to indicate that the field is too large ex: dcml (5.2) and the field made 8.3 / 8 is invalid the user must also be responsible, but must be given the means to know if the field is correct, prerequisite IBM

If I remember correctly in the IBM version, there is an out-of-definition cleaning function in case 5.3 for 8.2 it's done with truncate and round in your tests

I just redid the tests for the DB: postgresql it rounds automatically if scale 8.3 -> 8.2 rounding makes the mistake if 10.2 -> 8.2

JohnAD commented 4 years ago

The latest release v0.1.1 should have this all working.

If both precision and scale are passed on newDecimal128 then it:

  1. Sets the value to the scale (that takes priority).
  2. Checks to see if the resulting precision is the same (or less) than the precision.

@AS400JPLPC

If the precision of the final value is greater, then a ValueError is raised.

So,newDecimal("43.2", precision=4, scale=2) works and becomes "43.20".

But,newDecimal("43.2", precision=3, scale=2) generates the error:

Error: unhandled exception: a precision of 3 was requested, but 43.20 has a precision of 4 at scale 2 [ValueError]

But,newDecimal("43.2", precision=5, scale=2) also works because precision and scale are least-significance measures. (Which is why 0005.0 and 05.0 and 5.0 all have the same precision of 2 and scale of 1.)