Closed JohnAD closed 4 years ago
If both precision and scale are set, what should I do?
Opinions?
it is a mistake: look, if i have to switch to a DB field (8.2) there is an error if you put a 10.2 field it is a mistake: if the field leaves its definition there is an error
you just need an indicator to indicate that the field is too large ex: dcml (5.2) and the field made 8.3 / 8 is invalid the user must also be responsible, but must be given the means to know if the field is correct, prerequisite IBM
If I remember correctly in the IBM version, there is an out-of-definition cleaning function in case 5.3 for 8.2 it's done with truncate and round in your tests
I just redid the tests for the DB: postgresql it rounds automatically if scale 8.3 -> 8.2 rounding makes the mistake if 10.2 -> 8.2
The latest release v0.1.1 should have this all working.
If both precision
and scale
are passed on newDecimal128
then it:
scale
(that takes priority).@AS400JPLPC
If the precision of the final value is greater, then a ValueError
is raised.
So,newDecimal("43.2", precision=4, scale=2)
works and becomes "43.20".
But,newDecimal("43.2", precision=3, scale=2)
generates the error:
Error: unhandled exception: a precision of 3 was requested, but 43.20 has a precision of 4 at scale 2 [ValueError]
But,newDecimal("43.2", precision=5, scale=2)
also works because precision and scale are least-significance measures. (Which is why 0005.0
and 05.0
and 5.0
all have the same precision of 2 and scale of 1.)
This issue came from the discussion in issue #1
For each of the
newDecimal128([T])
procs, add two optional parameters:precision
to set the absolute precision of the decimal, regardless of what is seen in the incoming number. That is, this will artificially set the number of significant digits. Any value greater than 34 is treated as 34.scale
to set the precision of the decimal, regardless of what is seen in the incoming number, to enable a fixed number of decimal places of the fractional part. This can be a negative number, though that is not a common use of it.Example:
Any rounding is based on bankers rounding aka "round-up-to-even".