Open bendavies opened 3 years ago
Hi, what would be the difference between getScale()
and getPrecision()
?
hi @BenMorel
The PostgreSQL manual will say it better than me:
We use the following terms below: The precision of a numeric is the total count of significant digits in the whole number, that is, the number of digits to both sides of the decimal point. The scale of a numeric is the count of decimal digits in the fractional part, to the right of the decimal point. So the number 23.5141 has a precision of 6 and a scale of 4. Integers can be considered to have a scale of zero.
@BenMorel any interest? thanks!
@BenMorel
Working with precision is generally better than working with scale
(0.03/360)*8*30 = 0.02
Woking with scale of 10: 0.03/360 = 0.0000833333(3) 0.0000833333*8=0.0006666664(0) 0.0006666664*30=0.0199999920(0)
❌ 0.0199999920 == 0.02
Working with precision of 10: 0.03/360 = 0.00008333333333(3) 0.00008333333333*8=0.0006666666666(40) 0.0006666666666*30=0.01999999999(80)=0.02
✔0.02==0.02
@alexviar i'm not sure what your example shows but it's not precision of 10
Working with precision of 10: 0.03/360 = 0.00008333333333(3)
0.00008333333333(3)
has as precision of 16
@alexviar i'm not sure what your example shows but it's not precision of 10
Working with precision of 10: 0.03/360 = 0.00008333333333(3)
0.00008333333333(3)
has as precision of 16
It has a scale of 14 but a precision of 10. The precision refers to how many significant digits it has, not how many decimal places it has. https://www.logicbig.com/quick-info/programming/precision-and-scale.html
For example Big.js uses scale whereas Decimal.js uses precision
@alexviar i'm not sure what your example shows but it's not precision of 10
Working with precision of 10: 0.03/360 = 0.00008333333333(3)
0.00008333333333(3)
has as precision of 16
I did put in parentheses digits that was rounded, and underlined digits are the Periodo of decimal
ah right thanks, i forgot that leading zeros are not significant.
but also for storage purposes (postgres), 0 <= scale <= precision so, It has a scale of 14 but a precision of 10.
doesn't work, in practise.
anyway, the point of this of was to simple add a getPrecision()
method, similar to getScale
, nothing mnore
Hi @bendavies, sorry for the late reply. I'm not against adding a getPrecision()
method if it is useful to you, so please feel free to open a PR along with a few tests, in particular to clarify what happens for leading zeros.
@alexviar I'm not sure what the decimal period has to do with the precision? Anyway, you make me think that this would be a great addition to BigRational
: a way to output 10/3
as 3.3(3)
for example. Would you want to work on this?
I opened issue #74 to track the BigRational as decimal number with period separately.
@alexviar I'm not sure what the decimal period has to do with the precision?
I think I didn't explain myself well. I did put rounded decimals (and decimal period) in parentheses for illustrative purposes only. Precision is given by working with n significant digits instead of n decimal place:
Operation | Rounded to 10 significant digits | Rounded to 10 decimal places |
---|---|---|
0.03 | 0.03000000000 | 0.0300000000 |
360 | 360.0000000 | 360.0000000000 |
8 | 8.000000000 | 8.0000000000 |
30 | 30.00000000 | 8.0000000000 |
0.03/360 | 0.00008333333333 | 0.0000833333 |
(0.03/360)*8 | 0.0006666666666 | 0.0006666666 |
(0.03/360)*8*30 | 0.02000000000 | 0.0200000000 |
Now, if we take the rounded values to do subsequent operations, we have:
Operation | Rounded to 10 significant digits | Rounded to 10 decimal places |
---|---|---|
0.03 | 0.03000000000 | 0.0300000000 |
360 | 360.0000000 | 360.0000000000 |
8 | 8.000000000 | 8.0000000000 |
30 | 30.00000000 | 8.0000000000 |
0.03/360 | 0.00008333333333 | 0.0000833333 |
(0.03/360)*8 | 0.0006666666666 | 0.0006666664 |
(0.03/360)*8*30 | 0.02000000000 | 0.0199999920 |
Currently it can be circunvent by using BigRational
, but this workaround doesn't work with irrational opreations like square root for example ( Never mind, I had read an article with an example like this, but I can not found it)sqrt(2)*sqrt(2)
Hi,
Similar to
getScale
, would you be on favor of adding a methodgetPrecision
? I want to perform app side validation on BigDecimal before passing to Postgres numeric fields, andgetPrecision
would make that easier.Thanks!