search

K2InformaticsGmbH / oranif

Oracle OCI driver using dirty NIF
Apache License 2.0
4 stars 2 forks source link

`data_get` precision loss for numbers with many digits #126

Open acautin opened 5 years ago

acautin commented 5 years ago

Using data_get after executing the following query results in loss of precision: SELECT 2.700000000000005 FROM dual Is returned as 2.70000000000001

I am not really convinced we are properly using the full precision of erlang as this is getting rounded even before the erlang precision is a problem.

Also maybe related: https://github.com/oracle/odpi/issues/4

acautin commented 5 years ago

Question to odpi, ref: https://github.com/oracle/odpi/issues/100#issuecomment-518266931

stoch commented 5 years ago

@acautin Just a small remark:

Assuming that tools like TOAD are the norm, we have to be careful:

SELECT to_char(2.700000000000005) FROM dual gives TO_CHAR(2.700000000000005)

2.700000000000005
1 row selected.

whereas

SELECT 2.700000000000005 FROM dual gives: 2.700000000000005

   2.70000000

1 row selected.

The inherent precision is conserved but the default rendering in TOAD (probably Oracle) does some default rounding to 8 digits.

SELECT 2.700100021200305 FROM dual gives 2.700100021200305

   2.70010002

1 row selected.

c-bik commented 5 years ago

@acautin is https://github.com/oracle/odpi/issues/100#issuecomment-518328988 good enough to close this issue?

acautin commented 5 years ago

@c-bik haven't tried it yet, as soon as I try it out (after finishing with edits) will confirm and close.