keeps / dbptk-developer

DBPTK Developer - library and command-line tool for execution of database preservation actions
http://www.database-preservation.com
GNU Lesser General Public License v3.0
43 stars 19 forks source link

oracle module converts NUMBER into NUMERIC(10) #496

Open Laurira opened 2 years ago

Laurira commented 2 years ago

We discovered that Oracle default field type "NUMBER" is not integer... It is actually a number with maximum number of digits and scale.

So at the moment dbptk converted Oracle NUMBER -> NUMERIC(10) So siard is created but it can not be loaded into dbptk for browsing as it tries to insert 14.68 into integer field...

Please see the documentation of Oracle and try to repair the application.

kuldaraas commented 2 years ago

To add from Oracle resources (https://www.oracletutorial.com/oracle-basics/oracle-number-data-type/):

"The Oracle NUMBER data type has precision and scale.

The precision is the number of digits in a number. It ranges from 1 to 38. The scale is the number of digits to the right of the decimal point in a number. It ranges from -84 to 127."

"Both precision and scale are in decimal digits and optional. If you skip the precision and scale, Oracle uses the maximum range and precision for the number."

As such, we understand that the default "number" where precision and scale are skipped is "number(38,127)"?

Laurira commented 2 years ago

What makes that conversion even more weird is that some of the Oracle NUMBER fields are converted into NUMERIC(10,2). So why dbptk decides for some of the NUMBER fields that these should be integer and some of these should be decimal?

Our database statistics: Oracle NUMBER fields: 632 Oracle NUMBER fields converted into NUMERIC(10): 412 Oracle NUMBER fields converted into NUMERIC(1): 112 Oracle NUMBER fields converted into NUMERIC(38): 46 Oracle NUMBER fields converted into NUMERIC(10,2): 21 Oracle NUMBER fields converted into NUMERIC(20): 15 Oracle NUMBER fields converted into NUMERIC(5): 14 Oracle NUMBER fields converted into NUMERIC(12,4): 6 Oracle NUMBER fields converted into NUMERIC(14,4): 4 Oracle NUMBER fields converted into NUMERIC(11): 2

So I understand DBPTK is trying to decide what data type to select on analyzing the content itself?

hmiguim commented 2 years ago

I'll investigate this kind of behaviour. Can you provide the Oracle version?

Laurira commented 2 years ago

Oracle Oracle Database 10g Release 10.2.0.5.0 - 64bit Production With the Real Application Clusters option

hmiguim commented 2 years ago

Which version of DBPTK developer are you using?

Laurira commented 2 years ago
DBPTK Developer (version 2.9.10)
hmiguim commented 2 years ago

I was unable to reproduce the bug, however I cannot use the same database. I don't have access to a Oracle DB 10g version. @Laurira is there a way to provide me access to an Oracle DB 10g?

Laurira commented 2 years ago

I do not have the access either. The error came up when one of our agencies tried to use the tool.

Can you just view the code and see that at the moment Oracle NUMBER is converted to integer which is not the correct behavior. It should be converted to some non-integer format with digits.

hmiguim commented 2 years ago

And can I have access to the log and report? You can send directly to my email. So I can analyze it and try to understand why those conversions happened.

Either way the actual strategy implement in cases which no metadata information about the column size is provided by the JDBC driver we are are assuming (wrongly) the number is an integer. Although for what I have tested there is no precision loss during the extraction process.

Laurira commented 2 years ago

I do not have your e-mail, but I have skype, so please contact me there.

Laurira commented 1 year ago

Hi,

We have the same issue with one of our other agencies.

Maybe I was not clear enough that this error occurs during the solr indexing. The creation of SIARD finishes with no errors but when you upload it to the DBPTK Enterprise and click "Browse" then it will be stuck in some point.

DBPTK Developer (version 2.10.1) Oracle Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production Version 19.17.0.0.0

I sent the log and important part from siard file to your (Miguel Guimarães) e-mail with heading "Issue 496".

Laurira commented 4 months ago

Hi, The same issue occurs again. Can you please solve the issue. You can get Oracle 19c for testing from https://www.oracle.com/database/technologies/oracle-database-software-downloads.html#db_free

Thanks, Lauri