search

P3-Core-Dev-Team / P3-Q-A

This repo is for tracking features
Other
1 stars 0 forks source link

ADS - Data Mismatch for Decimal Number - Neste #138

Open suseela22 opened 1 month ago

suseela22 commented 1 month ago

Describe the bug

Source data contains decimal value and its datatype is Number. While ingesting to ADS, decimal values are converted to whole number.

Screenshots

Source Data: image

Extracted Data: image

ADS Data: image

Additional context Build no : 2024.09.45542 Region : https://neste-dev.platform3solutions.com/ Client : Neste

Nadarajan-suyambu commented 1 month ago

@suseela22 In Oracle databases, data types like NUMBER are used, but during the ETL process, the NUMBER data type is being converted into DECIMAL. However, in the extracted metadata, the data type is still represented as NUMBER, which caused the issue.

kilam005 commented 1 month ago

Vetri or Logapriya, Please update the ETL metadata generation logic.

harsha-kotha commented 1 month ago

Please add RCA, this seems to be serious data issue can one of update? We need a clear understanding with documentation on how data types are handled from source/ETL/ADS. Data loss due to data type handling or any form is not acceptable!!!

kilam005 commented 1 month ago

@UmashankarPalanichamy Please provide the details requested by @harsha-kotha Pass the same to @maheswarareddyi and Priya for updating in Documentation appendix.

@UmashankarPalanichamy As informed earlier, this seems like a issue with metadata generation. Hope it is getting addressed in ETL. Either you or Vetri or Logapriyan kindly update the issue tracker with the status of the fix, expected release info.

Also detail out the fix details.

kilam005 commented 2 weeks ago

Code fix are ready. Demo to be given to @suseela22 before build generation. Expected to deliver build to testing team today.

@abishekkp this is high priority so checking and upate before 11am tomorrow.

abishekkp commented 2 weeks ago

@kilam005 _ETL Tested Version: 2024.10.45588P2 Server: 192.168.1.75

Oracle_Number Datatype mismatch issue is working as expected Expected datatype conversion from the implementation team image Source Database image Extracted Data from ETL image Ingested Data in ADS, Datatype is matched as expected image

suseela22 commented 2 weeks ago

@abishekkp Please mention the ADS version tested and also it is on-prem or serverless. Screenshot from ADS is the expected behavior but it is not displayed as such with the current build (2024.09.45542_P3) in Neste

@kilam005 It is decided to have zero for the whole number which has 'DOUBLE' datatype. As per the screenshot provided by @abishekkp , COLUMN2 has no zero to the whole number. As per understanding, no fix will be given from ADS side for this issue.

Please make sure the ADS/ETL testing team to test the version and the region used similar to the client environment before release.

Current build in Neste - 2024.09.45542_P3

Column which has Double datatype

image

For whole number .0 is present

image

abishekkp commented 2 weeks ago

@suseela22 This is the ADS server and version given for testing,

image

maheswarareddyi commented 1 week ago

An internal review is in progress.

kilam005 commented 1 day ago

ADS fixes: RCA: The result interpreation of double values when it cames from Trino and Athena were different. Athena retruned String output for the double value. Where as Trino gives correct represenation.

Fix Details: Code updates were made to receive and format the result value according to the expected data type. Now Trino and Athena side results will be uniform and double value will be represented corrected.

Input ADS output
2.0 2
1.65268 1.65268
-1.20 -1.2
kilam005 commented 1 day ago

Fix is merged and is expected to roll out in 2024.09.xxxx_P6 or P7