GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
Apache License 2.0
408 stars 119 forks source link

Validate custom query row throws exception for Oracle FLOAT(126) column #1302

Closed nj1973 closed 1 month ago

nj1973 commented 1 month ago

Test table:

create table dvt_test.tab_float_126 (id number(5), col_flt_126 float(126));

Standard table validation is successful:

data-validation validate row --source-conn ora --target-conn ora \
--tables-list "dvt_test.tab_float_126" --primary-keys id --concat="*"

But custom-query validation throws an exception:

data-validation validate custom-query row --source-conn ora_local --target-conn ora_local \
-sq "select id,col_flt_126 from dvt_test.tab_float_126" \
-tq "select id,col_flt_126 from dvt_test.tab_float_126" \
-pk id --hash="*"
...
    raise ValueError('Decimal type scale cannot be negative')
ValueError: Decimal type scale cannot be negative
nj1973 commented 1 month ago

After adding a test for this I've discovered that we also have an issue with PostgreSQL numeric type in custom-query validations. For test column "col_num_10_2" we are losing precision and scale even though Ibis initially detects the data type string as "numeric(10,2)".