gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.
Apache License 2.0
8 stars 2 forks source link

Improvement to sampled frontend decimals on a BigQuery backend #174

Closed nj1973 closed 1 month ago

nj1973 commented 1 month ago

When faced with an Oracle NUMBER(*) column we sample approx. 1GB of data in the source database to estimate the correct precision/scale to use in the backend.

On BigQuery we should record these as an unsafe sample and roll the precision and scale up to the max of NUMERIC/BIGNUMERIC. We used to do that but at some point in the recent past we enabled support for copying the exact precision/scale to BigQuery, we should not do that for unsafe/sampled values.