feat: Remove BigQuery NUMERIC precision/scale decorators for unsafe (i.e. sampled) mappings

gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.

Apache License 2.0

8 stars 2 forks source link

feat: Remove BigQuery NUMERIC precision/scale decorators for unsafe (i.e. sampled) mappings #179

Closed nj1973 closed 1 month ago

nj1973 commented 1 month ago

When we sample precision and scale for a decimal column the canonical mapping is marked as unsafe. This PR removes the precision and scale from the backend BigQuery column for these unsafe (sampled) columns. This just gives us a bit more wiggle room if the sampling missed some values.

User specified precision/scale and precision/scale taken from the source schema settings are unaffected. This is only about sampled columns.

nj1973 commented 1 month ago

@abb9979, after reflecting on this change I wonder if any sampled column identified as having a fractional element should automatically roll up to BIGNUMERIC rather than the limit of NUMERIC, any thoughts to add?

abb9979 commented 1 month ago

Could we have a scale threshold? For example, if sampling comes back with a max of 4dp, then it's probable that NUMERIC limits will not be breached. If it's pushing 7 or 8, say, then choosing NUMERIC would start to feel a bit more of a risk. We could auto-set the threshold to, 5, and have it configurable by the user.