search

gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.
Apache License 2.0
8 stars 2 forks source link

Add support for BigQuery parameterized data types #28

Closed nj1973 closed 10 months ago

nj1973 commented 10 months ago

GOE work was paused around the same time that BigQuery introduced parameterized data types for string types and precision/scale for numeric types.

A customer has experienced a problem with Informatica where if a length is not specified on the STRING data type for a table, it assumes length of 255. Columns declared in Oracle as VARCHAR2(<255) become STRING without length in BigQuery and Informatica assumes 255 and cuts off data it is subsequently adding to the table.

We should look to support:

  1. string(l)
  2. numeric(p,s)
  3. bignumeric(p,s)
  4. bytes(l)
nj1973 commented 10 months ago

In order to test this properly I've made a start on restructuring story tests.

I've created a new directory, gluentlib/tests/integration/scenarios which will eventually replace the old test_stories directory. For now only a single set of "scenarios" has been refactored.

gluentlib/tests/integration/scenarios/data_type_controls.py

Runs with this command:

pytest tests/integration/scenarios/data_type_controls.py