gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.
Apache License 2.0
8 stars 2 forks source link

Decouple table creation and data loading #139

Closed nj1973 closed 2 months ago

nj1973 commented 6 months ago

Current behaviour prevents the user from first time offload of data to an existing table. We can append to a partitioned table but not a non-partitioned one.

We should support offloading to a table if it is empty without need for extra options.

If the table is empty and has no metadata then we can just allow the Offload to continue, after first validating column names and supported data type combinations. If metadata exists then something is amiss and we should supporting --reuse-backend-table as a variation on --reset-backend-table.

Some customers do not allow tools to create tables, object creation must be via existing processes. Therefore we should also provide a --ddl-file option to generate DDL to create the backend table and output it to a file and stop there without copying data. The use of this option will deactivate any -x flag, processing is verification only.