search

gluent / goe

GOE: a simple and flexible way to copy data from an Oracle Database to Google BigQuery.
Apache License 2.0
8 stars 2 forks source link

Change Query Import to support Oracle partitioned tables #39

Closed nj1973 closed 9 months ago

nj1973 commented 9 months ago

This will make testing much easier and should be pretty easy.

  1. Check that segment size used to check under small table threshold works across partitions
  2. Check that transport source SQL is same for Query Import as for Spark by Query

If it gets gnarly then abort. But worth doing if it's trivial.

nj1973 commented 9 months ago

After a quick review I believe only two changes are required:

  1. Remove the partition_type restriction from the is_query_import_available() function
  2. Change _query_import_to_local_fs() to use the same source query as used by Spark, from self._get_transport_row_source_query()

On first glance it seems easy enough.

I guess the risk is that we don't have good testing yet and it's a pretty far reaching change. Not pressing the "abort" button yet.