estuary / connectors

Connectors for capturing data from external data sources
Other
38 stars 10 forks source link

source-oracle-flashback: discover index-organised tables [0] #1681

Closed mdibaiee closed 1 week ago

mdibaiee commented 1 week ago

Description:

[0] index organised tables are in fact stored as multiple "overflow" tables, and when querying all_tables, we only see the overflow tables, but they refer to the index organised table through their IOT_NAME, which is the table we need to query.

also fixed an issue where if we could see a table but didn't have access to it, we would panic

Tested manually by discovering and capturing an index organised table, and on a database which has tables which the user cannot access.

Fixed the tests by isolating the tables used for testing in a specific schema so that the tests are more deterministic.

Fixed some issues with capturing empty tables and identifier quoting.

Refactored fetching of MAX(ROWID) into a PL/SQL so we don't have round-trips for each table

Oracle's numeric types have precision 38 by default. Updated the discovery and encoding logics to use string for numbers with precision higher than 18

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

(anything that might help someone review this PR)


This change is Reviewable