search

MeltanoLabs / tap-snowflake

Other
4 stars 9 forks source link

Streamline schema name ignore for 'information_schema' #6

Open aaronsteers opened 1 year ago

aaronsteers commented 1 year ago

Proposal would be to add something like excluded_schema_names to streamline the operation below, which excludes information_schema from catalog generation:

https://github.com/MeltanoLabs/tap-snowflake/blob/5d7aef29e8178ea2b47bd44c13578fb9fe979902/tap_snowflake/client.py#L52-L77

Should probably be an SDK feature so that target developers can more easily implement.

aaronsteers commented 1 year ago

@kgpayne - Can you log or link to an issue in the SDK? Thanks!

aaronsteers commented 1 year ago

@kgpayne - Possible generic solve here:

In this approach, the setting would need to be ignore: ['information_schema-*']. We could make that the default setting, and if someone implemented a different rule, they'd provide the full ignore: ['information_schema-*', 'my-extra-ignores',...]. Or to get the information schema tables included in the sync, you could set the empty set: ignore: []