search

airbytehq / terraform-provider-airbyte

Terraform Provider for Airbyte API
https://api.airbyte.com/
MIT License
44 stars 19 forks source link

Ability to exclude columns from replication #69

Open nurikk-sa opened 9 months ago

nurikk-sa commented 9 months ago

Hello! Thanks for a great software! TF provider is really good addition to airbyte.

I'm looking for a way to exclude column from replication. Currently terraform provider doesn't support field list/exclusion. Example reason: remove sensitive columns from replica, like emails/passowrds

hirads commented 8 months ago

I plus one this! Thanks for the really nice tool, I was so happy to see specially the Terraform provider and everything else looks amazing, but this little use-case which is missing is actually a huge deal-breaker for our use-case. It would be amazing if we could directly select columns via Terraform, otherwise the manual selection via UI is reset every time you configure the connection and you have to redo the selection for each and every stream (eg. when adding a new stream to the connection). I couldn't find any workarounds for it, there is no way to select them on the source (in my case redshift) and I've even tried creating a view on top of my table to contain only the columns needed but the streams don't include the views in the stream.

dantonbertuol commented 7 months ago

That would be excellent!

Having a way to use terraform with airbyte in itself is incredible, but with this functionality it would be amazing.

Anxiously awaiting a response.

dis-sid commented 7 months ago

Not having column selection here prevents us (and I believe everyone who needs this feature) from using this TF provider for connections. It's still great to have source and destinations defined once and versioned as TF configurations :+1:

rGaillard commented 6 months ago

+1 this feature is really missing :(

kamegg13 commented 6 months ago

It would be great if this feature will be available soon. Terraform is not used to his full potential. Getting streams on a connection is good, but not enough. Airbyte UI can be a source of error if we misclick on the webpage.

Roman77450 commented 6 months ago

+1 We also need this feature. Today we are waiting for it to be available to use airbyte

lucasfcosta commented 3 months ago

++ for this! Much needed

sowla commented 3 months ago

As of v0.63.4, it's possible to enable or disable fields within each stream via the API ( https://github.com/airbytehq/airbyte/issues/35798#issuecomment-2201583381) which I guess is a prerequisite for this issue to get tackled :)

pankaj-lmg commented 2 months ago

The support is added now. But there is an open issue https://github.com/airbytehq/terraform-provider-airbyte/issues/124