search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.27k stars 4.15k forks source link

Plugins to anonymize data #2189

Open MChorfa opened 3 years ago

MChorfa commented 3 years ago

Eventually we will have to have a plugins integration mechanism that will allow to run binary like Inline WASM transforms, as https://vectorized.io do with 'RedPanda'. For example you want to anonymize your data at the entrance ... for example health data

┆Issue is synchronized with this Asana task by Unito

michel-tricot commented 3 years ago

Agreed 100%. We have that in our (long term) roadmap.

Is it something that you need today?

MChorfa commented 3 years ago

Agreed 100%. We have that in our (long term) roadmap.

Is it something that you need today?

Not at the moment

livefire2015 commented 2 years ago

Is this kind of plugin available today? We want to desensitize some columns in our source data during the sync. One option is custom DBT transformation. However, it is not supported in k8s deploy of Airbyte. So we are a little stuck here.

cgardens commented 2 years ago

@livefire2015 adding custom transformations is being worked on in Q4. That work is being tracked here.