search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.29k stars 3.95k forks source link

External Secrets Store Support: HashiCorp Vault #10519

Open cgardens opened 2 years ago

cgardens commented 2 years ago

epic: https://github.com/airbytehq/airbyte/issues/837

Tell us about the problem you're trying to solve

Currently, Airbyte only integrates with one external secrets store: GCP Secrets Manager. Users who already use HashiCorp Vault would like to be able to have Airbyte integrate. with it.

Describe the solution you’d like

Add HashiCorp Vault as an option for external secrets.

Describe the alternative you’ve considered or used

A clear and concise description of any alternative solutions or features you've considered or are using today. We already have a SecretsPersistence interface to work with. The existing GCP Secrets Manager implementation can be found in the class GoogleSecretManagerPersistence. Basically we need to:

  1. Implement SecretsPersistence for HashiCorp Vault with tests similar to what already exists.
  2. Allow setting appropriate env variable to configure it and pipe through that configuration into the app.

In terms of external contribution part 1 should be very doable. We would be happy to help on part 2 once tests are passing for part 1.

cgardens commented 1 year ago

@benmoriceau we have VaultSecretPersistence implemented already. What is the gap between what we have now and saying this is usable for OSS?

benmoriceau commented 1 year ago

@cgardens A community user submitted the Vault persistence. So it is usable in OSS at the moment. That being say we currently only support a token based authentication and not all the supported auth by Vault.

Also the vault connection lib that we are using is not supported anymore so if we were to invest more on this we should probably implement our own lib since there was no other java lib available last time I checked.

seanglynn-thrive commented 1 year ago

Hey @benmoriceau @cgardens I'm wondering if you guys have faced any issues while connecting to a vault location to persist secrets, that requires a particular SSL cert? We're currently blocked from using the Vault secretstore as a result of this :( See: https://github.com/airbytehq/airbyte/issues/22065