search

ConduitIO / conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
https://conduit.io
Apache License 2.0
399 stars 49 forks source link

Encryption of secrets #137

Open hariso opened 2 years ago

hariso commented 2 years ago

Feature description

We have connectors which require secrets of different kinds to be able to function properly (e.g. the S3 connector requires the secret key, the Kafka connector will require certificates). We need to be able to store them securely with Conduit.

neovintage commented 2 years ago

What's the state of the art for secrets management? There's services like Hashicorp Vault and Doppler. I'm asking this question because I don't want to invent something from scratch when best practices exist.

hariso commented 2 years ago

@neovintage I completely agree with you that we shouldn't be reinventing the wheel here, especially when it comes to security-related topics. I didn't want to go into implementation details in the ticket itself, but it appears to me we have three options:

  1. Make it possible to fetch the secrets from a service a Conduit user controls/defines (e.g. AWS Secrets Manager)
  2. Embed a library for securely managing the secrets
  3. Integrate Conduit with an open-source secrets manager, which would always be deployed together with Conduit.

Option 3 is probably least desirable, since it will make the deployment more complex, and it will be much more difficult (impossible) to provide a single binary.

lyuboxa commented 2 years ago

I want to suggest that secrets encryption at rest is a priority here, because that is what compliance will look for. Without TLS all these secrets are likely flying through bare wire and can be extracted by MITM attacks.

With that:

  1. Secrets encryption at rest
  2. Secrets interface which will allow secrets to be fetched/leased from services (hashicorp, ssm etc)

The latter option is long winded, there has been a lot of chatter about JWT and OIDC for distributed credentials.

lovromazgon commented 2 years ago

We agreed to start with the following functionality:

  1. Conduit will a CLI flag to configure a secret that will be used for encrypting data in the DB.
  2. If the CLI flag is not provided, Conduit will generate a key and store it in a file alongside the binary (unless it already exists). Additionally, Conduit will log a warning that a secret was generated and that the user should consider re-encrypting the data and providing their own key.

In the future the user will be able to configure this secret through a config file (see https://github.com/ConduitIO/conduit/issues/31).