search

conda-incubator / conda-project

Tool for encapsulating, running, and reproducing projects with Conda environments
https://conda-incubator.github.io/conda-project/
BSD 3-Clause "New" or "Revised" License
28 stars 11 forks source link

Secret support #74

Open AlbertDeFusco opened 1 year ago

AlbertDeFusco commented 1 year ago

I will be useful for variables to be overridden by docker/k8s style secret storage. The path to the runtime secrets folder can be provided as an env-var config to conda-project or a global cli argument (--secrets-dir).

If a mathching secret value is found it will override a value provided in the conda-project.yml file, but .env and shell variables will still retain the ability to override values.

I have identified cases where docker/k8s secrets are used to store the contents of a secret file, but my project requires the path to the file rather than the contents. For example this happens in Google services auth routines that need the path to a Json file with service account credetials. For these cases I propose a parameter that declares that if a secret is found for the variable its path is returned rather than its contents.

variables:
  FOO: bar
  CREDENTIALS_PATH:
    secret_path: True

In this example FOO has a default value that may get overridden by the contents of a secret at <secrets_dir>/FOO. CREDENTIALS_PATH will be set to the path <secrets_dir>/CREDENTIALS_PATH if found.

Finally, much of the functionality already implemented for variables and the secrets described above are implemented in pydatnic. There may be value in using its models for project variables.

https://docs.pydantic.dev/usage/settings/

mattkram commented 1 year ago

I like this idea a lot. But I think we will have to be very crisp about our definitions and assumptions here. Do we support multiple secret files, and if so how do we prioritize? There is also the likely use case of using a non-secret .env file instead of putting the values in conda-project.yml, and in that case what do we do, if anything?

There is a similar pattern for kubernetes manifests which is roughly similar and we may want to draw inspiration from: https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables

AlbertDeFusco commented 1 year ago

I was originally thinking that loading values from k8s/docker secret path (like /var/run/secrets/<variable-name>) would be in the following order

  1. Shell/process-level variable
  2. k8s/docker secret file
  3. .env file at the project root
  4. Value in the conda-project.yml file

I would say just supporting one directory for point 2 above is sufficient. Every secret gets its own file within /var/run/secret/ (or wherever its configured)

Now, if k8s-secret-as-env-var is enabled then the variable takes precedence anyway.