search

nf-core / ops

1 stars 2 forks source link

Pulumi ESC or GitHub Secrets #60

Open edmundmiller opened 1 month ago

edmundmiller commented 1 month ago 
          We can, the reason this one isn't is because I was struggling with the 1password Pulumi ESC integration, and I didn't realize you have to copy the **plain** service key into the environment file, and then it encrypts it in place for that specific environment file.

Anyways there's a few options:

  1. GitHub Secret
  2. Pulumi ESC
  3. Encrypting them in place like so(idk if you could run this for example or not)

This one doesn't really matter, because it's just to the nf-core-tf account. I can update it to use Pulumi ESC.

Leaning Pulumi ESC for now as:

  1. That gives us a better access management for the secrets.
  2. It also allows you to develop locally easily, instead of pushing to GitHub anytime you want to preview the changes.
  3. Already have 1Password integration setup with it (So you just pull the secrets in from there instead of copying them, which allows you to roll and update them all in one place)

We could do all of that with GitHub actions, and pass all of these things, but the secret management is already a complicated web, but it's working currently.

TL;DR something to explore, I'll update this one and move it to Pulumi ESC though.

_Originally posted by @edmundmiller in https://github.com/nf-core/ops/pull/59#discussion_r1686920841_

edmundmiller commented 1 month ago

Think the main focuses:

  1. Security of secrets
  2. Allowing anyone in nf-core to make a PR to this repo and see a preview, and if the PR gets merged, the infrastructure gets updated.
  3. Secrets are stored in 1Password
  4. Transparency of environment (Anyone on GitHub can see how we're setting variables, this is a drawback of Pulumi ESC)
  5. Ease of Local development (If someone has access to the 1password dev vault, they should be able to run stuff locally. This will probably matter less overtime as we get a clear workflow)