feature: Hashicorp Vault secrets support

[justyns]

Hello!

I currently use hashicorp vault for secret management. I'm trying out windmill.dev and appreciate that it has support for encrypted secrets, but I'd also like the ability to integrate directly with vault.

Could this be considered as a future enhancement? Or if it's already possible, do you have any suggestions on how to integrate the two? So far it looks like the scripts I run would have to support pulling secrets from Vault if I want to do it in the current version.

[rubenfiszel]

You're exactly right. Vault is a feature that we will tackle soon and there are still some open questions about how to do it properly. I will update this thread or create an issue for update.

[rubenfiszel]

I am reading more into the Vault documentation. It seems to me that the windmill integration should be quite lightweight and that windmill should only store a token for each user. Then the token should be set to be renewable and windmill can take care of doing the refresh like it does for other OAuth access_tokens: https://developer.hashicorp.com/vault/docs/auth/token

In the scripts, one would use a library such as https://www.npmjs.com/package/node-vault-client and pass their user token that is windmill stored.

So the only difficulty in this method is how to ease the process of setting up this initial renewable token for a user. I believe what we can suggest is to generate it via the vault cli and document the process. It would be nice to have an automatic way of doing it and i'm open to ideas.

[SpiderD555]

Can you consider making the solution more generic, so it can be quickly extended into other vaults quickly in the future ? I am thinking about Cyberark PAM Safe solution here. You can supply the Cyberark API with your client certificate, key, safe name and object/account name. In return you get the password. Hashicorp Vault is not the only solution out there.

[rubenfiszel]

@SpiderD555 It seems that is something that would already work within a script, if the client certificate, key safe, and object/account name is packaged as a resource.

Then the code within a script would be something like:

async function main() {
     let creds = getResource("f/folder/cyberark_creds")
     let pass = cyberarkApi.getPassword(...creds, <variable name>
}

The suggested idea above is simply to refresh the token stored by let creds = getResource("f/folder/cyberark_creds") in the case of a renewable token from vault. If Cyberark also had a renewable token system, yes we could include it.

If you have such a need, feel free to contact us on our discord so we can study your exact needs.

[justyns]

@rubenfiszel what do you think about having resources map to secrets inside of vault (or another secrets manager even)?

This is what I'm currently doing for a simple workflow that backs up postgres databases:

1. In a hashicorp nomad job for a windmill worker, use a consul-template template to write secrets to a .env file:

     template {
       data =<<EOF
   {{with secret "kv/data/nomad/pgbackup/homelab"}}
   {{ range $k, $v := .Data.data }}
   {{ $k }}={{ $v | toJSON }}
   {{end}}
   {{end}}
   EOF
       destination = "secrets/secrets-pgbackup-homelab.env"        
     }

2. In a windmill script:

   
   #!/bin/bash
   cluster="$1"
   echo "Loading secrets for cluster $cluster"
   source "/secrets-pgbackup-${cluster}.env"

docker run --rm --entrypoint /bin/sh \ -e "PGHOST=$PGHOST" \ -e "PGPASSWORD=$PGPASSWORD" \ -e "PGUSER=$PGUSER" \ -e "PGPORT=${PGPORT:-5432}" \ ...

3. In vault, there's a secret at `kv/nomad/pgbackup/homelab` with keys for PGHOST, PGPASSWORD, PGUSER.

This is working okay but basically means I need to get the secrets from vault onto the windmill worker and then use a worker group tag that includes a worker with the right secrets.  It doesn't scale very well because I have to pass the secrets to worker when the worker is created.   It also means those secrets are just sitting there on the worker.

I think my ideal scenario would be one of these:

1. Map a resource to a specific vault secret.  Resources can already be considered secrets, so one difference would be that windmill doesn't store the content of these resources.  Whenever it needs access to a vault secret resource, it'd connect to the vault api to retrieve it and provide it to the job.
2. Similar to what you mentioned about generating a token and providing it to the script.   This requires more work in the script itself, e.g. having to use the vault cli or a vault library.  I think windmill would still need the concepts of Vault policies though.  A user should be able to choose a vault policy for each job to limit the secrets that job has access to.  Windmill itself should expect to have a policy that lets it create child tokens with specific policies.

For some scripts, I'd actually prefer the 2nd method but for most things like my postgres backup example, I'd rather just use a docker image that has psql/pg_dump and not worry about where the secrets came from.

[rubenfiszel]

Thanks for this detailed response.

1.

For an integration where one can fetch the resources and variables directly from windmill, I would favor something done in the background by a scheduled job which would probably be a script that take in a token and a list of resources and fill them from time to time. A big advantage of this approach is that push complexity out of the backend and into arbitrary scripts which would be very easy to fit into anyone's setup. It also keep the concept of variables and resources more simple and hence easier to reason about.

A possibility for this setup is to encourage using infisical since an integration should done somewhat soon: https://github.com/Infisical/infisical/issues/679

Currently, variables can in some cases change their values in a just-in-time manner if they are tied to an oauth account. In which case, at expiry, they exchange a refresh token for a new access token. It wouldn't be too much of a stretch to make some custom logic for vault tokens where it would renew the vault token instead.

I think windmill would still need the concepts of Vault policies though. A user should be able to choose a vault policy for each job to limit the secrets that job has access to. Windmill itself should expect to have a policy that lets it create child tokens with specific policies.

I am not sure I follow this one. It seems to me that vault policies should stay a vault concept and that policies are attached to the token itself. If one want different policies, then one should have different tokens. On the other hand, windmill already has some concept of policies under the abstraction of groups and folders. It would then be possible to have the desired behavior by having different jobs use different tokens and by differentiating who can use those tokens using the already existing abstractions of windmill.

for most things like my postgres backup example, I'd rather just use a docker image that has psql/pg_dump and not worry about where the secrets came from.

I think if windmill were to provide some utility function that would take in a token and a vault secret and initialize the env, then the syntax would make it quite plesant to use.

[justyns]

1. This makes a lot of sense. I'll have to think about it some more, but I think the biggest caveat would be that the secrets would end up getting stored by both vault/infiscal and windmill. Windmill does encrypt secrets (iirc), but I assume it has to store a plaintext key somewhere to decrypt them. Vault, as an example, doesn't store the key needed to "unseal" its vault. So if someone were to get a copy of the data on disk, the vault data would remain encrypted without also having the separate unseal key.
2. What you said makes sense to me. I was thinking Windmill would need to know what policy to apply to which job, but you're right that it'd be simpler to just use different tokens as needed!

[rarango9]

Something else to consider is that vault has various login methods https://developer.hashicorp.com/vault/docs/auth.

For example my org uses the aws and approle methods and we even have multiple vault backends separated by regions and envs.

That being said we'd likely only use a singular vault path and backend to store any windmill used secrets with sub paths. But I'd wonder the performance impact if say I set 1 script to get a path and return a secret, then have that script be included as the first step in say 1000 flows with say 100 executions a day. Both from a windmill and vault perspective.

A more involved approach that I've seen other systems like Rundeck do is actually specify the secrets be stored in a separate backend like vault instead of using the default mechanism that windmill does. This makes it transparent from a user pov and enforces a standard for teams and orgs.

Similarly hashicorp consul has the ability to store non-secret data which similarly replace the variable storage mechanism.

But from both system it's basically login, get a session for x time, then write or read values from a path.

But overall great offering that looks like it's heading in a good direction thumbs 👍