Support for secret stores other than Hashicorp Vault

[reinkrul]

E.g. Azure KeyVault. Maybe there's a common standard?

[stevenvegt]

The Azure integration looks doable, but we need people to help with testing and writing documentation. https://learn.microsoft.com/en-us/azure/key-vault/secrets/quick-create-go

[HansSDBGroep]

For Azure key vault I can help. Maybe we can discuss it on the hackaton.

[reinkrul]

As stated in https://github.com/nuts-foundation/nuts-node/issues/1488 Hashicorp Vault supports mounting a directory so applications can simply read from it as if it were a regular file on disk, Azure KeyVault has the same feature. Question remains whether you can write keys through this route, this has to be found out.

[reinkrul]

I couldn't find a definite answer online, but I'm almost sure a mounted Azure KeyVault won't allow you to write to the KeyVault through the filesystem. There's a Go client library for Azure services (including keyVault), but judging the examples (https://github.com/Azure-Samples/azure-sdk-for-go-samples/tree/main/sdk/resourcemanager/keyvault), I'm not sure if we'd want to introduce this complexity into the Nuts Node (and including another, quite large, dependency). Since, how many secret engine clients are we going to have to support (what about GCP/AWS?).

Ideally there would be a standardized API that all secret engines support, but I'm not sure about that. Alternative route would be to develop a microservice for each supported secret engine (Hashicorp Vault/Azure KeyVault/GCP/AWS/etc), which can be deployed as (k8s) sidecar to the Nuts Node, through which all crypto operations are routed. Then the Hashicorp Vault integration would be extracted into a microservice as well.

This route would have the benefit of easier community contributed adapters for their specific secrets engine (we don't have to support every imaginable secrets engine).

We'd need some kind of protocol (preferably standardized) for the Nuts Node to communciate with the sidecars. Probably a simple REST KV-API would fit.

Or have a sidecar watch the crypto datadir and sync the secrets to the external secrets engine (have to make sure data is encrypted at rest, though).

[pascaldekloe]

On a previous assignment I had the pleasure of working with those specific parts of Azure. Most API calls are under development, poorly documented, tech-stack heavy and always deviating from standards in some minor way. Microsoft tells its (paying) customers to use the libraries. SDKs are maintained for compatibility—the HTTP interfaces are not. You will experience loads of dependencies (due to heavy tech-stacks), poor code quality, lots of configuration [cache, account registration, cloud integrations]. Amazon is similar in many ways, although they care about HTTP interface stability a bit more. Google's new Key Store works well. It still has some weird quirks at this stage.

The Go Cloud Development Kit did not manage to universalise any of the basic key storage operations.

[woutslakhorst]

Ahhh interesting, partners always request features and we look at libraries for their capabilities. Quality is almost never visible and always with hindsight.

[reinkrul]

I discussed this today in-person, apparently we have at least 3 parties running on Azure. For those, it doesn't make much sense to invest in a Hashicorp Vault deployment when their cloud environment has a similar service out-of-the-box. So it makes sense to make it possible for parties to use their secret manager of choice.

On the other hand, I don't think we should introduce that complexity (of supporting X, Y and Z) in the Nuts Node, since it'll probably be a poorly supported part of it. Prime example is the buggy Vault implementation, simply because who build it, don't/didn't actually use it.

We want to stimulate community members to contribute with actual tech/code, but adding a feature to the Nuts Node itself is for most a bridge too far (for those who I've talked to). Defining a simple REST API that should be implemented allows parties to implement an adapter service for their secret manager of choice in their own stack.

[reinkrul]

Diving into the possibility to perform cryptographic operations in an external secure device (HSM):

Microsoft Azure Has 2 options for key management (https://learn.microsoft.com/en-us/azure/key-vault/keys/about-keys-details):

• Dedicated HSM (starting at $4,85/hour): supports ECIES
• Key Vault: doesn't support ECIES

Amazon AWS Does not appear to support ECIES, only its own encryption ciphertext formats based on AES-GCM.

[stevenvegt]

TLDR: I looked into using Kubernetes Secrets to see if these can be used for storing private keys managed by the Nuts node and by so providing a secret store abstraction.

Conclusion: No, they cant be used.

Motivation: In the context of #1721: we want to configure a different default keystore back-end than the filesystem back-end. Currently we only support Hashicorp Vault. It feels a bit strange to make that the default without providing an alternative. Especially since a lot of parties running a Nuts node on the Azure cloud.

Since secret stores such as Hashicorp Vault and Microsoft Key Vault use a simple path based approach, my hunch was we could perhaps mount them on the filesystem and just read, list and write to mapped files. This would result in a simple abstraction instead of including several of the possible secret store SDK's in our Nuts node.

There is a Kubernetes Container Storage Interface (CSI) driver for secret stores: https://secrets-store-csi-driver.sigs.k8s.io/ which offers multiple providers for Vault, Azure, GCP and AWS.

I tested this with a local minikube setup, Vault and a test application, following this tutorial.

The limitation of this approach is that every secret must be explicitly mounted (and thus known during deployment) via the container configuration. As @reinkrul stated above, it is not possible to write to these directories to create new secrets. Rendering this approach worthless for our need to dynamically create and remove private keys from a secret store.

[stevenvegt]

Ideally there would be a standardized API that all secret engines support, but I'm not sure about that. Alternative route would be to develop a microservice for each supported secret engine (Hashicorp Vault/Azure KeyVault/GCP/AWS/etc), which can be deployed as (k8s) sidecar to the Nuts Node, through which all crypto operations are routed. Then the Hashicorp Vault integration would be extracted into a microservice as well.

I'm in favor of this approach, it keeps the code-base clean and allows for expansion and custom implementations.

[reinkrul]

Given the approach to deploy a sidecar for each (external) secret store; ideally, from API and security perspective, cryptographic operations are not performed in the Nuts Node, but in these adapters. Because then, you're compatible with stores that can protect the private keys by not exporting them (HSMs), since an HSM performs cryptographic operations inside it. And the private keys themselves are never available to the Nuts Node, so they can't be misused/leaked.

The downside to this, is that most stores won't support it and thus have to re-implement the cryptographic operations, instead of just providing a key-value store interface. We could simplify this by providing a Go implementation for a on-disk storage (which we currently have), which is both embedded in the Nuts Node and can be used as library for other implementations.

Question remains whether we're realistically going to need it;

• I highly doubt any vendor is actually using HSMs (maybe hospitals?), given the high costs
• Key management solutions from Cloud providers don't support the operations we need (ECIES), or at high cost (Azure)

---

On the other hand, if we're going to support it through a simple key-value interface, we should look into existing interfaces that abstract Key-Value storage.

[woutslakhorst]

ECIES might just go out of the window if we want to migrate to OIDC4VCI

[stevenvegt]

Today we discussed a few options. (See attached presentation)

Conclusions:

• Create an API specification for a simple key value store to store and retrieve secrets.
• Storage proxy implements the server
• Nuts will be the client
• We keep the FS backend which will be unavailable in strict mode for nuts node v6
• Will provide an official supported Vault proxy

Presentation with options. Proposed is option 3 secure storage abstration proposal v3.pdf

[reinkrul]

Now possible through implementation of secret store API: https://nuts-node.readthedocs.io/en/latest/pages/development/6-external-secret-store.html

To be released in v5.1