techcobweb commented 6 months ago

Story

As an installer of galasa, I want to configure my ecosystem to draw secrets from a remote centralised secrets manager, so that I know the secrets are being stored and organised in a way consistent with my mandated security policies.

Background

Current solution

Currently credentials are stored in an etcd instance, unencrypted, accessed through the CredentialsStoreService
This is insecure. Anyone with etcd access can get all the secrets stored within.

Proposed solution

We need to do this:

ie:

The testcase author writes a secret into the secrets manager.
The helm chart setting up the pods needs to supply a 'salt' encryption/decryption secret to the API pod and the testcase pods.
The pods can get the content from the creds via a new adapter, which copes with decrypting the contents of etcd when secrets are required using the 'salt' key.
The REST API accepts secrets in the payload. TLS encrypts the secrets in transit
The REST API uses the 'salt' secret to encrypt the secret data and place it into the etcd component.
The mechanism needs to allow someone to add a secret to the external secrets manager, then write a testcase which draws on that secret, with no listing of the secret anywhere except the testcase. So the person adding an additional secret does not need kubernetes admin rights.

Why we chose IBM Cloud secrets manager

There are many secret stores we could decide to use. Each cloud vendor has one for example. This story will code the extension which talks to the IBM Cloud Secrets Manager. (The IBM team have access to instances of this, so it will be possible to code and test). Other extensions may be written in future also. A mechanism needs to be provided to configure the ecosystem to draw from a single credentials extension (etcd, IBMCloud, a.n.other).

How the connector could work

The IBM cloud secrets manager supports a 'group' or 'namespace' of secrets, which can have different ACSs assigned for each user ID... so if the secrets manager instance is used for many things, all the secrets which Galasa testcases need could be put into the same 'group', and the access token used by Galasa can be restricted to only be able to read from that group.
To set this configuration up, someone with admin rights is going to need to create a kubernetes secret secrets_manager_access_token. This should be an external secret which is also drawn from the same external secrets manager (though it could be in a separate group/namespace), as this secret is for galasa to use, not individual test cases.
Some configuration will be needed by the connector also, some of which needs to be stored in kubernetes secrets, and supplied to the connector code at runtime.

To reset the seed secret

When there is a secrets connector

stop all test pods
set the new seed value into the kubernetes secret
stop+restart the API pod
- This could be automated. etcd has a callback mechanism such that if something changes we get to hear about it, and can shut down the API pod automatically.
- Kubernetes secrets (if mounted as files rather than environment variables) will appear as updated in running pods. So code written to use that seed secret just needs to re-read the cached secret every 10 seconds, and it will become updated and start using the correct seed.
stop+restart the secrets manager connector pod (it will synchronise to re-write all the secrets available, which will all be encrypted using the new salt value
start using test pods again

When there isn't a secrets connector

stop all test pods
backup all the secrets stored in etcd, so the values are in the clear. Encrypting any backup files if you store them away anywhere.
set the new seed value into the kubernetes secret
stop+restart the API pod
restore all the secrets stored in etcd
start using the test pods again

Reasons why test pods do not get secrets on demand direct from the IBM cloud secret store

The IBM cloud secret store is limited to 20 calls per second. With 100 testcases trying to get 5 secrets each, that could add up to 500 calls to the secret store. The service doesn't have a hard-stop, but delays calls if the call rate is greater than 20. But it would still take 25 seconds for those tests to get all those secrets. There are also limits on the number of concurrent accesses from unique clients. Which itself rules out on-demand access. The docs also state that it is a cold-storage for secrets. See the documentation for more details.

This is exacerbated by the fact that each pod would have to get IBM Cloud IAM credentials by logging in... another process which will take some time.

Conclusion: We need to draw the secrets out of the secret store and cache them, sharing them via etcd, which is built for clustered parallel access.

REST interface

GET /secrets - lists the secrets available GET /secrets?name=xxx - gets the definition of the secret, with the value POST /secrets/name - creates a new secret PUT /secrets/name - updates an existing secret DELETE /secrets/name - deletes a secret

POST /resources - Allows a secret resource to be created/updated/deleted.

A secret looks like this:

apiVersion: galasa-dev/v1alpha1
kind: GalasaSecretToken
metadata:
    name: my.secret.name
    encoding: base64
data:
    token: bXktc2VjcmV0LXZhbHVlLTEyMzQ1 << This is "my-secret-value-12345" but encoded.

or

apiVersion: galasa-dev/v1alpha1
kind: GalasaSecretUserName
metadata:
    name: my.secret.name
    encoding: base64
data:
    username: bXktc2VjcmV0LXZhbHVlLTEyMzQ1 << This is "my-secret-value-12345" but encoded.

or

apiVersion: galasa-dev/v1alpha1
kind: GalasaSecretUserNamePassword
metadata:
    name: my.secret.name
    encoding: base64
data:
    username: bXktc2VjcmV0LXZhbHVlLTEyMzQ1 << This is "my-secret-value-12345" but encoded.
    password: bXktc2VjcmV0LXZhbHVlLTEyMzQ1 << This is "my-secret-value-12345" but encoded.

or

apiVersion: galasa-dev/v1alpha1
kind: GalasaSecretUserNameToken
metadata:
    name: my.secret.name
    encoding: base64
data:
    username: bXktc2VjcmV0LXZhbHVlLTEyMzQ1 << This is "my-secret-value-12345" but encoded.
    token: bXktc2VjcmV0LXZhbHVlLTEyMzQ1 << This is "my-secret-value-12345" but encoded.

Limiting who can get what secret information

We don't have Role-based access implemented yet. When we do we can limit who can see which secrets. ie: The GET data could be refused unless you have secret reading permissions in your role.

So until that point, everyone can see all the secret values.

Observations

The existing solution needs to be improved so it can hold caches of secret information within etcd. That will make everyone more secure, and allow easier setting of secrets for people without the IBM Cloud Secrets Manager and connector.
The connector for IBM Cloud Secrets Manager can be added once the basic solution is improved.

Tasks

Improve the existing credentials solution
- secrets in the creds store are encrypted
- kube supplies the 'salt' secret to the REST API pod
- kube supplies the 'salt' secret to the test pods
- the REST API encrypts secrets before putting them into etcd
- the REST API decrypts secrets after getting them out of etcd
- the creds supplier in the framework decrypts secrets using the 'salt'
- the galasactl program can set a secret + value
- the galasactl program can get a secret + value
The helm chart needs to be set up with values which control the connector to IBM secrets manager.
- A config map detailing how often the secrets are synchronized
- secrets supplied by kubernetes to allow it to access the secrets manager
- details of which namespaces from IBM secrets manager which should be pulled
- all the secrets in the namespace should be replicated into the REST API layer
- a secret token which allows the connector to contact the rest API layer and authenticate

techcobweb commented 6 months ago

I had a discussion with @Tom-Slattery today, and ran this outline plan past him, to which he said that it fitted well with the direction he was going to have to take with secrets management in the future.

techcobweb commented 5 months ago

secrets.drawio.zip

This is the source for the diagram in the description.