Missing private key in working cluster

[dil-sthaens]

I tried the command to backup the private key of the cluster

$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >master.key

Unfortunately the created file was empty because there is no sealed-secrets-key in kube-system namespace. My questione here would be: Can I check somehow which key in K8s is currently used as private key? And can I backup this one for an emergency restore?

[tewfik-ghariani]

All keys having the sealedsecrets.bitnami.com/sealed-secrets-key set to active are considered by the controller during the creation and registered in its key registry. The key with the most recent creation time is the one that is currently being used and exposed via the v1/cert.pem endpoint

The reason the exported key is empty might be due to the fact that the controller exists in a different namespace. Can you double check that the service/pod/deployment do exist in that namespace?

If not, if you have permissions, you can list all secrets across the cluster with the -A option

[dil-sthaens]

Yes I have a lot of keys in the same namespace in which the sealed-secrets-controller service is located. So when I pick the one with most recent creation time and export it, its also possible to use this key to recover if the keys in the cluster get lost somehow?

[tewfik-ghariani]

The most recent key is the one that is currently used to encrypt/decrypt newly applied secrets.

However it is better to export all "active" keys ( meaning with the label set to active ) as those might have encrypted old secrets and they are needed to decrypt them. [ This is of course applicable if you have not disabled the automated renewal of sealing keys every 30 days ]

kubectl get secret -n ${CORRECT_NAMESPACE} -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml

Hint: -l sealedsecrets.bitnami.com/sealed-secrets-key is used as selector

[dil-sthaens]

Thanks for your answers! This was really helpful!

[dil-sthaens]

Ohh wait a second. Just for my understanding: We created at first a key pair with private and public key. We use that public key to seal the secrets with kubeseal. So the automated generated new keys in sealed secret arent used because they should have different public keys which we do not use for sealing when we manually use the public key from the creation time to seal a secret, right?

kubeseal --cert=publickeyfromfirstgeneration.pem -o yaml

So the key we use has to be the oldest one in that case? And to be sure I should rather export all?

[tewfik-ghariani]

That should be correct. I do not know if it would be exactly the first generation keypair as it might actually be the second since the controller might have created its own keypair. But you should be able to identify it with the following command

kubectl get secrets -l sealedsecrets.bitnami.com/sealed-secrets-key=active  -o yaml | grep -B13 -A1 $(base64 publickeyfromfirstgeneration.pem)

[dil-sthaens]

@tewfik-ghariani thx for you answer. Unfortunately the command doesnt work as expected and just deliveres the output of base46 certname.pem with the note of not finding file or directory (the XXX replaces the base46 output) ❯ kubectl get secrets -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml | grep -B13 -A1 $(base64 pubcert.pem) grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX: No such file or directory grep: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==: No such file or directory

[tewfik-ghariani]

@dil-sthaens Can you make sure that you are using the correct namespace? Is the first command displaying any results at all?

[dil-sthaens]

@tewfik-ghariani the first commands prints out all the sealed-secrets in yaml form so I think the grep is somehow the issue?

[tewfik-ghariani]

Not sure what can be the issue, grepping the pipe output of the previous command should be simple and straightforward

Maybe try it this way?

enc_cert=$(base64  pubcert.pem)
grep -B13 -A1 $enc_cert <(kubectl get secrets -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml )

[dil-sthaens]

@tewfik-ghariani that worked but the output doesn't tell the sealed-secret key name

apiVersion: v1
items:
- apiVersion: v1
  data:
    tls.crt:XXXXXXX
    tls.key:XXXXXX 

[tewfik-ghariani]

@dil-sthaens Cool, you might want to increase the number of the "after" lines to find the uuid and name grep -B13 -A11 ...

[dil-sthaens]

@tewfik-ghariani thanks, that did the trick