search

bitnami-labs / sealed-secrets

A Kubernetes controller and tool for one-way encrypted Secrets
Apache License 2.0
7.68k stars 684 forks source link

After update to Helm release 2.6.4 `Error updating SealedSecret "secret" not found` message in log #961

Closed 3deep5me closed 2 years ago

3deep5me commented 2 years ago

Which component: Helm release 2.6.4

Describe the bug After the update from chart 2.6.2 to 2.6.4 i get following error on recreated/new created sealedsecrets:

2022/09/15 10:26:11 Updating namespace/secret
2022/09/15 10:26:11 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"namespace", Name:"secret", UID:"5d1436b0-3655-4e69-9a9dc774c0dd", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"379770364", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 10:26:11 Error updating SealedSecret namespace/secret status: sealedsecrets.bitnami.com "secret" not found

To Reproduce Steps to reproduce the behavior:

  1. Install Helm release 2.6.4
  2. Create a new or recreate a sealed-secret
  3. check the logs
  4. See error

Expected behavior No not found message in the log

Version of Kubernetes: v1.21.6 (Openshift) v1.22.6 (AKS)

(AKS)

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"ece9ecf2f9aecbd86d3eba31f0be62e4b6353a5a", GitTreeState:"clean", BuildDate:"2022-07-28T23:33:17Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

(Openshift)

WCAlRiNeInNtG :V evresrisoinon :d iffveerresnicoen .bIentfwoeen{ clieMnatj o(r1.2:3") a1nd "s,e rveMri n(o1r.21:)" ex2c3eed"s,  theG istuVpeprosriteodn mi:n"orv 1v.e2r3s.i0o"n,  skeGwi toCfo m+m/i-t1:
"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6+4b61f94", GitCommit:"cefce093e4e5bc9a1916eb5a489ed37c7d467f6f", GitTreeState:"clean", BuildDate:"2022-02-09T02:16:16Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

Additional context It seems like this also breaks the argocd health-check which caused that the resources is always on status "processing" in argocd with health-detail "Waiting for Sealed Secret to be decrypted" even its decrypted. https://github.com/argoproj/argo-cd/blob/master/resource_customizations/bitnami.com/SealedSecret/health.lua

josvazg commented 2 years ago

Could you provide more details on the steps to reproduce the error?

Here is an example we tried with that version, and we could not reproduce:

$ helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller --version 2.6.4 sealed-secrets/sealed-secrets
...
$ helm ls -A
NAME            NAMESPACE   REVISION    UPDATED                                     STATUS      CHART                   APP VERSION
sealed-secrets  kube-system 1           2022-09-15 17:09:13.055100484 +0200 CEST    deployed    sealed-secrets-2.6.4    v0.18.2
...
# test
$ echo -n foobar | kubectl create secret generic secret --dry-run=client --from-file=foo=/dev/stdin -o json > /tmp/secret.json
$ kubeseal < /tmp/secret.json > /tmp/sealedsecret.json
$ kubectl create -f /tmp/sealedsecret.json
sealedsecret.bitnami.com/secret created
$ kubectl get sealedsecrets
NAME     AGE
secret   9s
$ kubectl get secrets
NAME     TYPE     DATA   AGE
secret   Opaque   1      14s
$ kubectl get secrets/secret -oyaml
apiVersion: v1
data:
  foo: Zm9vYmFy
kind: Secret
...
$ echo Zm9vYmFy | base64 -d
foobar
josvazg commented 2 years ago

Sending us the logs from the controller would also help:

For example, for the above test:

$ kubectl get pods -A |grep sealed
kube-system   sealed-secrets-controller-68dd8fb447-jtt25   1/1     Running   0              8m49s

$ kubectl logs -n kube-system sealed-secrets-controller-68dd8fb447-jtt25
controller version: 0.18.2
2022/09/15 15:09:19 Starting sealed-secrets controller version: 0.18.2
2022/09/15 15:09:19 Searching for existing private keys
2022/09/15 15:09:19 ----- sealed-secrets-keyt2jq9
2022/09/15 15:09:19 HTTP server serving on :8080
2022/09/15 15:12:03 Updating default/secret
2022/09/15 15:12:03 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"secret", UID:"fb4954ac-5dee-4f56-98f4-eeab95ab0824", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"25299", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 15:12:03 Updating default/secret
2022/09/15 15:12:03 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"secret", UID:"fb4954ac-5dee-4f56-98f4-eeab95ab0824", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"25301", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
3deep5me commented 2 years ago

Thanks for trying to reproduce. I tried to do the same

$ echo -n foobar | kubectl create secret generic secret --dry-run=client --from-file=foo=/dev/stdin -o json > /tmp/secret.json
$ kubeseal --controller-name sealed-secrets < /tmp/secret.json > /tmp/sealedsecret.json
$ kubectl create -f /tmp/sealedsecret.json
sealedsecret.bitnami.com/secret created
$ kubectl get sealedsecrets
NAME     AGE
secret   10s
$ kubectl get secret secret
NAME     TYPE     DATA   AGE
secret   Opaque   1      29s
$ kubectl get secrets/secret -oyaml
apiVersion: v1
data:
  foo: ITxzeW1saW5rPv/+LwBwAHIAbwBjAC8AcwBlAGwAZgAvAGYAZAAvADAAAAA=
kind: Secret
metadata:
  creationTimestamp: "2022-09-15T15:54:31Z"
  name: secret
  namespace: monitoring
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: secret
    uid: f9e580ff-9bb3-47af-add7-972a9a75f046
  resourceVersion: "32118004"
  uid: f6326167-5415-4650-ad09-b6ab8db44687
type: Opaque
$ echo ITxzeW1saW5rPv/+LwBwAHIAbwBjAC8AcwBlAGwAZgAvAGYAZAAvADAAAAA= | base64 -d
!<symlink>□□/proc/self/fd/0

The logs from the controller:

 $ k logs -n kube-system sealed-secrets-6dcc9776cf-57dfz
controller version: 0.18.2
2022/09/15 15:42:01 Starting sealed-secrets controller version: 0.18.2
2022/09/15 15:42:01 Searching for existing private keys
2022/09/15 15:42:01 ----- sealed-secrets-key4flrz
2022/09/15 15:42:01 ----- sealed-secrets-keyghwdt
2022/09/15 15:42:01 ----- sealed-secrets-keymhtw5
2022/09/15 15:42:01 HTTP server serving on :8080
2022/09/15 15:54:31 Updating monitoring/secret
2022/09/15 15:54:31 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"monitoring", Name:"secret", UID:"f9e580ff-9bb3-47af-add7-972a9a75f046", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"32118003", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 15:54:31 Error updating SealedSecret monitoring/secret status: sealedsecrets.bitnami.com "secret" not found

The only difference is that i did the install with argocd: (No values modified) 

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
  annotations:
spec:
  destination:
    namespace: kube-system
    server: https://kubernetes.default.svc
  project: infra
  source:
    path: helm/sealed-secrets
    repoURL: https://github.com/bitnami-labs/sealed-secrets.git
    targetRevision: helm-v2.6.4
  syncPolicy:
    automated: {}

I will try this also on a new cluster with a all new install tomorrow.

martin-jensen commented 2 years ago

If it can help, i have been having the same issue, tried with both a clean install of k3s or microk8s.

For extra info I am on windows and using the windows version of kubectl and kubeseal.

So with a new ubuntu vm (22.04) created with multipass and a default install of k3s.

  1. helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller --version 2.6.4 sealed-secrets/sealed-secrets
  2. kubectl create secret generic test --dry-run=client --output=json --from-literal=password=supersecret | kubeseal > mysealedsecret.json
  3. kubectl apply -f .\mysealedsecret.json

Results in the following log from the controller

controller version: 0.18.2
2022/09/15 15:52:26 Starting sealed-secrets controller version: 0.18.2
2022/09/15 15:52:26 Searching for existing private keys
2022/09/15 15:52:28 New key written to kube-system/sealed-secrets-keyj6s66
2022/09/15 15:52:28 Certificate is
-----BEGIN CERTIFICATE-----
xxxx
-----END CERTIFICATE-----

2022/09/15 15:52:28 HTTP server serving on :8080
2022/09/15 15:53:14 Updating default/test
2022/09/15 15:53:14 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"test", UID:"fdb2491d-3877-4f49-aac9-559fc4591dff", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"698", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 15:53:14 Error updating SealedSecret default/test status: sealedsecrets.bitnami.com "test" not found

Output of "kubectl version"

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4+k3s1", GitCommit:"c3f830e9b9ed8a4d9d0e2aa663b4591b923a296e", GitTreeState:"clean", BuildDate:"2022-08-25T03:45:26Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Output of "kubeseal --version" kubeseal version: 0.18.2

rtsp commented 2 years ago

This also happen to me on all secrets that unsealed after 2.6.4 upgrade.

2022/09/15 20:13:20 Updating gitlab-runner/gitlab-runner-secret
2022/09/15 20:13:20 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"kube-prometheus-stack", Name:"kube-prometheus-stack-grafana", UID:"abd6bae2-f1af-40b7-aa85-3152a70f7588", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3082499", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:13:20 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"gitlab-runner", Name:"gitlab-runner-secret", UID:"35c57d22-693d-430e-ba69-424ec1921405", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3734948", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:13:20 Error updating SealedSecret gitlab-runner/gitlab-runner-secret status: sealedsecrets.bitnami.com "gitlab-runner-secret" not found

2022/09/15 20:18:50 Updating gitlab-runner/gitlab-runner-secret
2022/09/15 20:18:50 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"gitlab-runner", Name:"gitlab-runner-secret", UID:"35c57d22-693d-430e-ba69-424ec1921405", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3737826", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:18:50 Error updating SealedSecret gitlab-runner/gitlab-runner-secret status: sealedsecrets.bitnami.com "gitlab-runner-secret" not found

2022/09/15 20:18:50 Updating gitlab-runner/gitlab-runner-secret
2022/09/15 20:18:50 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"gitlab-runner", Name:"gitlab-runner-secret", UID:"83934bde-5235-48ac-8979-ffa5bc3973e8", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3737842", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:18:50 Error updating SealedSecret gitlab-runner/gitlab-runner-secret status: sealedsecrets.bitnami.com "gitlab-runner-secret" not found

The secrets unsealed succesfully and working good except the status field of the parent SealedSecrets resource is missing. This cause Argo CD to stuck in "Progressing" state.

image

vizv commented 2 years ago

Found the issue, working on the fix...

alemorcuq commented 2 years ago

We will release a new chart including the patched CRD shortly.

alemorcuq commented 2 years ago

Chart 2.6.7 released including the patched CRD. Thanks, @vizv !