search

loft-sh / vcluster-plugins

The plugin repository for vcluster. Extend virtual Kubernetes clusters with plugins.
11 stars 6 forks source link

CrashLoopBackoff with the cert-manager-plugin #41

Open JPeer264 opened 10 months ago

JPeer264 commented 10 months ago

It seems that our vcluster cert-manager-plugin sidecar is having a CrashLoopBackoff.

We are running

From the logs it looks like there is some permission error with some existing CRDs. I double checked the service accounts and there are two service accounts within my-namespace: vc-my-namespace and vc-workload-my-namespace (but both without any secrets attached). Maybe it is important to say that recently we upgraded our Kubernetes cluster from 1.25 to 1.26.

Here are the logs from the restarting cert-manager-plugin sidecar:

I1221 16:53:53.918853       1 logr.go:249] plugin: Try creating context...
I1221 16:53:54.185675       1 logr.go:249] plugin: Plugin server listening on localhost:14000
I1221 16:53:54.189780       1 logr.go:249] plugin: Waiting for vcluster to become leader...
I1221 16:53:54.193890       1 logr.go:249] plugin: Starting syncers...
W1221 16:53:54.307447       1 util.go:16] Skip setting owner, because current namespace my-namespace != target namespace 
I1221 16:53:54.307625       1 logr.go:249] plugin: Start syncer certificate
I1221 16:53:54.307879       1 logr.go:249] plugin: Start syncer issuer
I1221 16:53:54.308329       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting Controller
I1221 16:53:54.308422       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting workers worker count 1
I1221 16:53:54.308357       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source kind source: *v1.Certificate
I1221 16:53:54.308790       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source &{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>  <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc0002852c0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I1221 16:53:54.308846       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source kind source: *v1.Ingress
I1221 16:53:54.308855       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting Controller
I1221 16:53:54.308516       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source &source.Kind{Type:(*v1.Issuer)(0xc0003ecf20), cache:(*cache.informerCache)(0xc0003ea2b0), started:(chan error)(nil), startCancel:(func())(nil)}
I1221 16:53:54.309124       1 logr.go:249] controller issuer controllerGroup cert-manager.io controllerKind Issuer: Starting EventSource source &source.kindWithCache{kind:source.Kind{Type:(*v1.Issuer)(0xc0003ecdc0), cache:(*cache.informerCache)(0xc0002a0498), started:(chan error)(nil), startCancel:(func())(nil)}}
I1221 16:53:54.309196       1 logr.go:249] plugin: Start syncer secret
I1221 16:53:54.309303       1 logr.go:249] plugin: Successfully started plugin.
I1221 16:53:54.310266       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source kind source: *v1.Secret
I1221 16:53:54.310443       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source &{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc0002852c0}) %!s(chan error=<nil>) %!s(func()=<nil>)}}
I1221 16:53:54.310644       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source kind source: *v1.Certificate
I1221 16:53:54.310732       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting EventSource source kind source: *v1.Issuer
I1221 16:53:54.310766       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting Controller
W1221 16:53:54.314512       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:53:54.314729       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:53:54.315073       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:53:54.315222       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:53:54.315358       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:53:54.315506       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:53:55.237477       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:53:55.237513       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:53:55.352838       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:53:55.352886       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:53:55.502413       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:53:55.502475       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:53:56.952989       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:53:56.953276       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:53:57.169682       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:53:57.169723       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:53:57.642061       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:53:57.642127       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:01.016241       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:01.016575       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:54:01.346556       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:54:01.346599       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:54:03.834723       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:54:03.835195       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:10.455922       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:10.456534       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:54:11.039885       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:54:11.040501       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:54:12.496913       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:54:12.497322       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:26.682542       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:54:26.682691       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:54:27.680297       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:27.680653       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:54:31.015092       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:54:31.015508       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:54:53.777878       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:54:53.778049       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
W1221 16:55:06.997712       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:55:06.998159       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:55:20.182484       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
E1221 16:55:20.182677       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Certificate: failed to list *v1.Certificate: certificates.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "certificates" in API group "cert-manager.io" at the cluster scope
W1221 16:55:45.340782       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
E1221 16:55:45.340864       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "secrets" in API group "" at the cluster scope
W1221 16:55:49.019376       1 reflector.go:324] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope
E1221 16:55:49.019421       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.Issuer: failed to list *v1.Issuer: issuers.cert-manager.io is forbidden: User "system:serviceaccount:my-namespace:vc-my-namespace" cannot list resource "issuers" in API group "cert-manager.io" at the cluster scope

E1221 16:55:54.309581       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: Could not wait for Cache to sync failed to wait for certificate caches to sync: timed out waiting for cache to be synced
I1221 16:55:54.309653       1 deleg.go:121] Stopping and waiting for non leader election runnables
I1221 16:55:54.309665       1 deleg.go:121] Stopping and waiting for leader election runnables
E1221 16:55:54.309888       1 deleg.go:135] controller-runtime: source: failed to get informer from cache Timeout: failed waiting for *v1.Certificate Informer to sync
E1221 16:55:54.309923       1 deleg.go:135] controller-runtime: source: failed to get informer from cache Timeout: failed waiting for *v1.Issuer Informer to sync
E1221 16:55:54.309940       1 deleg.go:135] controller-runtime: source: failed to get informer from cache Timeout: failed waiting for *v1.Secret Informer to sync
I1221 16:55:54.310017       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Shutdown signal received, waiting for all workers to finish
I1221 16:55:54.310028       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: All workers finished
I1221 16:55:54.310352       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Starting workers worker count 1
I1221 16:55:54.310467       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: Shutdown signal received, waiting for all workers to finish
E1221 16:55:54.310888       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"k3s-serving", Namespace:"kube-system"} namespace kube-system name k3s-serving: reconcileID "7579cd32-a8fe-4102-af6e-70f69a8e1728": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311174       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v67", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v67: reconcileID "7bbb5ed4-439e-4adb-a334-53da4629b29c": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311426       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v68", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v68: reconcileID "14f25fc7-a0f2-4a42-876e-5cb718ddebef": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311591       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"basic-auth", Namespace:"my-namespace"} namespace my-namespace name basic-auth: reconcileID "494fbdea-9885-4dfc-8bbd-1e13b9b16683": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.311779       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"my-namespace-0.node-password.k3s", Namespace:"kube-system"} namespace kube-system name my-namespace-0.node-password.k3s: reconcileID "b349391a-bf97-4f4a-a145-63c7bb64faa2": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312196       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"quiz-devopscycle-com-tls", Namespace:"my-namespace"} namespace my-namespace name quiz-devopscycle-com-tls: reconcileID "defec9a0-61ae-4d7c-b3e8-0fac2c0a5295": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312298       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v70", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v70: reconcileID "7d1fa0e5-83f5-4fa0-a635-b941e1451696": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312524       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v71", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v71: reconcileID "5af977d6-ad06-431c-ad64-2029e63d0b22": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.312789       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v73", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v73: reconcileID "2141b103-7994-4b30-8280-88fdd0cdfc74": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313070       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v69", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v69: reconcileID "b87876d0-9385-4b9b-93e1-3d31efa1ec6c": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313278       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"api-secrets-my-namespace", Namespace:"my-namespace"} namespace my-namespace name api-secrets-my-namespace: reconcileID "3a783b87-774d-43fd-9f6f-d7e56608ee10": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313497       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"regcred", Namespace:"my-namespace"} namespace my-namespace name regcred: reconcileID "06afaf48-815f-47f2-870a-0712676c6b4c": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313680       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"my-namespace-tls", Namespace:"my-namespace"} namespace my-namespace name my-namespace-tls: reconcileID "09fe2b41-0be8-476a-95b2-c49ae470859d": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.313815       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v74", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v74: reconcileID "2cbfb54f-ddc3-4434-9a35-af7c6480b671": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314092       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v66", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v66: reconcileID "7d8a7110-bd5f-42c9-90df-4dd2817af1b9": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314333       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v75", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v75: reconcileID "e51e8f75-73cf-4920-afa3-84f33b1f29cf": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314462       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"sh.helm.release.v1.my-namespace.v72", Namespace:"my-namespace"} namespace my-namespace name sh.helm.release.v1.my-namespace.v72: reconcileID "eabfdc67-c950-4179-a455-4c08b3471bc6": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
E1221 16:55:54.314594       1 logr.go:265] controller secret controllerGroup  controllerKind Secret: secret klog.ObjectRef{Name:"quiz-redirect-devopsberatung-at-tls", Namespace:"my-namespace"} namespace my-namespace name quiz-redirect-devopsberatung-at-tls: reconcileID "9de7bcad-2fd9-479b-ad28-c6af0d9c3898": Reconciler error Timeout: failed waiting for *v1.Secret Informer to sync
I1221 16:55:54.314614       1 logr.go:249] controller secret controllerGroup  controllerKind Secret: All workers finished
I1221 16:55:54.314679       1 deleg.go:121] Stopping and waiting for caches
I1221 16:55:54.314992       1 deleg.go:121] Stopping and waiting for webhooks
I1221 16:55:54.315115       1 deleg.go:121] Wait completed, proceeding to shutdown the manager
panic: failed to wait for certificate caches to sync: timed out waiting for cache to be synced

goroutine 286 [running]:
github.com/loft-sh/vcluster-sdk/plugin.(*manager).start.func4()
    /go/vcluster/vendor/github.com/loft-sh/vcluster-sdk/plugin/plugin.go:569 +0x4e
created by github.com/loft-sh/vcluster-sdk/plugin.(*manager).start
    /go/vcluster/vendor/github.com/loft-sh/vcluster-sdk/plugin/plugin.go:566 +0x54f

In the meantime the logs from the syncer sidecar:

2023-12-21 16:53:54 INFO    plugin/plugin.go:225    Registering plugin cert-manager-plugin  {"component": "vcluster"}
2023-12-21 16:53:54 INFO    plugin/plugin.go:304    Register client hook for networking.k8s.io/v1 Ingress in plugin cert-manager-plugin {"component": "vcluster"}
2023-12-21 16:53:54 INFO    loghelper/klog.go:24    Stopped tunnel to 127.0.0.1:6443    {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info"}
2023-12-21 16:53:54 INFO    loghelper/klog.go:24    Connecting to proxy {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info", "url": "wss://10.244.0.81:8443/v1-k3s/connect"}
2023-12-21 16:53:54 INFO    loghelper/klog.go:24    Proxy done  {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info", "err": "context canceled", "url": "wss://127.0.0.1:6443/v1-k3s/connect"}
2023-12-21 16:53:54 INFO    loghelper/klog.go:24    error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF    {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info"}
2023-12-21 16:53:54 INFO    loghelper/klog.go:24    Handling backend connection request [my-namespace-0]    {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:53:54Z", "level": "info"}
2023-12-21 16:54:06 INFO    loghelper/klog.go:24    COMPACT compactRev=110215 targetCompactRev=110246 currentRev=111246 {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:54:06Z", "level": "info"}
2023-12-21 16:54:06 INFO    loghelper/klog.go:24    COMPACT deleted 31 rows from 31 revisions in 2.931345ms - compacted to 110246/111246    {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:54:06Z", "level": "info"}
2023-12-21 16:54:37 ERROR   filters/wrap.go:54  timeout or abort while handling: method=GET URI="/api/v1/namespaces?allowWatchBookmarks=true&resourceVersion=111059&timeout=7m46s&timeoutSeconds=466&watch=true" audit-ID="1d12f7af-6add-4be9-82dc-934dc86bf880"    {"component": "vcluster"}
2023-12-21 16:54:54 ERROR   filters/wrap.go:54  timeout or abort while handling: method=GET URI="/v1-k3s/connect" audit-ID="25f381b0-eba5-46ae-9403-31d57bb73929"   {"component": "vcluster"}
2023-12-21 16:55:55 INFO    loghelper/klog.go:24    Stopped tunnel to 10.244.0.81:8443  {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info"}
2023-12-21 16:55:55 INFO    loghelper/klog.go:24    Connecting to proxy {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info", "url": "wss://127.0.0.1:6443/v1-k3s/connect"}
2023-12-21 16:55:55 INFO    loghelper/klog.go:24    Proxy done  {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info", "err": "context canceled", "url": "wss://10.244.0.81:8443/v1-k3s/connect"}
2023/12/21 16:55:55 websocketproxy: Error when copying from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF
2023-12-21 16:55:55 INFO    loghelper/klog.go:24    Handling backend connection request [my-namespace-0]    {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info"}
2023-12-21 16:55:55 ERROR   filters/timeout.go:142  post-timeout activity - time-elapsed: 1m0.689537949s, GET "/v1-k3s/connect" result: <nil>   {"component": "vcluster"}
2023-12-21 16:55:55 INFO    loghelper/klog.go:24    error in remotedialer server [400]: websocket: bad close code 1006  {"component": "vcluster", "component": "k3s", "time": "2023-12-21T16:55:55Z", "level": "info"}

I would be more than happy to share more information if needed.

FabianKramm commented 9 months ago

@JPeer264 thanks for creating this issue! We will take a look

trondvindenes-hvikt commented 9 months ago

I'm experiencing the same issue

rarg27 commented 8 months ago

I'm experiencing the same issue

Perchework commented 8 months ago

Same here. cert-manager 1.11 vcluster 0.18.1 kubernetes 1.26.9

Working with same configuration on vcluster 0.15.0