search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
28.03k stars 2.35k forks source link

k3s service does not start when datastore configured with external Postgresql database connections pool (PgBouncer). #7808

Closed alex1704 closed 1 year ago

alex1704 commented 1 year ago

Disclaimer I'm new to k3s so feel free to point me out if I do something stupid.

Environmental Info: K3s Version: k3s version v1.26.5+k3s1 (7cefebea) go version go1.19.9

Node(s) CPU architecture, OS, and Version: Linux k3s-0 5.19.0-45-generic #46-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 7 09:08:58 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: Clean install and start: HA cluster with external postgresql database. Postgresql use PgBouncer connections pool. Cloud: DigitalOcean.

Describe the bug: k3s service does not start after clean install if I connect to DigitalOcean's Postgresql managed database connections pool. When I connect to Postgresql managed database directly k3s starts successfully.

Steps To Reproduce:

Expected behavior: k3s service starts successfully.

Actual behavior: k3s service failed to start: Failed to start Lightweight Kubernetes.

Additional context / logs: journalctl logs are flooded with errors like: ERROR: prepared statement \"stmtcache_19\" does not exist (SQLSTATE 26000)".

Logs:

Jun 21 03:19:10 k3s-0 sh[114997]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Found ip 10.110.0.7 from iface eth1"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Starting k3s v1.26.5+k3s1 (7cefebea)"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Configuring pgx database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Database tables and indexes are up to date"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Kine available at unix://kine.sock"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Reconciling bootstrap data between datastore and disk"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Running kube-apiserver --advertise-address=10.110.0.7 --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-servers=unix://kine.sock --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="To join server node to cluster: k3s server -s https://134.122.62.243:6443 -t ${SERVER_NODE_TOKEN}"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="To join agent node to cluster: k3s agent -s https://134.122.62.243:6443 -t ${AGENT_NODE_TOKEN}"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Run: k3s kubectl"
Jun 21 03:19:10 k3s-0 k3s[115000]: W0621 03:19:10.965744  115000 feature_gate.go:241] Setting GA feature gate JobTrackingWithFinalizers=true. It will be removed in a future release.
Jun 21 03:19:10 k3s-0 k3s[115000]: I0621 03:19:10.966138  115000 server.go:569] external host was not specified, using 10.110.0.7
Jun 21 03:19:10 k3s-0 k3s[115000]: I0621 03:19:10.966929  115000 server.go:171] Version: v1.26.5+k3s1
Jun 21 03:19:10 k3s-0 k3s[115000]: I0621 03:19:10.967107  115000 server.go:173] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 21 03:19:10 k3s-0 k3s[115000]: time="2023-06-21T03:19:10Z" level=info msg="Waiting for API server to become available"
Jun 21 03:19:10 k3s-0 k3s[115000]: I0621 03:19:10.987378  115000 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Jun 21 03:19:10 k3s-0 k3s[115000]: I0621 03:19:10.987770  115000 plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
Jun 21 03:19:10 k3s-0 k3s[115000]: I0621 03:19:10.990644  115000 shared_informer.go:270] Waiting for caches to sync for node_authorizer
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.046293  115000 genericapiserver.go:660] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
Jun 21 03:19:11 k3s-0 k3s[115000]: I0621 03:19:11.047920  115000 instance.go:277] Using reconciler: lease
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/apiextensions.k8s.io/customresourcedefinitions/ /registry/apiextensions.k8s.io/customresourcedefinitions/: ERROR: prepared statement \"stmtcache_13\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/apiextensions.k8s.io/customresourcedefinitions/ /registry/apiextensions.k8s.io/customresourcedefinitions/: ERROR: prepared statement \"stmtcache_5\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.077Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b44c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_5\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.077Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b44c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_13\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.078225  115000 reflector.go:424] storage/cacher.go:/apiextensions.k8s.io/customresourcedefinitions: failed to list *apiextensions.CustomResourceDefinition: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_13" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.078779  115000 cacher.go:450] cacher (customresourcedefinitions.apiextensions.k8s.io): unexpected ListAndWatch error: failed to list *apiextensions.CustomResourceDefinition: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_13" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/events/ /registry/events/: ERROR: prepared statement \"stmtcache_14\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.103Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0020128c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_14\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/podtemplates/ /registry/podtemplates/: ERROR: prepared statement \"stmtcache_18\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/resourcequotas/ /registry/resourcequotas/: ERROR: prepared statement \"stmtcache_5\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.259Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b44c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_17\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/limitranges/ /registry/limitranges/: ERROR: prepared statement \"stmtcache_19\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/podtemplates/ /registry/podtemplates/: ERROR: prepared statement \"stmtcache_20\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.280Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002013c00/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_5\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.280738  115000 reflector.go:424] storage/cacher.go:/resourcequotas: failed to list *core.ResourceQuota: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_5" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.281012  115000 cacher.go:450] cacher (resourcequotas): unexpected ListAndWatch error: failed to list *core.ResourceQuota: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_5" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.281Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ed3500/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_18\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.281390  115000 reflector.go:424] storage/cacher.go:/podtemplates: failed to list *core.PodTemplate: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_18" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.281557  115000 cacher.go:450] cacher (podtemplates): unexpected ListAndWatch error: failed to list *core.PodTemplate: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_18" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/limitranges/ /registry/limitranges/: ERROR: prepared statement \"stmtcache_22\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/persistentvolumes/ /registry/persistentvolumes/: ERROR: prepared statement \"stmtcache_24\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.300Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002012c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_22\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/namespaces/ /registry/namespaces/: ERROR: prepared statement \"stmtcache_31\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/persistentvolumes/ /registry/persistentvolumes/: ERROR: prepared statement \"stmtcache_32\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/secrets/ /registry/secrets/: ERROR: prepared statement \"stmtcache_26\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/persistentvolumeclaims/ /registry/persistentvolumeclaims/: ERROR: prepared statement \"stmtcache_25\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.320Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ed3500/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_20\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.320Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002012c40/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_19\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.320980  115000 reflector.go:424] storage/cacher.go:/limitranges: failed to list *core.LimitRange: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_19" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.321178  115000 cacher.go:450] cacher (limitranges): unexpected ListAndWatch error: failed to list *core.LimitRange: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_19" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/configmaps/ /registry/configmaps/: ERROR: prepared statement \"stmtcache_33\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/services/endpoints/ /registry/services/endpoints/: ERROR: prepared statement \"stmtcache_34\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.340Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0026d48c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_31\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.341327  115000 reflector.go:424] storage/cacher.go:/namespaces: failed to list *core.Namespace: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_31" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.341534  115000 cacher.go:450] cacher (namespaces): unexpected ListAndWatch error: failed to list *core.Namespace: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_31" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.341Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002434540/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_32\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.361Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002434540/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_24\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.361569  115000 reflector.go:424] storage/cacher.go:/persistentvolumes: failed to list *core.PersistentVolume: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_24" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.361754  115000 cacher.go:450] cacher (persistentvolumes): unexpected ListAndWatch error: failed to list *core.PersistentVolume: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_24" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/serviceaccounts/ /registry/serviceaccounts/: ERROR: prepared statement \"stmtcache_41\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/persistentvolumeclaims/ /registry/persistentvolumeclaims/: ERROR: prepared statement \"stmtcache_42\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/pods/ /registry/pods/: ERROR: prepared statement \"stmtcache_40\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/secrets/ /registry/secrets/: ERROR: prepared statement \"stmtcache_43\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/resourcequotas/ /registry/resourcequotas/: ERROR: prepared statement \"stmtcache_35\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/pods/ /registry/pods/: ERROR: prepared statement \"stmtcache_37\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.387Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002435c00/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_25\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.387757  115000 reflector.go:424] storage/cacher.go:/persistentvolumeclaims: failed to list *core.PersistentVolumeClaim: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_25" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.387974  115000 cacher.go:450] cacher (persistentvolumeclaims): unexpected ListAndWatch error: failed to list *core.PersistentVolumeClaim: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_25" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.388Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022f5180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_26\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.388324  115000 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_26" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.388475  115000 cacher.go:450] cacher (secrets): unexpected ListAndWatch error: failed to list *core.Secret: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_26" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.388Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002592fc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_33\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/services/endpoints/ /registry/services/endpoints/: ERROR: prepared statement \"stmtcache_42\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.401Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0026d5dc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_34\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.401Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0029d28c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_37\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.402100  115000 reflector.go:424] storage/cacher.go:/pods: failed to list *core.Pod: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_37" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.402294  115000 cacher.go:450] cacher (pods): unexpected ListAndWatch error: failed to list *core.Pod: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_37" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/minions/ /registry/minions/: ERROR: prepared statement \"stmtcache_44\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/serviceaccounts/ /registry/serviceaccounts/: ERROR: prepared statement \"stmtcache_46\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.421Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002013c00/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_35\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.421Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022f5180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_43\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.421Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0029d28c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_40\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.421Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002435c00/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_42\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.422Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002afa1c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_41\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="failed to list /registry/configmaps/ for revision 203"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.441Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0026d5dc0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_42\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.441810  115000 reflector.go:424] storage/cacher.go:/services/endpoints: failed to list *core.Endpoints: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_42" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.442000  115000 cacher.go:450] cacher (endpoints): unexpected ListAndWatch error: failed to list *core.Endpoints: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_42" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/services/specs/ /registry/services/specs/: ERROR: prepared statement \"stmtcache_49\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/controllers/ /registry/controllers/: ERROR: prepared statement \"stmtcache_50\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.461Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002afa1c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_46\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.461973  115000 reflector.go:424] storage/cacher.go:/serviceaccounts: failed to list *core.ServiceAccount: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_46" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.462230  115000 cacher.go:450] cacher (serviceaccounts): unexpected ListAndWatch error: failed to list *core.ServiceAccount: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_46" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.462Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0027fb180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_44\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/services/specs/ /registry/services/specs/: ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.463Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b44700/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_49\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.464225  115000 reflector.go:424] storage/cacher.go:/services/specs: failed to list *core.Service: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_49" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.464393  115000 cacher.go:450] cacher (services): unexpected ListAndWatch error: failed to list *core.Service: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_49" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: I0621 03:19:11.468294  115000 instance.go:621] API group "internal.apiserver.k8s.io" is not enabled, skipping.
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/minions/ /registry/minions/: ERROR: prepared statement \"stmtcache_53\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.487Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008e1c00/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_50\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.487Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b44700/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.488006  115000 reflector.go:347] storage/cacher.go:/configmaps: watch of *core.ConfigMap ended with: very short watch: storage/cacher.go:/configmaps: Unexpected watch close - watch lasted less than a second and no items received
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.527Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0027fb180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_53\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.528331  115000 reflector.go:424] storage/cacher.go:/minions: failed to list *core.Node: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_53" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.528613  115000 cacher.go:450] cacher (nodes): unexpected ListAndWatch error: failed to list *core.Node: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_53" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: ERROR: prepared statement \"stmtcache_44\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: ERROR: prepared statement \"stmtcache_29\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.631Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0027fa000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_44\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.634Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0041a9180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_29\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.635209  115000 reflector.go:424] storage/cacher.go:/horizontalpodautoscalers: failed to list *autoscaling.HorizontalPodAutoscaler: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_29" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.635482  115000 cacher.go:450] cacher (horizontalpodautoscalers.autoscaling): unexpected ListAndWatch error: failed to list *autoscaling.HorizontalPodAutoscaler: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_29" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.636Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0041a9180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.642Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00420c700/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/jobs/ /registry/jobs/: ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/cronjobs/ /registry/cronjobs/: ERROR: prepared statement \"stmtcache_29\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.705Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b45500/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_29\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.705737  115000 reflector.go:424] storage/cacher.go:/cronjobs: failed to list *batch.CronJob: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_29" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.706023  115000 cacher.go:450] cacher (cronjobs.batch): unexpected ListAndWatch error: failed to list *batch.CronJob: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_29" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: ERROR: prepared statement \"stmtcache_60\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.724Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ea7180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_52\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/certificatesigningrequests/ /registry/certificatesigningrequests/: ERROR: prepared statement \"stmtcache_56\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.764Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0027fa000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_60\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.765218  115000 reflector.go:424] storage/cacher.go:/horizontalpodautoscalers: failed to list *autoscaling.HorizontalPodAutoscaler: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_60" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.765481  115000 cacher.go:450] cacher (horizontalpodautoscalers.autoscaling): unexpected ListAndWatch error: failed to list *autoscaling.HorizontalPodAutoscaler: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_60" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: ERROR: prepared statement \"stmtcache_62\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.786Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00217f500/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_56\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/leases/ /registry/leases/: ERROR: prepared statement \"stmtcache_56\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="fail to list latest changes: ERROR: prepared statement \"stmtcache_65\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.806Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00420c700/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_62\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.807270  115000 reflector.go:424] storage/cacher.go:/horizontalpodautoscalers: failed to list *autoscaling.HorizontalPodAutoscaler: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_62" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.807483  115000 cacher.go:450] cacher (horizontalpodautoscalers.autoscaling): unexpected ListAndWatch error: failed to list *autoscaling.HorizontalPodAutoscaler: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_62" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/cronjobs/ /registry/cronjobs/: ERROR: prepared statement \"stmtcache_68\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/certificatesigningrequests/ /registry/certificatesigningrequests/: ERROR: prepared statement \"stmtcache_66\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.810Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00031e000/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_56\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/endpointslices/ /registry/endpointslices/: ERROR: prepared statement \"stmtcache_56\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.815Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b45500/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_68\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.815Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00217f500/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_66\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.816043  115000 reflector.go:424] storage/cacher.go:/certificatesigningrequests: failed to list *certificates.CertificateSigningRequest: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_66" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.816210  115000 cacher.go:450] cacher (certificatesigningrequests.certificates.k8s.io): unexpected ListAndWatch error: failed to list *certificates.CertificateSigningRequest: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_66" does not exist (SQLSTATE 26000); reinitializing...
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.842Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00031f500/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_56\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/networkpolicies/ /registry/networkpolicies/: ERROR: prepared statement \"stmtcache_44\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/jobs/ /registry/jobs/: ERROR: prepared statement \"stmtcache_70\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/ingressclasses/ /registry/ingressclasses/: ERROR: prepared statement \"stmtcache_56\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.946Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007eea80/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_44\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: time="2023-06-21T03:19:11Z" level=error msg="error while range on /registry/leases/ /registry/leases/: ERROR: prepared statement \"stmtcache_71\" does not exist (SQLSTATE 26000)"
Jun 21 03:19:11 k3s-0 k3s[115000]: {"level":"warn","ts":"2023-06-21T03:19:11.967Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ea7180/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = ERROR: prepared statement \"stmtcache_70\" does not exist (SQLSTATE 26000)"}
Jun 21 03:19:11 k3s-0 k3s[115000]: W0621 03:19:11.967822  115000 reflector.go:424] storage/cacher.go:/jobs: failed to list *batch.Job: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_70" does not exist (SQLSTATE 26000)
Jun 21 03:19:11 k3s-0 k3s[115000]: E0621 03:19:11.968016  115000 cacher.go:450] cacher (jobs.batch): unexpected ListAndWatch error: failed to list *batch.Job: rpc error: code = Unknown desc = ERROR: prepared statement "stmtcache_70" does not exist (SQLSTATE 26000); reinitializing...
brandond commented 1 year ago

Yep, looks like pgbouncer connection pools don't handle prepared statements properly. I guess we could add this to the docs as a limitation. It seems unlikely that we'll be able to add support for pgbouncer or other external pooling/aggregation layers to kine, although a PR would be welcome if you have ideas.

brandond commented 1 year ago

Huh, apparently pgbouncer just says to disable prepared statements when transaction pooling is enabled - which is terrible advice, as almost every database engine operates much, much more efficiently with prepared statements.

https://www.pgbouncer.org/faq.html

How to use prepared statements with transaction pooling? To make prepared statements work in this mode would need PgBouncer to keep track of them internally, which it does not do. So the only way to keep using PgBouncer in this mode is to disable prepared statements in the client.

I personally find this pretty brain-dead.

DillonN commented 3 months ago

Hey @brandond FYI looks like pgbouncer improved this in version 1.21.0. Your link is updated and suggests setting the max_prepared_statements to support prepared statements in transaction pooling.

I got here from googling for the same error and can confirm setting this param allows k3s to work (so far)

brandond commented 3 months ago

A docs PR would be welcome!

DillonN commented 3 months ago

Will vet it out for a bit longer then check back for that!