When I install the latest version(0.2.18) on my cluster through HELM, I found the status is pending-install and after the pod is in CrashLoopBackOff status.
Here I go into the container named and try to call the livenessProbe address, I got 500 response code:
/ # curl -kv [https://localhost:443/healthz](https://localhost/healthz)
* Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: [NONE]
* start date: Aug 23 02:50:14 2022 GMT
* expire date: Aug 23 02:50:14 2023 GMT
* issuer: CN=ca.designate-certmanager-webhook.cert-manager
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5563b4258260)
> GET /healthz HTTP/2
> Host: localhost
> user-agent: curl/7.79.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 1000)!
< HTTP/2 500
< audit-id: 70b1d8ab-9d2e-47fe-876f-ecbb497fea26
< cache-control: no-cache, private
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 269
< date: Tue, 23 Aug 2022 08:15:06 GMT
<
[+]ping ok
[+]log ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[-]poststarthook/solver-designateDNS-init failed: reason withheld
healthz check failed
Here is the logs of the pod:
I0823 09:06:18.674627 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0823 09:06:18.674595 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0823 09:06:18.674674 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0823 09:06:18.674584 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0823 09:06:18.674675 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0823 09:06:18.675112 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0823 09:06:18.675431 1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/tls/tls.crt::/tls/tls.key"
I0823 09:06:18.675702 1 secure_serving.go:266] Serving securely on [::]:443
I0823 09:06:18.675754 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
time="2022-08-23T09:06:18Z" level=info msg="Using OpenStack Keystone at https://xxxxxxxx.xxxxxx.xxxxxx.xxx/v3/"
I0823 09:06:18.676684 1 apf_controller.go:317] Starting API Priority and Fairness config controller
W0823 09:06:18.679277 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:18.679315 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:18.679324 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:18.679340 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
I0823 09:06:18.774810 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0823 09:06:18.775308 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0823 09:06:18.775328 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
W0823 09:06:19.824338 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:19.824372 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:19.833831 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:19.833855 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:21.542106 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:21.542142 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:22.526426 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:22.526460 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:25.246408 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:25.246439 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:26.040194 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:26.040232 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:33.579107 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:33.579142 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0823 09:06:35.740597 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0823 09:06:35.740635 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cic-system:monsoon-cert-foundation-designate-certmanager-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
I0823 09:06:47.023965 1 object_count_tracker.go:84] "StorageObjectCountTracker pruner is exiting"
I0823 09:06:47.024013 1 tlsconfig.go:255] "Shutting down DynamicServingCertificateController"
I0823 09:06:47.024017 1 configmap_cafile_content.go:222] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0823 09:06:47.024084 1 requestheader_controller.go:183] Shutting down RequestHeaderAuthRequestController
I0823 09:06:47.024107 1 dynamic_serving_content.go:145] "Shutting down controller" name="serving-cert::/tls/tls.crt::/tls/tls.key"
I0823 09:06:47.024115 1 configmap_cafile_content.go:222] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0823 09:06:47.024090 1 secure_serving.go:311] Stopped listening on [::]:443
Hi,
When I install the latest version(0.2.18) on my cluster through HELM, I found the status is pending-install and after the pod is in CrashLoopBackOff status.
Kubernetes version: 1.22.11 cert-manager version: v1.9.1
Here I go into the container named and try to call the livenessProbe address, I got 500 response code:
Here is the logs of the pod: