puzzle / cert-manager-webhook-dnsimple

A cert-manager ACME DNS01 solver webhook for DNSimple.
Apache License 2.0
19 stars 24 forks source link

cert-manager-webhook-dnsimple throws errors on GKE running Kubernetes 1.17.15-gke.800 #8

Closed parmus closed 2 years ago

parmus commented 3 years ago

Deploying cert-manager-webhook-dnsimple on GKE cluster running Kubernetes 1.17.15-gke.800 result in the follow errors:

I0119 23:04:51.030632       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0119 23:04:51.030689       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0119 23:04:51.030723       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0119 23:04:51.030721       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0119 23:04:51.030725       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0119 23:04:51.030818       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0119 23:04:51.031057       1 dynamic_serving_content.go:130] Starting serving-cert::/tls/tls.crt::/tls/tls.key
I0119 23:04:51.032297       1 secure_serving.go:197] Serving securely on [::]:443
I0119 23:04:51.032772       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0119 23:04:51.033118       1 apf_controller.go:249] Starting API Priority and Fairness config controller
E0119 23:04:51.036000       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:51.036045       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
I0119 23:04:51.130931       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0119 23:04:51.130993       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I0119 23:04:51.131026       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
E0119 23:04:52.179462       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:52.188664       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:53.896237       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:54.880564       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:57.600281       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:58.393101       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:05:05.928899       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:05:08.093017       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope

The problem seems to be a mismatch between the v0.20.0 client libraries and Kubernetes 1.17. Downgrading to the v0.19.0 client libraries solves the problem. Kubernetes 1.17.15-gke.800 is the latest stable release on GKE, so upgrading the cluster is not an option for users running production environments on stable.

cert-manager-webhook-dnsimple seems to work despite these errors, but on the other hand, there is nothing in cert-manager-webhook-dnsimple that needs the newer client libraries.

arnediekmann commented 3 years ago

Hey, I just saw this error in our clusters an google'd it and then ended here in the ticket of my own project :sweat_smile: - this totally got lost in my inbox. Sorry! I think a downgrade is the most viable solution, but I will check what the other webhooks are doing and will report back.

ebrianne commented 3 years ago

Looking around I noticed that one can add in the rbac yaml file to the ClusterRole :domain-solver (from https://github.com/gattytto/cert-manager-acme-he-webhook/blob/master/deploy/acme-he-webhook/templates/rbac.yaml)

  - apiGroups:
      - "flowcontrol.apiserver.k8s.io"
    resources:
      - 'prioritylevelconfigurations'
      - 'flowschemas'
    verbs:
      - 'list'
      - 'watch'

But it seems to still not solve the problem. I get the same issue ebrianne/cert-manager-webhook-duckdns#2

parmus commented 3 years ago

@ebrianne Which Kubernetes version are you seeing this with?

parmus commented 3 years ago

@arnediekmann Any news?

parmus commented 3 years ago

Just to be clear: FlowSchema and PriorityLevelConfiguration were in Alpha in Kubernetes v1.19 (https://v1-19.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#flowschema-v1alpha1-flowcontrol-apiserver-k8s-io and https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#prioritylevelconfiguration-v1alpha1-flowcontrol-apiserver-k8s-io). They don't enter Beta until v1.20. So any client library that tries to list those resources in the Beta namespace will fail on a pre-1.20 cluster.

ebrianne commented 3 years ago

@parmus I am using a k3s cluster v1.20.4 and thought I could solve the problem finally. Last time I downgraded to 1.19 for the client which solved the problem as indeed the feature was in alpha at that time. It seems at the moment a viable solution.

arnediekmann commented 3 years ago

Whoops, this got auto-closed by my commit. I just release version 0.1.0. In our clusters (Version 1.19.8) 11fb703 and the release seem to do the trick. But please do check in your environments and report back. Sorry for taking so long with this and thanks for your patience :innocent:

arnediekmann commented 3 years ago

@parmus somewhat off-topic but the release also encompasses your PRs. Thanks again for those contributions!

deyaeddin commented 3 years ago

I found this issue when I was searching solution for the same problem, I solved it by adding new clusterRole/Binding to the webhook service account (not the cert-manager service account ) ...like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: {{ include "cert-manager-webhook-hetzner.fullname" . }}:flowcontrol-solver
  labels:
    app: {{ include "cert-manager-webhook-hetzner.name" . }}
    chart: {{ include "cert-manager-webhook-hetzner.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
rules:
  - apiGroups:
      - "flowcontrol.apiserver.k8s.io"
    resources:
      - 'prioritylevelconfigurations'
      - 'flowschemas'
    verbs:
      - 'list'
      - 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: {{ include "cert-manager-webhook-hetzner.fullname" . }}:flowcontrol-solver
  labels:
    app: {{ include "cert-manager-webhook-hetzner.name" . }}
    chart: {{ include "cert-manager-webhook-hetzner.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {{ include "cert-manager-webhook-hetzner.fullname" . }}:flowcontrol-solver
subjects:
  - apiGroup: ""
    kind: ServiceAccount
    name: {{ include "cert-manager-webhook-hetzner.fullname" . }}
    namespace: {{ .Release.Namespace | quote }}
---