Dopingus / cert-manager-webhook-dynu

Unofficial cert-manager webhook for dynu
Apache License 2.0
7 stars 7 forks source link

re-queuing item due to error processing" "error"="dynu.dynu-webhook-1661649439 is forbidden #7

Closed genwch closed 2 years ago

genwch commented 2 years ago

I got issues when using this webhook in my k3s server. It would be great if anyone can confirm my steps or help to solve my issue.

  1. may be something missing in helm install command

    helm install cert-manager-dynu-webhook/dynu-webhook Error: INSTALLATION FAILED: must either provide a name or specify --generate-name the problem will be solved if I add -g / --generate-name, but I am not sure will it impact to the following issue. afterward, I capture the name (dynu-webhook-1661649439) from the helm install as the groupName value in ClusterIssuer yaml

  2. I try to capture the logs from the cert-manager pods, but get the issue about the rbac. but I am the newbie of kube, it would be great if someone can show me how to use the "template" rbac to solve the problem.

    E0828 02:32:57.174888 1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="dynu.dynu-webhook-1661649439 is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot create resource \"dynu\" in API group \"dynu-webhook-1661649439\" at the cluster scope" "key"="default/

so sorry for my poor English. thank you

anon-software commented 2 years ago

There are a couple of errors in the README file, I am not sure if that is causing your first problem. First, you want to use the correct repository address when adding it:

helm repo add cert-manager-dynu-webhook https://dopingus.github.io/cert-manager-webhook-dynu

Next, use com.github.dopingus.cert-manager-webhook-dynu as the groupName when defining an issuer.

Which pod's log contain the second error message?

Dopingus commented 2 years ago

The error message is probably coming from cert-manager itself. I think this happens when the service account used by cert-manager is not able to create the required K8s resources.

I'll have to check this on my own cluster to find out which permissions are missing exactly. Can you please add which version of K8s you're running?

genwch commented 2 years ago

There are a couple of errors in the README file, I am not sure if that is causing your first problem. First, you want to use the correct repository address when adding it:

helm repo add cert-manager-dynu-webhook https://dopingus.github.io/cert-manager-webhook-dynu

Next, use com.github.dopingus.cert-manager-webhook-dynu as the groupName when defining an issuer.

Which pod's log contain the second error message?

the log is from the pod of cert-manager

same error occurred, when I check the log with below command k logs -n cert-manager cert-manager-58dd98c6d6-8t78m -f

E0829 10:14:38.762585 1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="dynu.com.github.dopingus.cert-manager-webhook-dynu is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot create resource \"dynu\" in API group \"com.github.dopingus.cert-manager-webhook-dynu\" at the cluster scope" "key"="default/

genwch commented 2 years ago

The error message is probably coming from cert-manager itself. I think this happens when the service account used by cert-manager is not able to create the required K8s resources.

I'll have to check this on my own cluster to find out which permissions are missing exactly. Can you please add which version of K8s you're running?

k version --output=json

{ "clientVersion": { "major": "1", "minor": "25", "gitVersion": "v1.25.0", "gitCommit": "a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", "gitTreeState": "clean", "buildDate": "2022-08-23T17:44:59Z", "goVersion": "go1.19", "compiler": "gc", "platform": "linux/amd64" }, "kustomizeVersion": "v4.5.7", "serverVersion": { "major": "1", "minor": "23", "gitVersion": "v1.23.4+k3s1", "gitCommit": "43b1cb48200d8f6af85c16ed944d68fcc96b6506", "gitTreeState": "clean", "buildDate": "2022-02-24T22:38:17Z", "goVersion": "go1.17.5", "compiler": "gc", "platform": "linux/amd64" } }

genwch commented 2 years ago

I fixed the issue by amend the ClusterRoleBinding (domain-solver & secret-reader) manually.

from

  • apiGroup: "" kind: ServiceAccount name: {{ .Values.certManager.serviceAccountName }} namespace: {{ .Release.Namespace }}

to

  • apiGroup: "" kind: ServiceAccount name: {{ .Values.certManager.serviceAccountName }} namespace: cert-manager

I am not sure why get the correct namespace of cert-manager by using " {{ .Release.Namespace }} "

But I found another issue

k logs -n cert-manager cert-manager-f7749dbb4-2zxbn

E0903 01:11:50.348135 1 sync.go:190] cert-manager/challenges "msg"="propagation check failed" "error"="DNS record for \"xxx.xxx.xxx\" not yet propagated"

k describe challenge xxx

Spec: Authorization URL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3457012774 Dns Name: xxx Issuer Ref: Kind: ClusterIssuer Name: letsencrypt-staging Key: xxx Solver: dns01: Cname Strategy: Follow Webhook: Config: Secret Name: dynu-secret Group Name: com.github.dopingus.cert-manager-webhook-dynu Solver Name: dynu Token: xxxx Type: DNS-01 URL: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3457012774/mqXYWg Wildcard: true Status: Presented: true Processing: true Reason: Waiting for DNS-01 challenge propagation: DNS record for "xxx" not yet propagated State: pending Events:

masked some personal Information by xxx

anon-software commented 2 years ago

I installed the web hook into the same namespace as cert-manager itself, that was why I did not have this problem.

Hardcoding the namespace does not appeal like a great idea to me. I looked back into the file's history and found the commit where the change was introduced. Can you try putting {{ .Values.certManager.namespace }} instead of cert-manager? If that works we can restore it the way if was before. Using this variable will be a much better solution.

Regarding the new problem, I suggest you check DNS records immediately after the webhook is invoked by cert-manager. Click on "Manage DNS records for hostname" in Dynu console next to the domain you are requesting a certificate for. I remember seeing the challenge records there for a short period until they were propagated to where Let's Encrypt could see them.

rbaumgar commented 2 years ago

@anon-software agree with you. I made the change from the fixed namespace name, because you should never use fixed names!

genwch commented 2 years ago

I have tried to install the webhook to the same namespace of cert-manager and everything good. thank you all for your help.