automatic webhook tls generation bug

dukkhadevops commented 4 days ago

What is the issue?

When running linkerd check 1 day after install I get an error about the proxy injector webhook certificate not being issued by the trust anchor. Which doesn't make sense to me because I've followed the install steps pretty closely. I had all sorts of issues with cert-manager so I even opted to go around that and just let linkerd manage things by just passing certs via the following parameters

identityTrustAnchorsPEM
identity.issuer.tls.crtPEM
identity.issuer.tls.keyPEM

I've reproduced it probably close to 10 times at this point but I just can't figure out where I've gone wrong.

Also important is that if it's a fresh install/deploy, things work fine. No issues with linkerd check on day 1. Its like the certs get renewed and aren't getting the right certificate somehow? Maybe?

How can it be reproduced?

Starting with the install docs

The helm charts I'm using are below. I've tried some of the latest stable releases as well but had issues with the various linkerd pods coming up for whatever reason so I switched back to edge. I've tried some of the October ones and now the latest one I grabbed yesterday as well but no matter which version I've tried the result is the same

"helm_repository_url": "helm.linkerd.io/edge",
"helm_chart_path": "linkerd-control-plane",
"helm_chart_version": "2024.11.3"
"helm_chart_path": "linkerd-crds",
"helm_chart_version": "2024.11.3"

Here is how we are pushing things - via an ArgoCD application and passing the helm chart the certificates as parameters. I created them via the linkerd documentation with step commands.

gist link - https://gist.github.com/dukkhadevops/f1fb7ff21ae97e98158831ff474984a2

Logs, error output, etc

When running linkerd check the next day after install I get this error

linkerd-webhooks-and-apisvc-tls × proxy-injector webhook has valid cert cert is not issued by the trust anchor: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "linkerd-proxy-injector.linkerd.svc") see https://linkerd.io/2/checks/#l5d-proxy-injector-webhook-cert-valid for hints

output of `linkerd check -o short`

linkerd-webhooks-and-apisvc-tls × proxy-injector webhook has valid cert cert is not issued by the trust anchor: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "linkerd-proxy-injector.linkerd.svc") see https://linkerd.io/2/checks/#l5d-proxy-injector-webhook-cert-valid for hints

Environment

kubernetes version = 1.30.4
cluster env = AKS
linkerd version = edge-24.11.3

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None

olix0r commented 2 days ago

This cert is configured here:

https://github.com/linkerd/linkerd2/blob/0ecc74c6477cbafb866bd50c652f2b3d0c474642/charts/linkerd-control-plane/templates/proxy-injector-rbac.yaml#L59-L76

If you haven't configured it manually (it sounds like you have not?), then it should be generated by helm. Can you share the value of the tls.crt in your secret?

dukkhadevops commented 2 days ago

This cert is configured here:

https://github.com/linkerd/linkerd2/blob/0ecc74c6477cbafb866bd50c652f2b3d0c474642/charts/linkerd-control-plane/templates/proxy-injector-rbac.yaml#L59-L76

If you haven't configured it manually (it sounds like you have not?), then it should be generated by helm. Can you share the value of the tls.crt in your secret?

We are not configuring these manually no. The additional complexity of cert-manager & trying to set these sent me down a rabbit hole of wasted time which is why I'm trying to simplify things and just provide the bare minimum from the helm install docs.

I went ahead and reinstalled today to test something else but heres the cert from the secret. Worth noting it is working currently because I just reinstalled today (with the expectation it should be broken at some point tomorrow if other changes didn't fix things...)

kubectl get secret linkerd-proxy-injector-k8s-tls -n linkerd -o jsonpath='{.data.tls\.crt}' | base64 --decode

dukkhadevops commented 31 minutes ago

For anyone else who stumbles across this it looks like my issue was resolved by making sure these are in place which I pulled from the examples given in the linkerd gitops documentation - https://github.com/linkerd/linkerd-examples

ignoreDifferences:
  - kind: Secret
    name: linkerd-proxy-injector-k8s-tls
    jsonPointers:
      - /data/tls.crt
      - /data/tls.key
  - kind: Secret
    name: linkerd-sp-validator-k8s-tls
    jsonPointers:
      - /data/tls.crt
      - /data/tls.key
  - kind: Secret
    name: linkerd-policy-validator-k8s-tls
    jsonPointers:
      - /data/tls.crt
      - /data/tls.key
  - group: admissionregistration.k8s.io/
    kind: MutatingWebhookConfiguration
    name: linkerd-proxy-injector-webhook-config
    jsonPointers:
      - /webhooks
  - group: admissionregistration.k8s.io/
    kind: ValidatingWebhookConfiguration
    name: linkerd-sp-validator-webhook-config
    jsonPointers:
      - /webhooks
  - group: admissionregistration.k8s.io/
    kind: ValidatingWebhookConfiguration
    name: linkerd-policy-validator-webhook-config
    jsonPointers:
      - /webhooks

linkerd / linkerd2