search

linkerd / linkerd2

Ultralight, security-first service mesh for Kubernetes. Main repo for Linkerd 2.x.
https://linkerd.io
Apache License 2.0
10.68k stars 1.28k forks source link

daemon:identity: linkerd_proxy_identity::certify: Received invalid certificate: invalid certificate: UnknownIssuer #6428

Closed rumanzo closed 3 years ago

rumanzo commented 3 years ago

Bug Report

I try to setup linkerd using helm chart chart version 2.10.2. I use cert-manager with vault issuer (setup described here). Values is close to values from helm-chart. I get ca.crt file like this

$ kubectl get secret -n linkerd linkerd-identity-issuer -o json | jq -r '.data."ca.crt"' | base64 -d > ca.crt

and use this values:

identity:
  issuer:
    scheme: kubernetes.io/tls

But I still cant setup linkerd2 to my cluster. Linkerd identity start to work, but linkerd-proxy don't. I get errors in linkerd-proxy container in linkerd-identity pod until container shutdown:

$ kubectl -n linkerd  logs linkerd-identity-6cdcc8f5-lcpvq -c linkerd-proxy -f
time="2021-07-02T10:30:28Z" level=info msg="running version stable-2.10.2"
[     0.003677s]  INFO ThreadId(01) linkerd2_proxy::rt: Using multi-threaded proxy runtime cores=2
[     0.004835s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.004851s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.004856s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.004860s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.004865s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.004870s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via localhost:8080
[     0.004874s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.014370s] ERROR ThreadId(04) daemon:identity: linkerd_proxy_identity::certify: Received invalid certificate: invalid certificate: UnknownIssuer
[    10.021287s] ERROR ThreadId(04) daemon:identity: linkerd_proxy_identity::certify: Received invalid certificate: invalid certificate: UnknownIssuer
[    20.028114s] ERROR ThreadId(04) daemon:identity: linkerd_proxy_identity::certify: Received invalid certificate: invalid certificate: UnknownIssuer
...
$ kubectl get secret -n linkerd linkerd-identity-issuer -o json | jq -r '.data."ca.crt"' | base64 -d > ca.crt
$ kubectl get secret -n linkerd linkerd-identity-issuer -o json | jq -r '.data."tls.crt"' | base64 -d > tls.crt
$ openssl verify -verbose -CAfile ca.crt tls.crt
tls.crt: OK

I can't use linkerd2 utility (and linkerd2 don't actually work with my cluster due enabled psp and taints), and I don't know how to debug it.

rumanzo commented 3 years ago

I found that I used wrong path for vault issuer, and got non CA certificates with wrong usages. Linkerd error message gave me nothing, but finally I found a problem. This part of linkerd + cert-manager + vault integration very poorly described in the documentation

alpeb commented 3 years ago

Indeed, I could reproduce the problem using your setup. I've raised #6453 so that we check those certs are not leafs and properly issue a warning.

However, Vault ignoring the Issuer's isCA config is indeed problematic and should raise a warning on their side as well. But I think the Vault Cert-manager issuer config docs is out of scope in linkerd's docs, since that's totally orthogonal to linkerd's operation, and would unnecessarily duplicate info with what they have. I would love to know though, out of curiosity, what you found out to be the correct path for issuing intermediate certs :slightly_smiling_face:

rumanzo commented 3 years ago

Indeed, I could reproduce the problem using your setup. I've raised #6453 so that we check those certs are not leafs and properly issue a warning.

Thank you. It will be very helpful

config docs is out of scope in linkerd's docs

You are right. I'll try to influence this.

I would love to know though, out of curiosity, what you found out to be the correct path for issuing intermediate certs 🙂

I have build debug version of cert-manager and saw that CSR is correct, as other query fields, and looked to vault documentation, and found this, I change path in cert-manager vault issuer and policy paths in vault, and got correct intermediate-CA with required fields and usages. And finally the linkerd2 worked :-)

alpeb commented 3 years ago

Excellent! Thanks for your feedback, this will serve as reference for anyone else attempting to do the same :+1:

alpeb commented 3 years ago

@rumanzo btw, if you'd like to contribute back to linkerd, #6453 would be a nice and isolated place to start. I'd be happy to answer any questions :wink: :wink:

cortopy commented 3 years ago

@rumanzo thanks so much for this! It really helped me a lot.

For anybody who lands here and is trying to configure a vault CA, bear in mind that the cert-manager docs use the pki/sign/<role> endpoint, which is documented by vault here as a certificate generator. However, as per @rumanzo findings, one actually needs an intermediate certificate, which is what the /pki/root/sign-intermediate endpoint does

It would look something like this:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: sandbox
spec:
  vault:
    path: /pki/root/sign-intermediate
    server: https://vault.local
    caBundle: <base64 encoded CA Bundle PEM file>
    auth:
      ...