/var/run/secrets/kubernetes.io/serviceaccount/ca.crt missing ?

[suau]

I'm trying to setup fluentbit for kubernetes but I'm getting an error with the certificates: "X509 - Certificate verification failed, e.g. CRL, CA or signature check".
fluentbit's kuberentes filter accesses the Kubernetes API server to pull some data and expects the servers cert.ca file at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but that doesn't seem to be the case on digitalocean. Disabling tls verification works, but isn't really the option I'm hoping for.

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS}
        Merge_Log           On
        K8S-Logging.Parser  On
        tls.debug           -1
        tls.verify          Off

Now I'm unsure what's going on here, even with tls.debug 4 it doesn't really show me if there is a server-cert mismatch or if the cert file is missing.

[timoreimann]

@suau both the variables and the CA certificate should be injected into pods running on DOKS. Here's the output of a test run on a fresh 1.13 cluster created:

$ kubectl run -it --restart=Never test --image=ubuntu /bin/bash
If you don't see a command prompt, try pressing enter.
root@test:/# echo $KUBERNETES_SERVICE_HOST
10.245.0.1
root@test:/# echo $KUBERNETES_SERVICE_PORT_HTTPS
443
root@test:/# ls /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Does your pod have a service account associated? This should be the case by default unless I think you have disabled it explicitly through something like the automountServiceAccountToken field.

[suau]

@timoreimann Yes I do use a service account and I checked the CA cert as you suggested and everything was present.

So I changed
Kube_URL https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS}
To https://kubernetes.default.svc.cluster.local:443 (which is actually the default)
And it worked ! So I guess the cert is signed for the hostname, but not for the IP. I'm not sure if it is even possible to sign it for the IP as it's not a public one, what is the expected behavior here ?

If it is expected to be signed for the hostname only, then fluentbit's default kubernetes configmap makes no sense.