search

fluent / fluent-bit-kubernetes-logging

Fluent Bit Kubernetes Daemonset
Apache License 2.0
468 stars 248 forks source link

v1.6.7 seems to have broken Kubernetes API connection #86

Open hiiamelliott opened 3 years ago

hiiamelliott commented 3 years ago

I have a daemonset of fluent-bit v1.6 pods running in three different (EKS) Kubernetes clusters. For historical reasons, I have a cronjob to restart them every 6 hours, and after they restarted this morning, they could not contact https://kubernetes.default.svc:443 in any cluster. After doing a bit of digging, I spotted that my daemonset was configured to 

      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.6
        imagePullPolicy: Always

and that v1.6.7 was released a few hours ago. Pinning the version to v1.6.6 resolved the issue, so there seems to be a bug in the new version.

The specific errors I was seeing in fluent-bit's logs were:

[2020/12/03 10:39:42] [error] [io] connection #42 failed to: kubernetes.default.svc:443
[2020/12/03 10:39:42] [error] [filter:kubernetes:kubernetes.0] upstream connection error
shinebayar-g commented 3 years ago

Fixed in 1.6.7 https://github.com/fluent/fluent-bit/issues/2829

Unfortunately 1.6.8 also bugged https://github.com/fluent/fluent-bit/issues/2830