search

fluent / helm-charts

Helm Charts for Fluentd and Fluent Bit
Apache License 2.0
374 stars 443 forks source link

Fluentd daemonset not starting and throwing liveness and readiness probe error #370

Open monduofficial opened 1 year ago

monduofficial commented 1 year ago

I have been facing this issues since last month. I have recently upgraded the k8s from 1.24.6 to 1.25.6 using kubespray. After that the fluentd pod keeps on restarting and error logs doesn't say much. I have just tried removing the readiness probe and liveness probe from the daemonset deployment file and its working fine now. Can someone explain me what could be the real cause or can we expect an update in the helm-charts related to the same as i am using helm-charts.

Regards Mondu

Maxime-Carlier commented 1 year ago

Hello, I had the same problem following an upgrade from chart 0.3.9 to 0.4.3.

Long story short, the /metrics endpoint used by the probes was not reachable. Adding the following to my values files (which is no longer present in the default values of 0.4.3 if you compare both versions and check charts/fluentd/values.yaml) fixed my issue.

I hope it also solves your problem.

configMapConfigs:
  - fluentd-prometheus-conf
monduofficial commented 1 year ago

Thanks Maxime,

I have checked my helm values and i could see that i already have the below configuration :(

Add fluentd config files from K8s configMaps

configMapConfigs:

Regards Mondu

discostur commented 1 year ago

Adding the configmap option to the chart from @Maxime-Carlier fixed it for me ;) Thanks

vinerich commented 1 year ago

This was introduced by https://github.com/fluent/helm-charts/commit/09607ddf83eec00645bcdcac2a9534c23d48ec7a#diff-2653b619946787bdcdc2caaab6175bfcd27c06e2c1939725a2577119b005b7dcL296 and I believe should be reverted.

Atleast if the liveness and readiness probes stay this way, the default daemonset deployment will never work as the /metrics endpoint is not reachable.

I can provide a PR if needed. So if a maintainer can give a quick thumbs up I'll be happy to fix that.

bertinmeshita-dt commented 1 year ago

Adding the configmap option to the chart fixed it for me as well. This should be reverted ASAP.

mflopez79 commented 10 months ago

One day of debugging because of this issue. Thanks @Maxime-Carlier for the comment!