Failed to connect to elasticsearch after elasticsearch been upgraded

[jerry153fish]

Bug Report

Describe the bug

After upgrading elasticsearch in AWS, the fluentbit Pods seems to stop talking to it. The error message is timeout to connect to xxx (URL)

After rollout restarted, all the fluentbit pods can reconnect again.

To Reproduce

• configure fluentbit shiping log to elasticsearch in AWS ( Connect will be OK )
• upgrade the elastic search cluster ( maybe trigger the DNS switch )
• Pod stop sending logs to elasticsearch reporting connection timeout

Your Environment

• Version used: latest
• Configuration:

  [INPUT]
  Name              tail
  Tag               xxx
  Path             xxx
  Exclude_Path      /var/log/containers/*istio*.log
  Parser            cdocker
  DB                /var/log/flb_kube.db
  Mem_Buf_Limit     5MB
  Skip_Long_Lines   On
  Refresh_Interval  10
  [FILTER]
  Name modify
  Match xxx
  Add xxx
  Remove stream
  [FILTER]
  Name             kubernetes
  Match            es.*
  Kube_URL         https://kubernetes.default.svc:443
  Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
  Merge_Log        On
  Keep_Log         Off
  [OUTPUT]
  Name  es
  Match es.*
  Host  xxx
  Port  443
  Logstash_Format True
  Logstash_Prefix cwl-log-fra-prod
  Logstash_DateFormat %Y.%m
  AWS_Auth On
  AWS_Region us-west-2
  tls     On

• Environment name and version : eks 1.19
• Operating System and version: eks nodegroup 19.6

Maybe in somewhere, we are caching the ip ?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.