search

fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.85k stars 1.58k forks source link

Upstream load balancing with kubernetes headless service #864

Closed Misterhex closed 6 years ago

Misterhex commented 6 years ago

Does the new upstream feature works with kubernetes headless service? How could we check where is fluentbit sending to which pod ip?

Thanks for the help

[Upstream]
name forward_balancing

[Node]
Host headless-service.cluster.local <- resolve to multiple A records of fluentd pods
Port 24434
edsiper commented 6 years ago

I am not familiar with headless service concept in Kubernetes, would you please elaborate?

Note that the upstream feature (balance connections to different nodes) is only available in Forward output plugin for now.

Misterhex commented 6 years ago

Hi @edsiper , headlesss service in kubernetes is a dns hostname that resolve to multiple ip addresses. These ip addresses points to fluentd instances that accept in_forward.

Misterhex commented 6 years ago

Tried in my own lab setup and confirmed the followings load balance correctly.

1) upstream server with single node hostname that resolves multiple ip. ( headless service in k8 ) 2) out_forward with hostname that resolves multiple ip. 3) out_forward with hostname to k8 service load balancer without session affinity.

bobykus31 commented 6 years ago

Does it failover to second ip if fluentd on one of ip addresses not responding?

ZPerling commented 1 year ago

out_forward with hostname to k8 service load balancer without session affinity.

I found that in this case a long connection is created between flunt-bit and fluentd, the connection is not reload balanced after it is established, and if a new instance of fluntd is expanded, the traffic is not forwarded to the new instance

@Misterhex