search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.16k stars 4.91k forks source link

The async Logstash client does not support the "ttl" option #39127

Open mbudge opened 6 months ago

mbudge commented 6 months ago

Hi,

Any idea when TTL will be supported with the async client?

"The ttl option is not yet supported on an asynchronous Logstash client (one with the pipelining option set)"

https://www.elastic.co/guide/en/fleet/current/logstash-output.html

Thanks

pierrehilbert commented 6 months ago

@nimarezainia could you have a look here please?

elasticmachine commented 6 months ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

nimarezainia commented 5 months ago

@mbudge you should be able to use those configurations in the advanced yaml section of the Logstash. I believe the options should be available. @cmacknz please keep me honest.

we can look at adding the UI to the output to configure these extra options.

cmacknz commented 5 months ago

It isn't supported in the output itself, it's not a matter of being able to provide the configuration or not.

https://github.com/elastic/beats/blob/8f8f3137f2ed1a78a9db4d48831d6578831619f8/libbeat/outputs/logstash/async.go#L76-L79

cmacknz commented 5 months ago

The documentation on the TTL option from https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_ttl:

Time to live for a connection to Logstash after which the connection will be re-established. Useful when Logstash hosts represent load balancers. Since the connections to Logstash hosts are sticky, operating behind load balancers can lead to uneven load distribution between the instances. Specifying a TTL on the connection allows to achieve equal connection distribution between the instances. Specifying a TTL of 0 will disable this feature.

The default value is 0. This setting accepts duration data type values.

nimarezainia commented 5 months ago

Yeah apologies. @mbudge wrote that also in the description, totally missed it.

ar3diu commented 5 months ago

So... is it just an error message and the TTL should be applied even if pipelining is enabled? Or the TTL setting is simply not supported in Logstash Output?

cmacknz commented 5 months ago

The TTL setting is ignored and does nothing when in async mode. In sync mode it works as described.

ar3diu commented 5 months ago

The TTL setting is ignored and does nothing when in async mode. In sync mode it works as described.

And how should the async mode be disabled? pipelining: 0?

cmacknz commented 5 months ago

Yes per https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_pipelining

https://github.com/elastic/beats/blob/a4f576f34928669b821a0243af00b35d8d74918e/libbeat/outputs/logstash/logstash.go#L75-L79

ar3diu commented 5 months ago

I tried to set the following for the Logstash output in Fleet, but I still get some errors in regards to agent connections to the Logstash hosts. Any idea what could cause them?

pipelining: 0
ttl: 30s
error closing connection to logstash host <redacted>: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp <redacted>-><redacted>: wsasend: An established connection was aborted by the software in your host machine., reconnecting...
daniel-vasek commented 3 months ago

I think the answer has not been answered. Setting the TTL is the only option how to evenly distribute the load through the load balancer (btw it's the default way how to load balance in Kubernetes). Setting pipelining to 0 reduces the throughput of the logging pipeline significantly, so the possibility of using TTL with pipelining would be super useful for those who need to deliver huge number of logs.