Logstash RELP Input via HAProxy Loadbalancer (Renegotiation issues)

[SuperFunks]

Using latest Rsyslog 8.26 RELP Output - towards 2 logstash 5.3 nodes listening with latest RELP Input - balanced with HAProxy 1.7 TCP loadbalancing i get tons of these errors:

[2017-04-18T12:38:03,084][WARN ][logstash.inputs.relp ] Relp error: Relp::InappropriateCommand open expecting syslog

It seems the input module isnt understanding a failover scenario where 1 logstash node is no longer available (renegotiation). I also get log entries stating the other way around, where it extected open but got syslog.

In order to give RELP high availability - i would like it to be a supported option to loadbalance the RELP protocol.

Best regards,

SuperFunks

[SuperFunks]

I believe i found out why this was an issue!

The HAProxy was configured to send keepalives to the client with the following option:

option clitcpka

Reading through the HAProxy documentation - i decided to disable it on the frontend binding - and i nolonger see the "InappropriateCommand" log issues - and the RELP log flow is stabilized even when failovers occur.

Something to remember when using HAProxy together with RELP.

Best regards,

SuperFunks

[SuperFunks]

After working on more failover scenarios - i still get the "InappropriateCommand" loops - so maybe it is still worth looking into improving the renegotiation in the input module.

[SuperFunks]

I looked into the HAProxy and Rsyslog options on timing and found that if HAProxy has a lower/equal timeout to Rsyslog RELP timeout - the failover can happen succesfully:

HAProxy Settings: timeout connect 2000 timeout server 2000 timeout client 2000

Rsyslog Settings: timeout="2" conn.timeout="2" action.ResumeInterval="5" action.ResumeRetryCount="-1"

So far this is working for me - both for VIP failover and logstash shutdowns.