Haproxy connection reset

[tmanninger]

Describe the bug We are using haproxy to load balance our traffic to different elasticsearch- nodes in our cluster. Haproxy sends a RST package to the elastic-node after the connection is closed by our client.

Opendistro logs a Connection Reset as ERROR in the elasticsearch.log, but i should be logged as DEBUG mesage.

See: https://stackoverflow.com/questions/21550337/haproxy-netty-way-to-prevent-exceptions-on-connection-reset

[2021-05-21T09:07:08,397][ERROR][c.a.o.s.s.h.n.OpenDistroSecuritySSLNettyHttpServerTransport] [pay-prod-sandboxelasservice-esh-01.in.ixopay.com] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:367) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:398) ~[?:?]
        at org.elasticsearch.transport.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:141) ~[transport-netty4-client-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:126) ~[transport-netty4-client-7.10.2.jar:7.10.2]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]

To Reproduce Steps to reproduce the behavior:

1. Install haproxy
2. Add elasticsearch as backend server
3. Send http request to the elasticsearch over the haproxy sever
4. Check elasticsearch logs. You can see the "Connection Reset" "error"

Expected behavior Connection reset should be logged AS DEBUG

Configuration (please complete the following information):

• Opendistro 1.13.1.0
• Distribution DEB
• Host Machine Debian 7

Relevant information Please include any relevant log snippets or files here.