[CI] Netty4HttpServerTransportTests testChannelAcceptorCannotTamperThreadContext failing

[slobodanadamovic]

Build scan: https://gradle-enterprise.elastic.co/s/yqf7kpf3qc4jg/tests/:modules:transport-netty4:test/org.elasticsearch.http.netty4.Netty4HttpServerTransportTests/testChannelAcceptorCannotTamperThreadContext

Reproduction line:

gradlew ':modules:transport-netty4:test' --tests "org.elasticsearch.http.netty4.Netty4HttpServerTransportTests.testChannelAcceptorCannotTamperThreadContext" -Dtests.seed=F5D355E8719865F5 -Dtests.locale=hu -Dtests.timezone=Asia/Atyrau -Druntime.java=21

Applicable branches: main

Reproduces locally?: No

Failure history: Failure dashboard for org.elasticsearch.http.netty4.Netty4HttpServerTransportTests#testChannelAcceptorCannotTamperThreadContext&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testChannelAcceptorCannotTamperThreadContext'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.http.netty4.Netty4HttpServerTransportTests'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure excerpt:

java.lang.AssertionError: java.io.IOException: An established connection was aborted by the software in your host machine

  at __randomizedtesting.SeedInfo.seed([F5D355E8719865F5]:0)
  at org.elasticsearch.http.netty4.Netty4HttpClient$CountDownLatchHandler$1.exceptionCaught(Netty4HttpClient.java:210)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
  at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
  at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)
  at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
  at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)
  at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
  at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
  at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
  at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
  at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
  at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: java.io.IOException: An established connection was aborted by the software in your host machine

    at sun.nio.ch.SocketDispatcher.read0(SocketDispatcher.java:-2)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:340)
    at sun.nio.ch.IOUtil.read(IOUtil.java:294)
    at sun.nio.ch.IOUtil.read(IOUtil.java:269)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:425)
    at org.elasticsearch.transport.netty4.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:131)
    at org.elasticsearch.transport.netty4.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:116)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:1583)

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[slobodanadamovic]

Seems to fail consistently on Windows hosts. Will try to look closer.

[ywangd]

This fails quite often on Windows

• https://gradle-enterprise.elastic.co/s/insskiuls6rfi
• https://gradle-enterprise.elastic.co/s/vens6cugrf75a
• https://gradle-enterprise.elastic.co/s/rokh7j5qgicbm

[luigidellaquila]

It failed again https://gradle-enterprise.elastic.co/s/2ufy4uu7btph6/tests/task/:modules:transport-netty4:pooledTest/details/org.elasticsearch.http.netty4.Netty4HttpServerTransportTests/testChannelAcceptorCannotTamperThreadContext?top-execution=1

[albertzaharovits]

@Tim-Brooks do you have any tips on this one please?

[Tim-Brooks]

I imagine it is because the way a rejected channel is closed on windows breaks the assertion for caught exceptions in the client.

                @Override
                public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
                    if (cause instanceof PrematureChannelClosureException || cause instanceof SocketException) {
                        // no more requests coming, so fast-forward the latch
                        fastForward();
                    } else {
                        ExceptionsHelper.maybeDieOnAnotherThread(new AssertionError(cause));
                    }
                }

Windows appears to be returning this as java.io.IOException. Which seems fine. Just an issue with the test expectations.

[slobodanadamovic]

I've re-opened https://github.com/elastic/elasticsearch/pull/108362, which handles this java.io.IOException in Netty4HttpClient.

[valeriy42]

Another failure : https://gradle-enterprise.elastic.co/s/gdq6e3zg6ggea