search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.93k stars 24.74k forks source link

[CI] CrossClusterAccessHeadersForCcsRestIT testCrossClusterAccessHeadersSentMultipleRemotes failing #105038

Closed slobodanadamovic closed 8 months ago

slobodanadamovic commented 8 months ago

Build scan: https://gradle-enterprise.elastic.co/s/djvwugkus5glo/tests/:x-pack:plugin:security:qa:security-trial:javaRestTest/org.elasticsearch.xpack.security.crossclusteraccess.CrossClusterAccessHeadersForCcsRestIT/testCrossClusterAccessHeadersSentMultipleRemotes

Reproduction line:

./gradlew ':x-pack:plugin:security:qa:security-trial:javaRestTest' --tests "org.elasticsearch.xpack.security.crossclusteraccess.CrossClusterAccessHeadersForCcsRestIT.testCrossClusterAccessHeadersSentMultipleRemotes" -Dtests.seed=C17B5BDB897AA712 -Dtests.locale=es-CR -Dtests.timezone=VST -Druntime.java=21 -Dtests.fips.enabled=true

Applicable branches: main

Reproduces locally?: Yes

Failure history: Failure dashboard for org.elasticsearch.xpack.security.crossclusteraccess.CrossClusterAccessHeadersForCcsRestIT#testCrossClusterAccessHeadersSentMultipleRemotes&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testCrossClusterAccessHeadersSentMultipleRemotes'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.security.crossclusteraccess.CrossClusterAccessHeadersForCcsRestIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure excerpt:

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
#1:
    org.elasticsearch.transport.OutboundHandler.lambda$sendResponse$1(OutboundHandler.java:144)
    org.elasticsearch.core.IOUtils.close(IOUtils.java:71)
    org.elasticsearch.core.IOUtils.close(IOUtils.java:87)
    org.elasticsearch.core.IOUtils.close(IOUtils.java:63)
    org.elasticsearch.core.Releasables.close(Releasables.java:72)
    org.elasticsearch.core.Releasables.close(Releasables.java:40)
    org.elasticsearch.core.Releasables$3.close(Releasables.java:128)
    org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:131)
    org.elasticsearch.transport.OutboundHandler$1.onResponse(OutboundHandler.java:216)
    org.elasticsearch.transport.OutboundHandler$1.onResponse(OutboundHandler.java:212)
    org.elasticsearch.action.ActionListenerImplementations$NotifyOnceActionListener.onResponse(ActionListenerImplementations.java:339)
    org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addPromise$1(Netty4TcpChannel.java:79)
    io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
    io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
    io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
    io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
    io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:728)
    io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:283)
    io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:363)
    org.elasticsearch.transport.netty4.CopyBytesSocketChannel.doWrite(CopyBytesSocketChannel.java:102)
    io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:931)
    io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
    io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895)
    io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
    io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:921)
    io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:907)
    io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:893)
    org.elasticsearch.transport.netty4.Netty4WriteThrottlingHandler.doFlush(Netty4WriteThrottlingHandler.java:146)
    org.elasticsearch.transport.netty4.Netty4WriteThrottlingHandler.flush(Netty4WriteThrottlingHandler.java:101)
    io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:923)
    io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:941)
    io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:966)
    io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:934)
    io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
    io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
    org.elasticsearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:168)
    org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:212)
    org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:197)
    org.elasticsearch.transport.OutboundHandler.sendResponse(OutboundHandler.java:140)
    org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:61)
    org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:37)
    org.elasticsearch.xpack.security.crossclusteraccess.CrossClusterAccessHeadersForCcsRestIT.lambda$startTransport$4(CrossClusterAccessHeadersForCcsRestIT.java:1151)
    org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:288)
    org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:273)
    org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115)
    org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:96)
    org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:826)
    org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:124)
    org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:96)
    org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:61)
    org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:48)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    java.base/java.lang.Thread.run(Thread.java:1583)
#2:
    org.elasticsearch.core.RefCounted.mustIncRef(RefCounted.java:71)
    org.elasticsearch.transport.OutboundHandler.sendResponse(OutboundHandler.java:139)
    org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:61)
    org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:37)
    org.elasticsearch.xpack.security.crossclusteraccess.CrossClusterAccessHeadersForCcsRestIT.lambda$startTransport$4(CrossClusterAccessHeadersForCcsRestIT.java:1151)
    org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:288)
    org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:273)
    org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115)
    org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:96)
    org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:826)
    org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:124)
    org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:96)
    org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:61)
    org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:48)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    java.base/java.lang.Thread.run(Thread.java:1583)
Created at:
    org.elasticsearch.action.search.SearchResponse.<init>(SearchResponse.java:86)
    org.elasticsearch.action.search.SearchResponse.<init>(SearchResponse.java:136)
    org.elasticsearch.xpack.security.crossclusteraccess.CrossClusterAccessHeadersForCcsRestIT.lambda$startTransport$4(CrossClusterAccessHeadersForCcsRestIT.java:1153)
    org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:288)
    org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:273)
    org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115)
    org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:96)
    org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:826)
    org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:124)
    org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:96)
    org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:61)
    org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:48)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
    io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    java.base/java.lang.Thread.run(Thread.java:1583)]>

  at __randomizedtesting.SeedInfo.seed([C17B5BDB897AA712:32534FC65087A6CA]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2119)
  at org.elasticsearch.test.ESTestCase.checkStaticState(ESTestCase.java:729)
  at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:520)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)
elasticsearchmachine commented 8 months ago

Pinging @elastic/es-security (Team:Security)