elasticsearchmachine
commented
1 week ago Pinging @elastic/es-security (Team:Security)
Open elasticsearchmachine opened 1 week ago
elasticsearchmachine
commented
1 week ago Pinging @elastic/es-security (Team:Security)
elasticsearchmachine
commented
1 week ago This has been muted on branch main
Mute Reasons:
Build Scans:
elasticsearchmachine
commented
1 week ago This has been muted on branch 8.x
Mute Reasons:
Build Scans:
slobodanadamovic
commented
1 week ago Some observations (so far): All tests are executed successfully. The LicensingTests is failing during AfterClass execution when internal cluster and all nodes are being stopped. It's failing due to the assertion that's been introduced in https://github.com/elastic/elasticsearch/pull/114786. I think it makes sense to expect that thread context is starting "clean" (default) before message is processed.
For some reason we have a warning header in the tread context. I'm suspecting that there is a place where default thread context is "polluted" and never really cleared. This might happen even before the message is processed, because the warning is added in the checkExpiry method. I'm still going through all places where we check features and license expiry.
ThreadContext at the time of assertion in InboundHandler:
ThreadContextStruct{requestHeaders={}, transientHeaders={}, responseHeaders={Warning=[299 Elasticsearch-9.0.0-64e8659845a7d873c9580a997e422e46718e1346 "warning: license will expire soon"]}, isSystemContext=false, warningHeadersSize=0}The assertion error:
oct. 15, 2024 5:42:38 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
WARNING: Uncaught exception in thread: Thread[#385,elasticsearch-error-rethrower,5,TGRP-LicensingTests]
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([E42DD4D4FEABAB68]:0)
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:112)
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:98)
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:822)
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:125)
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:97)
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:62)
at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:58)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:1570)This has been muted on branch 8.16
Mute Reasons:
Build Scans:
Build Scans:
Reproduction Line:
Applicable branches: 8.16
Reproduces locally?: N/A
Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.license.LicensingTests'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.license.LicensingTests'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))
Failure Message:
Issue Reasons:
Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.