The testDeleteEndpointWhileReferencedByPipeline test fails and then any tests following it in this suite seems to fail. The node crashes with an assertion error indicating that we are tying up a transport thread (not good).
Caused by: java.lang.AssertionError: Expected current thread [Thread[#111,elasticsearch[test-cluster-0][transport_worker][T#8],5,main]] to not be a transport thread. Reason: [non-trivial nested loops over cluster state structures]
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.transport.Transports.assertNotTransportThread(Transports.java:66)
at org.elasticsearch.xcore@8.15.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.utils.InferenceProcessorInfoExtractor.pipelineIdsForResource(InferenceProcessorInfoExtractor.java:139)
at org.elasticsearch.inference@8.15.0-SNAPSHOT/org.elasticsearch.xpack.inference.action.TransportDeleteInferenceEndpointAction.lambda$masterOperation$1(TransportDeleteInferenceEndpointAction.java:93)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:382)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:302)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:205)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.lambda$andThen$0(SubscribableListener.java:469)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:433)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:128)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:469)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:433)
at org.elasticsearch.inference@8.15.0-SNAPSHOT/org.elasticsearch.xpack.inference.action.TransportDeleteInferenceEndpointAction.masterOperation(TransportDeleteInferenceEndpointAction.java:80)
at org.elasticsearch.inference@8.15.0-SNAPSHOT/org.elasticsearch.xpack.inference.action.TransportDeleteInferenceEndpointAction.masterOperation(TransportDeleteInferenceEndpointAction.java:39)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:126)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:237)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:244)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:237)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:174)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:56)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:70)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:48)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:182)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:462)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1033)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:999)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:476)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:190)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:466)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:442)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:329)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:248)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:155)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:192)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:210)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.xcore@8.15.0-SNAPSHOT/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:290)
at org.elasticsearch.xcore@8.15.0-SNAPSHOT/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.xcore@8.15.0-SNAPSHOT/org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:208)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:185)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:151)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:345)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$5(SecurityActionFilter.java:178)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:174)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:131)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:138)
at org.elasticsearch.inference@8.15.0-SNAPSHOT/org.elasticsearch.xpack.inference.rest.RestDeleteInferenceEndpointAction.lambda$prepareRequest$0(RestDeleteInferenceEndpointAction.java:63)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:119)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:478)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.rest.RestController$1.onResponse(RestController.java:472)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:248)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:472)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:632)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:487)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:583)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:460)
at org.elasticsearch.transport.netty4@8.15.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:127)
at org.elasticsearch.transport.netty4@8.15.0-SNAPSHOT/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:117)
at io.netty.transport@4.1.109.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
... 25 more
[2024-06-05T05:08:32,909][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [test-cluster-0] fatal error in thread [elasticsearch-error-rethrower], exiting java.lang.AssertionError: Expected current thread [Thread[#111,elasticsearch[test-cluster-0][transport_worker][T#8],5,main]] to not be a transport thread. Reason: [non-trivial nested loops over cluster state structures]
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.transport.Transports.assertNotTransportThread(Transports.java:66)
at org.elasticsearch.xcore@8.15.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.utils.InferenceProcessorInfoExtractor.pipelineIdsForResource(InferenceProcessorInfoExtractor.java:139)
at org.elasticsearch.inference@8.15.0-SNAPSHOT/org.elasticsearch.xpack.inference.action.TransportDeleteInferenceEndpointAction.lambda$masterOperation$1(TransportDeleteInferenceEndpointAction.java:93)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:382)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:302)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:205)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.lambda$andThen$0(SubscribableListener.java:469)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:433)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:128)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:469)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:433)
at org.elasticsearch.inference@8.15.0-SNAPSHOT/org.elasticsearch.xpack.inference.action.TransportDeleteInferenceEndpointAction.masterOperation(TransportDeleteInferenceEndpointAction.java:80)
at org.elasticsearch.inference@8.15.0-SNAPSHOT/org.elasticsearch.xpack.inference.action.TransportDeleteInferenceEndpointAction.masterOperation(TransportDeleteInferenceEndpointAction.java:39)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:126)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:237)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:244)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:237)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:174)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:56)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.MappedActionFilters$MappedFilterChain.proceed(MappedActionFilters.java:70)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.MappedActionFilters.apply(MappedActionFilters.java:48)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:182)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:462)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1033)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:999)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
at org.elasticsearch.security@8.15.0-SNAPSHOT/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:476)
at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
Failure history:Failure dashboard for org.elasticsearch.xpack.inference.InferenceCrudIT#testDeleteEndpointWhileReferencedByPipeline&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testDeleteEndpointWhileReferencedByPipeline'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.inference.InferenceCrudIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))
Failure excerpt:
org.apache.http.ConnectionClosedException: Connection is closed
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:924)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:304)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
at org.elasticsearch.xpack.inference.InferenceBaseRestTest.deleteModel(InferenceBaseRestTest.java:118)
at org.elasticsearch.xpack.inference.InferenceCrudIT.testDeleteEndpointWhileReferencedByPipeline(InferenceCrudIT.java:137)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
at java.lang.Thread.run(Thread.java:1570)
Caused by: org.apache.http.ConnectionClosedException: Connection is closed
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:1570)
The
testDeleteEndpointWhileReferencedByPipeline
test fails and then any tests following it in this suite seems to fail. The node crashes with an assertion error indicating that we are tying up a transport thread (not good).Build scan: https://gradle-enterprise.elastic.co/s/elweikioferq6/tests/:x-pack:plugin:inference:qa:inference-service-tests:javaRestTest/org.elasticsearch.xpack.inference.InferenceCrudIT/testDeleteEndpointWhileReferencedByPipeline
Reproduction line:
Applicable branches: main
Reproduces locally?: Didn't try
Failure history: Failure dashboard for
org.elasticsearch.xpack.inference.InferenceCrudIT#testDeleteEndpointWhileReferencedByPipeline
&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testDeleteEndpointWhileReferencedByPipeline'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.inference.InferenceCrudIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))Failure excerpt: