search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.59k stars 24.63k forks source link

[CI] XPackRestIT test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline} failing #106652

Open ldematte opened 6 months ago

ldematte commented 6 months ago

Many other tests in the same build scan (https://gradle-enterprise.elastic.co/s/uprnbtvfhrrvk) fail with the same message: "Unable to reset machine learning feature as there are ingest pipelines still referencing trained machine learning models"

Possibly related to: https://github.com/elastic/elasticsearch/issues/71072

Build scan: https://gradle-enterprise.elastic.co/s/uprnbtvfhrrvk/tests/:x-pack:plugin:yamlRestTest/org.elasticsearch.xpack.test.rest.XPackRestIT/test%20%7Bp0=ml%2Finference_crud%2FTest%20force%20delete%20given%20model%20with%20alias%20referenced%20by%20pipeline%7D

Reproduction line:

./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline}" -Dtests.seed=5314F766DCACC2B2 -Dtests.locale=uk -Dtests.timezone=Asia/Thimphu -Druntime.java=22

Applicable branches: 8.13

Reproduces locally?: Didn't try

Failure history: Failure dashboard for org.elasticsearch.xpack.test.rest.XPackRestIT#test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline}&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('test%20%7Bp0%3Dml/inference_crud/Test%20force%20delete%20given%20model%20with%20alias%20referenced%20by%20pipeline%7D'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.test.rest.XPackRestIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure excerpt:

java.lang.AssertionError: Failure at [ml/inference_crud:680]: expected [2xx] status code but api [ml.delete_trained_model] returned [503 Service Unavailable] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}],"type":"search_phase_execution_exception","reason":"Partial shards failure","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"yCCTleljQ7eVm9bxerfJ4Q","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}}],"stack_trace":"Failed to execute phase [query], Partial shards failure; shardFailures {[yCCTleljQ7eVm9bxerfJ4Q][.ml-stats-000001][0]: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n}\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:418)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.successfulShardExecution(AbstractSearchAsyncAction.java:616)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResultConsumed(AbstractSearchAsyncAction.java:600)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$onShardResult$12(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer$PendingMerges.consume(QueryPhaseResultConsumer.java:383)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer.consumeResult(QueryPhaseResultConsumer.java:116)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:120)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:324)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:33)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:18)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:26)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:48)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:628)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleResponse(TransportService.java:1742)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1465)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1562)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1536)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:35)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:32)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:289)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.NoShardAvailableActionException.forOnShardFailureWrapper(NoShardAvailableActionException.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:532)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:479)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1609)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1584)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:146)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\t... 5 more\n"},"status":503}]

  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:561)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:504)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1570)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [ml.delete_trained_model] returned [503 Service Unavailable] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}],"type":"search_phase_execution_exception","reason":"Partial shards failure","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"yCCTleljQ7eVm9bxerfJ4Q","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}}],"stack_trace":"Failed to execute phase [query], Partial shards failure; shardFailures {[yCCTleljQ7eVm9bxerfJ4Q][.ml-stats-000001][0]: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n}\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:418)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.successfulShardExecution(AbstractSearchAsyncAction.java:616)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResultConsumed(AbstractSearchAsyncAction.java:600)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$onShardResult$12(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer$PendingMerges.consume(QueryPhaseResultConsumer.java:383)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer.consumeResult(QueryPhaseResultConsumer.java:116)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:120)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:324)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:33)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:18)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:26)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:48)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:628)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleResponse(TransportService.java:1742)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1465)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1562)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1536)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:35)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:32)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:289)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.NoShardAvailableActionException.forOnShardFailureWrapper(NoShardAvailableActionException.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:532)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:479)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1609)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1584)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:146)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\t... 5 more\n"},"status":503}]

    at org.junit.Assert.fail(Assert.java:89)
    at org.elasticsearch.test.rest.yaml.section.DoSection.checkResponseException(DoSection.java:529)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:385)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:541)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:504)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke(Method.java:580)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1570)
elasticsearchmachine commented 6 months ago

Pinging @elastic/ml-core (Team:ML)

ldematte commented 6 months ago

Another set of failures: https://gradle-enterprise.elastic.co/s/6ksrvsmntrdr6

alex-spies commented 5 months ago

And another one: https://gradle-enterprise.elastic.co/s/td2aksmy5mjya

maxhniebergall commented 4 months ago

It seems like this issue is still blocked by the underlying issue: https://github.com/elastic/elasticsearch/issues/65846

ywangd commented 4 months ago

When the test fail, it seems to impact the rest of the tests running in the same suite

nielsbauman commented 3 months ago

Another failing build: https://gradle-enterprise.elastic.co/s/w4esogkgaw6xk