search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.42k stars 24.57k forks source link

[CI] MlDistributedFailureIT testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown failing #111078

Closed elasticsearchmachine closed 3 weeks ago

elasticsearchmachine commented 1 month ago

Build Scans:

Reproduction Line:

./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT.testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown" -Dtests.seed=D66D75CDB2C4305 -Dtests.locale=es-PY -Dtests.timezone=Pacific/Rarotonga -Druntime.java=22

Applicable branches: 8.14

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
Created at:
    in [elasticsearch[node_t3][ml_datafeed][T#1]][testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown]
    org.elasticsearch.action.search.ArraySearchPhaseResults.<init>(ArraySearchPhaseResults.java:27)
    org.elasticsearch.action.search.QueryPhaseResultConsumer.<init>(QueryPhaseResultConsumer.java:85)
    org.elasticsearch.action.search.SearchPhaseController.newSearchPhaseResults(SearchPhaseController.java:822)
    org.elasticsearch.action.search.TransportSearchAction$AsyncSearchActionProvider.newSearchPhase(TransportSearchAction.java:1333)
    org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1162)
    org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:934)
    org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:353)
    org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
    org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)
    org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)
    org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:465)
    org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:309)
    org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:113)
    org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)
    org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
    org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
    org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)
    org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)
    org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)
    org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)
    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:357)
    org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:55)
    org.elasticsearch.client.internal.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:64)
    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:357)
    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:343)
    org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:36)
    org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:43)
    org.elasticsearch.xpack.core.ClientHelper.executeWithHeaders(ClientHelper.java:266)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.executeSearchRequest(ScrollDataExtractor.java:139)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.initScroll(ScrollDataExtractor.java:126)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.tryNextStream(ScrollDataExtractor.java:120)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.next(ScrollDataExtractor.java:103)
    org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor.getNextStream(ChunkedDataExtractor.java:133)
    org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor.next(ChunkedDataExtractor.java:84)
    org.elasticsearch.xpack.ml.datafeed.DatafeedJob.run(DatafeedJob.java:376)
    org.elasticsearch.xpack.ml.datafeed.DatafeedJob.runLookBack(DatafeedJob.java:179)
    org.elasticsearch.xpack.ml.datafeed.DatafeedRunner$Holder.executeLookBack(DatafeedRunner.java:548)
    org.elasticsearch.xpack.ml.datafeed.DatafeedRunner$2.doRun(DatafeedRunner.java:247)
    org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    java.base/java.lang.Thread.run(Thread.java:1570)]>

Issue Reasons:

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 month ago

Pinging @elastic/ml-core (Team:ML)

maxhniebergall commented 3 weeks ago

I wasn't able to reproduce this test, the test isn't muted, and there have only been 4 failures in the last 8 weeks. This is probably related to some search changes, but doesn't seem to have having an impact currently. I'm going to close this issue.

The failures seem to be the same as https://github.com/elastic/elasticsearch/issues/111135, they probably have the same underlying issue which seems to have been fixed.