search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.42k stars 24.57k forks source link

[CI] MlDistributedFailureIT testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign failing #111135

Closed elasticsearchmachine closed 3 weeks ago

elasticsearchmachine commented 1 month ago

Build Scans:

Reproduction Line:

./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT.testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign" -Dtests.seed=6CC8DA6E3107EDC4 -Dtests.locale=en-ZA -Dtests.timezone=Europe/Vienna -Druntime.java=22

Applicable branches: 8.14

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

java.lang.AssertionError: 
Expected: an empty collection
     but: <[LEAK: resource was not cleaned up before it was garbage-collected.
Recent access records: 
Created at:
    in [elasticsearch[node_t2][ml_datafeed][T#1]][testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign]
    org.elasticsearch.action.search.ArraySearchPhaseResults.<init>(ArraySearchPhaseResults.java:27)
    org.elasticsearch.action.search.QueryPhaseResultConsumer.<init>(QueryPhaseResultConsumer.java:85)
    org.elasticsearch.action.search.SearchPhaseController.newSearchPhaseResults(SearchPhaseController.java:822)
    org.elasticsearch.action.search.TransportSearchAction$AsyncSearchActionProvider.newSearchPhase(TransportSearchAction.java:1333)
    org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1162)
    org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:934)
    org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:353)
    org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245)
    org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)
    org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)
    org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:465)
    org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:309)
    org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:113)
    org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96)
    org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)
    org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93)
    org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68)
    org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196)
    org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105)
    org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:83)
    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:357)
    org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:55)
    org.elasticsearch.client.internal.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:64)
    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:357)
    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:343)
    org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:36)
    org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:43)
    org.elasticsearch.xpack.core.ClientHelper.executeWithHeaders(ClientHelper.java:266)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.executeSearchRequest(ScrollDataExtractor.java:139)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.initScroll(ScrollDataExtractor.java:126)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.tryNextStream(ScrollDataExtractor.java:120)
    org.elasticsearch.xpack.ml.datafeed.extractor.scroll.ScrollDataExtractor.next(ScrollDataExtractor.java:103)
    org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor.getNextStream(ChunkedDataExtractor.java:133)
    org.elasticsearch.xpack.ml.datafeed.extractor.chunked.ChunkedDataExtractor.next(ChunkedDataExtractor.java:84)
    org.elasticsearch.xpack.ml.datafeed.DatafeedJob.run(DatafeedJob.java:376)
    org.elasticsearch.xpack.ml.datafeed.DatafeedJob.runLookBack(DatafeedJob.java:179)
    org.elasticsearch.xpack.ml.datafeed.DatafeedRunner$Holder.executeLookBack(DatafeedRunner.java:548)
    org.elasticsearch.xpack.ml.datafeed.DatafeedRunner$2.doRun(DatafeedRunner.java:247)
    org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    java.base/java.lang.Thread.run(Thread.java:1570)]>

Issue Reasons:

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 month ago

Pinging @elastic/ml-core (Team:ML)

maxhniebergall commented 3 weeks ago

I wasn't able to reproduce this test, the test isn't muted, and there have only been 4 failures in the last 8 weeks. This is probably related to some search changes, but doesn't seem to have having an impact currently. I'm going to close this issue.