[CI] TrainedModelAssignmentNodeServiceTests testLoadQueuedModelsWhenOneFails failing

elasticsearchmachine commented 3 days ago

Build Scans:

Reproduction Line:

./gradlew ':x-pack:plugin:ml:test' --tests "org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentNodeServiceTests.testLoadQueuedModelsWhenOneFails" -Dtests.seed=713CDEE7BEA268EF -Dtests.locale=es-AR -Dtests.timezone=America/Cordoba -Druntime.java=22

Applicable branches: main

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testLoadQueuedModelsWhenOneFails'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentNodeServiceTests'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

org.mockito.exceptions.verification.WantedButNotInvoked: 
Wanted but not invoked:
deploymentManager.stopDeployment(
    <Capturing argument: TrainedModelDeploymentTask>
);
-> at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentNodeServiceTests.testLoadQueuedModelsWhenOneFails(TrainedModelAssignmentNodeServiceTests.java:338)

However, there were exactly 2 interactions with this mock:
deploymentManager.startDeployment(
    Task{id=1, type='trained_model_assignment', action='xpack/ml/trained_model_assignment[n]', description='model_id[loading-foo]', parentTask=unset, startTime=1720189983528, startTimeNanos=254127493635},
    org.elasticsearch.action.support.SubscribableListener@5b91852b
);
-> at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentNodeService.lambda$loadModel$4(TrainedModelAssignmentNodeService.java:235)

deploymentManager.startDeployment(
    Task{id=2, type='trained_model_assignment', action='xpack/ml/trained_model_assignment[n]', description='model_id[failed-loading-foo]', parentTask=unset, startTime=1720189983528, startTimeNanos=254127551141},
    org.elasticsearch.action.support.SubscribableListener@1e86a220
);
-> at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentNodeService.lambda$loadModel$4(TrainedModelAssignmentNodeService.java:235)

Issue Reasons:

[main] 2 consecutive failures in test testLoadQueuedModelsWhenOneFails (0.3% fail rate)
[main] 2 consecutive failures in step part4 (2.2% fail rate)
[main] 2 consecutive failures in pipeline elasticsearch-intake (2.2% fail rate)
[main] 3 failures in test testLoadQueuedModelsWhenOneFails (0.3% fail rate)
[main] 3 failures in step part4 (2.2% fail rate)
[main] 3 failures in pipeline elasticsearch-intake (2.2% fail rate)
[main] 2.2% failure rate in step part4 (2.2% fail rate)
[main] 2.2% failure rate in pipeline elasticsearch-intake (2.2% fail rate)

Note: This issue was created using new, in-development muting automation. Please mind any issues.

elasticsearchmachine commented 3 days ago

This has been muted on branch main

Mute Reasons:

[main] 2 consecutive failures in test testLoadQueuedModelsWhenOneFails (0.3% fail rate)
[main] 2 consecutive failures in step part4 (2.2% fail rate)
[main] 2 consecutive failures in pipeline elasticsearch-intake (2.2% fail rate)
[main] 3 failures in test testLoadQueuedModelsWhenOneFails (0.3% fail rate)
[main] 3 failures in step part4 (2.2% fail rate)
[main] 3 failures in pipeline elasticsearch-intake (2.2% fail rate)
[main] 2.2% failure rate in step part4 (2.2% fail rate)
[main] 2.2% failure rate in pipeline elasticsearch-intake (2.2% fail rate)

Build Scans:

elasticsearchmachine commented 3 days ago

Pinging @elastic/ml-core (Team:ML)

elastic / elasticsearch

[CI] TrainedModelAssignmentNodeServiceTests testLoadQueuedModelsWhenOneFails failing #110536