[CI] TextEmbeddingCrudIT testPutE5WithTrainedModelAndInference failing

elasticsearchmachine commented 1 month ago

Build Scans:

Reproduction Line:

./gradlew ':x-pack:plugin:inference:qa:inference-service-tests:javaRestTest' --tests "org.elasticsearch.xpack.inference.TextEmbeddingCrudIT.testPutE5WithTrainedModelAndInference" -Dtests.seed=A2FD5FFFC3590F21 -Dtests.locale=ca-FR -Dtests.timezone=America/Catamarca -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches: 8.15

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:task.keyword,order:0,selectedOptions:!(),title:'GradleTask',type:optionsListControl),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:className.keyword,order:1,selectedOptions:!(org.elasticsearch.xpack.inference.TextEmbeddingCrudIT),title:'Suite',type:optionsListControl),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:name.keyword,order:2,selectedOptions:!(testPutE5WithTrainedModelAndInference),title:'Test',type:optionsListControl)))))

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:38627], URI [_ml/trained_models/.multilingual-e5-small/deployment/_start?wait_for=fully_allocated], status line [HTTP/1.1 408 Request Timeout]
{"error":{"root_cause":[{"type":"status_exception","reason":"Timed out after [30s] waiting for model deployment to start. Use the trained model stats API to track the state of the deployment."}],"type":"status_exception","reason":"Timed out after [30s] waiting for model deployment to start. Use the trained model stats API to track the state of the deployment."},"status":408}

Issue Reasons:

[8.15] 6 failures in test testPutE5WithTrainedModelAndInference (1.2% fail rate in 506 executions)
[8.15] 3 failures in step openjdk17_checkpart2_java-fips-matrix (18.8% fail rate in 16 executions)
[8.15] 4 failures in pipeline elasticsearch-periodic (23.5% fail rate in 17 executions)

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 month ago

This has been muted on branch main

Mute Reasons:

[main] 3 failures in test testPutE5WithTrainedModelAndInference (1.5% fail rate in 197 executions)
[main] 2 failures in step part-2 (2.4% fail rate in 82 executions)
[main] 2 failures in pipeline elasticsearch-pull-request (2.4% fail rate in 82 executions)

Build Scans:

elasticsearchmachine commented 1 month ago

Pinging @elastic/ml-core (Team:ML)

elasticsearchmachine commented 1 month ago

This has been muted on branch 8.x

Mute Reasons:

[8.x] 3 failures in test testPutE5WithTrainedModelAndInference (1.2% fail rate in 250 executions)
[8.x] 2 failures in step part-2 (2.6% fail rate in 77 executions)
[8.x] 2 failures in pipeline elasticsearch-pull-request (2.6% fail rate in 77 executions)

Build Scans:

elasticsearchmachine commented 1 week ago

This has been muted on branch 8.15

Mute Reasons:

[8.15] 6 failures in test testPutE5WithTrainedModelAndInference (1.2% fail rate in 506 executions)
[8.15] 3 failures in step openjdk17_checkpart2_java-fips-matrix (18.8% fail rate in 16 executions)
[8.15] 4 failures in pipeline elasticsearch-periodic (23.5% fail rate in 17 executions)

Build Scans:

elastic / elasticsearch

[CI] TextEmbeddingCrudIT testPutE5WithTrainedModelAndInference failing #114023