[CI] MlWithSecurityIT test {yaml=ml/3rd_party_deployment/Test getting and putting Fill Mask with two mask tokens, as well as exceptions caused by requests with the wrong token} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #4786 / sles-12_platform-support-unix
• elasticsearch-periodic-platform-support #4731 / ubuntu-2004-aarch64_checkpart4_platform-support-arm

Reproduction Line:

./gradlew ":x-pack:plugin:ml:qa:ml-with-security:yamlRestTest" --tests "org.elasticsearch.smoketest.MlWithSecurityIT.test {yaml=ml/3rd_party_deployment/Test getting and putting Fill Mask with two mask tokens, as well as exceptions caused by requests with the wrong token}" -Dtests.seed=4E18B0BD396755 -Dtests.locale=ar-SA -Dtests.timezone=America/Halifax -Druntime.java=23

Applicable branches: main

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:task.keyword,order:0,selectedOptions:!(),title:'GradleTask',type:optionsListControl),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:className.keyword,order:1,selectedOptions:!(org.elasticsearch.smoketest.MlWithSecurityIT),title:'Suite',type:optionsListControl),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:name.keyword,order:2,selectedOptions:!(test%20%7Byaml%3Dml%2F3rd_party_deployment%2FTest%20getting%20and%20putting%20Fill%20Mask%20with%20two%20mask%20tokens%2C%20as%20well%20as%20exceptions%20caused%20by%20requests%20with%20the%20wrong%20token%7D),title:'Test',type:optionsListControl)))))

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:43981], URI [/_ml/trained_models/_stats?size=10000], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[],"type":"exception","reason":"Searching for stats for models [roberta_fill_mask_model,lang_ident_model_1,bert_fill_mask_model,with_correct_mask_token,test_model,another_test_model,with_other_correct_mask_token] failed","caused_by":{"type":"search_phase_execution_exception","reason":"","phase":"query","grouped":true,"failed_shards":[],"caused_by":{"type":"search_phase_execution_exception","reason":"Search rejected due to missing shards [[.ml-stats-000001][0]]. Consider using `allow_partial_search_results` setting to bypass this error.","phase":"query","grouped":true,"failed_shards":[]}}},"status":500}

Issue Reasons:

• [main] 2 failures in test test {yaml=ml/3rd_party_deployment/Test getting and putting Fill Mask with two mask tokens, as well as exceptions caused by requests with the wrong token} (0.4% fail rate in 521 executions)
• [main] 2 failures in pipeline elasticsearch-periodic-platform-support (16.7% fail rate in 12 executions)

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

Fixed by #116476