[CI] JsonLogsFormatAndParseIT testElementsPresentOnAllLinesOfLog failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #3669 / amazonlinux-2023_platform-support-aws
• elasticsearch-periodic-platform-support #3625 / amazonlinux-2023_platform-support-aws

Reproduction Line:

./gradlew ':qa:unconfigured-node-name:javaRestTest' --tests "org.elasticsearch.unconfigured_node_name.JsonLogsFormatAndParseIT.testElementsPresentOnAllLinesOfLog" -Dtests.seed=71088A5130D5AC7C -Dtests.locale=ar-YE -Dtests.timezone=Europe/London -Druntime.java=22

Applicable branches: 8.15

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testElementsPresentOnAllLinesOfLog'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.unconfigured_node_name.JsonLogsFormatAndParseIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://127.0.0.1:46365], URI [_cluster/health?wait_for_events=languid], status line [HTTP/1.1 408 Request Timeout]
{"cluster_name":"javaRestTest","status":"green","timed_out":true,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":83,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":32813,"active_shards_percent_as_number":100.0}

Issue Reasons:

• [8.15] 2 failures in test testElementsPresentOnAllLinesOfLog (0.5% fail rate in 366 executions)
• [8.15] 2 failures in step amazonlinux-2023_platform-support-aws (20.0% fail rate in 10 executions)
• [8.15] 2 failures in pipeline elasticsearch-periodic-platform-support (16.7% fail rate in 12 executions)

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

Interestingly this looks to be failing with a similar timeout as https://github.com/elastic/elasticsearch/issues/111632 and on the same platform (Amazon Linux 2023). Related?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

The failure here has nothing to do with the test. The log test passed, and this failure happened during cleanup, where the health API timed out. So I am reassigning to data management for investigation.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

It looks like there is something in a task queue that is just not getting cleared out. One of those reports task_max_waiting_in_queue_millis of 32.8s, and the other 40s. I haven't been able to reproduce this locally (yet), and don't see any logging that might help.