[CI] KibanaThreadPoolIT testBlockedThreadPoolsRejectUserRequests failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #11132 / part1
• elasticsearch-periodic #4379 / openjdk21_checkpart1_java-matrix
• elasticsearch-intake #11077 / part1
• elasticsearch-intake #11039 / part1
• elasticsearch-pull-request #35835 / part-1
• elasticsearch-periodic #4371 / openjdk22_checkpart1_java-matrix
• elasticsearch-periodic-platform-support #4341 / ubuntu-1804_platform-support-unix
• elasticsearch-intake #10971 / part1
• elasticsearch-pull-request #35666 / part-1
• elasticsearch-pull-request #35543 / part-1

Reproduction Line:

./gradlew ":modules:kibana:internalClusterTest" --tests "org.elasticsearch.kibana.KibanaThreadPoolIT.testBlockedThreadPoolsRejectUserRequests" -Dtests.seed=7BFB6DB150801CFE -Dtests.locale=da-GL -Dtests.timezone=America/Monterrey -Druntime.java=22

Applicable branches: 8.x

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testBlockedThreadPoolsRejectUserRequests'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.kibana.KibanaThreadPoolIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

junit.framework.AssertionFailedError: Expected exception EsRejectedExecutionException but no exception was thrown

Issue Reasons:

• [8.x] 18 failures in test testBlockedThreadPoolsRejectUserRequests (3.1% fail rate in 576 executions)
• [8.x] 5 failures in step part1 (6.7% fail rate in 75 executions)
• [8.x] 6 failures in step part-1 (4.8% fail rate in 124 executions)
• [8.x] 2 failures in step openjdk21_checkpart1_java-matrix (14.3% fail rate in 14 executions)
• [8.x] 5 failures in pipeline elasticsearch-intake (6.7% fail rate in 75 executions)
• [8.x] 6 failures in pipeline elasticsearch-pull-request (5.3% fail rate in 114 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (21.4% fail rate in 14 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic-platform-support (21.4% fail rate in 14 executions)

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testBlockedThreadPoolsRejectUserRequests (0.4% fail rate in 779 executions)
• [main] 2 failures in step part-1 (0.8% fail rate in 240 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.8% fail rate in 242 executions)

Build Scans:

• elasticsearch-periodic-platform-support #4244 / ubuntu-1804_platform-support-unix
• elasticsearch-pull-request #33724 / part-1
• elasticsearch-pull-request #33248 / part-1

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 22 failures in test testBlockedThreadPoolsRejectUserRequests (2.8% fail rate in 778 executions)
• [8.x] 3 failures in step part1 (4.5% fail rate in 67 executions)
• [8.x] 7 failures in step part-1 (5.9% fail rate in 119 executions)
• [8.x] 2 failures in step windows-2019_checkpart1_platform-support-windows (15.4% fail rate in 13 executions)
• [8.x] 2 failures in step ubuntu-1804_platform-support-unix (10.5% fail rate in 19 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (14.3% fail rate in 21 executions)
• [8.x] 8 failures in pipeline elasticsearch-periodic-platform-support (40.0% fail rate in 20 executions)
• [8.x] 3 failures in pipeline elasticsearch-intake (4.5% fail rate in 67 executions)
• [8.x] 7 failures in pipeline elasticsearch-pull-request (6.4% fail rate in 110 executions)

Build Scans:

• elasticsearch-periodic #4371 / openjdk22_checkpart1_java-matrix
• elasticsearch-periodic-platform-support #4341 / ubuntu-1804_platform-support-unix
• elasticsearch-intake #10971 / part1
• elasticsearch-pull-request #35666 / part-1
• elasticsearch-pull-request #35543 / part-1
• elasticsearch-pull-request #35527 / part-1
• elasticsearch-periodic-platform-support #4333 / windows-2019_checkpart1_platform-support-windows
• elasticsearch-pull-request #35410 / part-1
• elasticsearch-periodic #4351 / concurrent-search-tests
• elasticsearch-periodic #4351 / openjdk21_checkpart1_java-matrix

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 18 failures in test testBlockedThreadPoolsRejectUserRequests (3.1% fail rate in 576 executions)
• [8.x] 5 failures in step part1 (6.7% fail rate in 75 executions)
• [8.x] 6 failures in step part-1 (4.8% fail rate in 124 executions)
• [8.x] 2 failures in step openjdk21_checkpart1_java-matrix (14.3% fail rate in 14 executions)
• [8.x] 5 failures in pipeline elasticsearch-intake (6.7% fail rate in 75 executions)
• [8.x] 6 failures in pipeline elasticsearch-pull-request (5.3% fail rate in 114 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (21.4% fail rate in 14 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic-platform-support (21.4% fail rate in 14 executions)

Build Scans:

• elasticsearch-intake #11132 / part1
• elasticsearch-periodic #4379 / openjdk21_checkpart1_java-matrix
• elasticsearch-intake #11077 / part1
• elasticsearch-intake #11039 / part1
• elasticsearch-pull-request #35835 / part-1
• elasticsearch-periodic #4371 / openjdk22_checkpart1_java-matrix
• elasticsearch-periodic-platform-support #4341 / ubuntu-1804_platform-support-unix
• elasticsearch-intake #10971 / part1
• elasticsearch-pull-request #35666 / part-1
• elasticsearch-pull-request #35543 / part-1

[prdoyle]

This is a problem in the test, so low-risk. Requests are not 100% always rejected even when the thread-pools are busy; there's more to figure out here, but it won't cause a problem for customers.