[CI] DataTierAllocationDeciderIT testShardsAreKeptInPreferredTierUntilTheNextTierIsInItsFinalState failing

[cbuescher]

Build scan: https://gradle-enterprise.elastic.co/s/f3fyvm7cm6agi/tests/:x-pack:plugin:core:internalClusterTest/org.elasticsearch.xpack.cluster.routing.allocation.DataTierAllocationDeciderIT/testShardsAreKeptInPreferredTierUntilTheNextTierIsInItsFinalState

Reproduction line:

./gradlew ':x-pack:plugin:core:internalClusterTest' --tests "org.elasticsearch.xpack.cluster.routing.allocation.DataTierAllocationDeciderIT.testShardsAreKeptInPreferredTierUntilTheNextTierIsInItsFinalState" -Dtests.seed=207BB246A1A95F51 -Dtests.locale=sr-BA -Dtests.timezone=Atlantic/Faroe -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches: 8.10

Reproduces locally?: No

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.cluster.routing.allocation.DataTierAllocationDeciderIT&tests.test=testShardsAreKeptInPreferredTierUntilTheNextTierIsInItsFinalState

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=275, name=elasticsearch[node_t7][clusterApplierService#updateTask][T#1], state=RUNNABLE, group=TGRP-DataTierAllocationDeciderIT]

  at __randomizedtesting.SeedInfo.seed([207BB246A1A95F51:2F7A7CD6D3A3AE90]:0)

  Caused by: java.lang.AssertionError: (No message provided)

    at __randomizedtesting.SeedInfo.seed([207BB246A1A95F51]:0)
    at org.elasticsearch.discovery.PeerFinder.closePeers(PeerFinder.java:182)
    at org.elasticsearch.cluster.coordination.Coordinator.onClusterStateApplied(Coordinator.java:415)
    at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleApplyCommit$8(Coordinator.java:403)
    at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.lambda$map$0(ActionListenerImplementations.java:108)
    at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:89)
    at org.elasticsearch.cluster.service.ClusterApplierService$ClusterApplyActionListener.onResponse(ClusterApplierService.java:598)
    at org.elasticsearch.cluster.service.ClusterApplierService$ClusterApplyActionListener.onResponse(ClusterApplierService.java:569)
    at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:439)
    at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:154)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:217)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:183)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(Thread.java:833)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[cbuescher]

Similar thing on main branch a few weeks ago (Aug 22th): https://gradle-enterprise.elastic.co/s/6szxs3dgjoua2 And also another one on 8.10 that same day: https://gradle-enterprise.elastic.co/s/6jxjxhvpnlmlw

[DaveCTurner]

Duplicates https://github.com/elastic/elasticsearch/issues/99023.