SearchableSnapshotsIntegTests.testMountedSnapshotHasNoReplicasByDefault failures on CI

[gwbrown]

Build scan: https://gradle-enterprise.elastic.co/s/budltddhs37os

Repro line: gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testMountedSnapshotHasNoReplicasByDefault" -Dtests.seed=EF7E78C79AB95B2C -Dtests.security.manager=true -Dtests.locale=en-IE -Dtests.timezone=America/Argentina/Cordoba -Druntime.java=11

Reproduces locally?: No

Applicable branches: master

Failure history: build stats&_a=(columns:!(branch,message),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'test:testMountedSnapshotHasNoReplicasByDefault'),sort:!(time,desc))) Failed 3 times in the past week.

Failure excerpt:

Failure stack trace:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=303, name=elasticsearch[node_s0][generic][T#5], state=RUNNABLE, group=TGRP-SearchableSnapshotsIntegTests]    
    at __randomizedtesting.SeedInfo.seed([EF7E78C79AB95B2C:8F1FF900D57AFBC]:0)  
Caused by: java.lang.AssertionError: shard eviction should be successful: [snapshotUUID=xqd-Lpa-Sz-VgqFdnvwe9Q, snapshotIndexName=gzzohvcxmw, shardId=[abvmnzglrp][7]]  
    at __randomizedtesting.SeedInfo.seed([EF7E78C79AB95B2C]:0)  
    at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService.runIfShardMarkedAsEvictedInCache(CacheService.java:412)   
    at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService$1.doRun(CacheService.java:351)    
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)  
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)  
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) 
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:680) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  
    at java.lang.Thread.run(Thread.java:834)

Also of interest in the log:

[2020-12-18T17:30:58,302][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s0] [abvmnzglrp/NhRyIzgXRPeH-OeBmDjm_g] deleting index   
Dec 18, 2020 5:30:58 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException    
WARNING: Uncaught exception in thread: Thread[elasticsearch[node_s0][generic][T#5],5,TGRP-SearchableSnapshotsIntegTests]    
java.lang.AssertionError: shard eviction should be successful: [snapshotUUID=xqd-Lpa-Sz-VgqFdnvwe9Q, snapshotIndexName=gzzohvcxmw, shardId=[abvmnzglrp][7]] 
    at __randomizedtesting.SeedInfo.seed([EF7E78C79AB95B2C]:0)  
    at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService.runIfShardMarkedAsEvictedInCache(CacheService.java:412)   
    at org.elasticsearch.xpack.searchablesnapshots.cache.CacheService$1.doRun(CacheService.java:351)    
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)  
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)    
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)   
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:680) 
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)    
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)    
    at java.base/java.lang.Thread.run(Thread.java:834)  

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[dimitris-athanasiou]

Another failure of this exact test in https://gradle-enterprise.elastic.co/s/v5z2xk3o5wdxo

[javanna]

Another one: https://gradle-enterprise.elastic.co/s/giq2coadc6q2k

[tlrx]

The shard eviction mechanism has been improved in #67160 and the failing assertion removed in #67265. This should now be fixed, hence I'm closing this issue.