[CI] Watcher BootStrapTests testMixedTriggeredWatchLoading failing

[astefan]

Build scan: https://gradle-enterprise.elastic.co/s/uechvlxybdeok

Repro line: ./gradlew ':x-pack:plugin:watcher:internalClusterTest' --tests "org.elasticsearch.xpack.watcher.test.integration.BootStrapTests.testMixedTriggeredWatchLoading" -Dtests.seed=6438A1D1DB20D548 -Dtests.locale=ca-ES -Dtests.timezone=Etc/GMT-11 -Druntime.java=17

Reproduces locally?: no

Applicable branches: master

Failure history: a similar failure in the same class was reported some time ago - https://github.com/elastic/elasticsearch/issues/67729 - looks like the fix would be applicable to this method as well, but I am not sure. Thus, creating a new issue for this failure.

Failure excerpt:

org.elasticsearch.xpack.watcher.test.integration.BootStrapTests > testMixedTriggeredWatchLoading FAILED
    java.lang.AssertionError: Count is 31 hits but 32 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([6438A1D1DB20D548:80FF2992E204B6FE]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount(ElasticsearchAssertions.java:282)
        at org.elasticsearch.xpack.watcher.test.integration.BootStrapTests.lambda$assertSingleExecutionAndCompleteWatchHistory$3(BootStrapTests.java:327)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1069)
        at org.elasticsearch.xpack.watcher.test.integration.BootStrapTests.assertSingleExecutionAndCompleteWatchHistory(BootStrapTests.java:313)
        at org.elasticsearch.xpack.watcher.test.integration.BootStrapTests.testMixedTriggeredWatchLoading(BootStrapTests.java:258)

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[pugnascotia]

More failures today, e.g. https://gradle-enterprise.elastic.co/s/pcqdyfriuc7h2

[pugnascotia]

Muted in master and 8.0.

[astefan]

This one still fails. In the past three weeks I'm seeing four failures. The most recent one is from today on 8.1 https://gradle-enterprise.elastic.co/s/naimp7eje4kye

org.elasticsearch.xpack.watcher.test.integration.BootStrapTests > testTriggeredWatchLoading FAILED
    java.lang.AssertionError: Count is 5 hits but 6 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([9CC0F602A70756F:30226FDA159A4D90]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount(ElasticsearchAssertions.java:282)
        at org.elasticsearch.xpack.watcher.test.integration.BootStrapTests.lambda$assertSingleExecutionAndCompleteWatchHistory$3(BootStrapTests.java:328)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1068)
        at org.elasticsearch.xpack.watcher.test.integration.BootStrapTests.assertSingleExecutionAndCompleteWatchHistory(BootStrapTests.java:314)
        at org.elasticsearch.xpack.watcher.test.integration.BootStrapTests.testTriggeredWatchLoading(BootStrapTests.java:309)

[gmarouli]

One more failure today: https://gradle-enterprise.elastic.co/s/co7i6rjspgefs/tests/:x-pack:plugin:watcher:internalClusterTest/org.elasticsearch.xpack.watcher.test.integration.BootStrapTests/testTriggeredWatchLoading?top-execution=1

[masseyke]

It would be interesting to re-enable this test and see if the failure happens again now that the history-writing code has been completely rewritten. I'll open a PR for that.

[masseyke]

Unmuted in #98407

[masseyke]

No failures in the two weeks since I unmuted it. I'm going to close this one.

[gmarouli]

@masseyke awesome work! Thank you!