Closed bpintea closed 1 year ago
Pinging @elastic/es-data-management (Team:Data Management)
I've tried several times to reproduce this with no luck.
Failed again: https://gradle-enterprise.elastic.co/s/fetuxpkqwegfo
trace:
org.elasticsearch.upgrades.WatcherRestartIT > testWatcherRestart FAILED
java.lang.AssertionError:
Expected: not a string containing "\"watcher_state\":\"stopped\""
but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"v7.9.3\",\"manually_stopped\":false,\"stats\":[{\"node_id\":\"oBkeLHbySVy7kQtaxg09kQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":1}},{\"node_id\":\"T8VQB60TQHOpLWR6mIhB0Q\",\"watcher_state\":\"started\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"NDzwUvKOTbCQtzwBGPx6jQ\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":1}}]}"
at __randomizedtesting.SeedInfo.seed([D1D81A0B0B2C0594:448014844030DCCD]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.upgrades.WatcherRestartIT.lambda$ensureWatcherStarted$3(WatcherRestartIT.java:180)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1123)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1096)
at org.elasticsearch.upgrades.WatcherRestartIT.ensureWatcherStarted(WatcherRestartIT.java:174)
at org.elasticsearch.upgrades.WatcherRestartIT.testWatcherRestart(WatcherRestartIT.java:42)
reproduce line:
./gradlew ':x-pack:qa:rolling-upgrade:v7.9.3#twoThirdsUpgradedTest' -Dtests.class="org.elasticsearch.upgrades.WatcherRestartIT" -Dtests.method="testWatcherRestart" -Dtests.seed=D1D81A0B0B2C0594 -Dtests.bwc=true -Dtests.locale=nl-BE -Dtests.timezone=Etc/GMT-8 -Druntime.java=8
Attached are the cluster logs from the test failure
Failed again in the same way that Ben described above in https://gradle-enterprise.elastic.co/s/ylll4e7wtprmm
Cluster logs attached as well. 161.zip
This test has been muted in 7.16 and 7.17 branches. It is failing in these branches very often and that is disruptive.
Pretty high confidence that the most recent failures are due the same root cause as https://github.com/elastic/elasticsearch/issues/81110#issuecomment-1002234837 as evident by "missing watcher index templates, not starting watcher service" and which step the failures happen. I would suggest to unmute this once that issue is resolved.
(also the OP error looks transient and not too concerning)
Since this is related to versions < 7.16 it doesn't seem relevant anymore. The test on main currently is not muted and there haven't been any failures reported recently.
Build scan: https://gradle-enterprise.elastic.co/s/umosjq57hlvzk/tests/:x-pack:qa:rolling-upgrade:v7.12.1%23oneThirdUpgradedTest/org.elasticsearch.upgrades.WatcherRestartIT/testWatcherRestart
Reproduction line:
./gradlew ':x-pack:qa:rolling-upgrade:v7.12.1#oneThirdUpgradedTest' -Dtests.class="org.elasticsearch.upgrades.WatcherRestartIT" -Dtests.method="testWatcherRestart" -Dtests.seed=1469FD150430CE11 -Dtests.bwc=true -Dtests.locale=no-NO -Dtests.timezone=PST8PDT -Druntime.java=8
Applicable branches: 7.16
Reproduces locally?: Didn't try
Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.upgrades.WatcherRestartIT&tests.test=testWatcherRestart
Failure excerpt: