search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.2k stars 24.84k forks source link

testWatcherRestart occasionally fails in REST tests #33753

Closed DaveCTurner closed 5 years ago

DaveCTurner commented 6 years ago

I count 10 failures of WatchBackwardsCompatibilityIT#testWatcherRestart since June 13th, all in the 6.x series (i.e. 6.2, 6.3 or 6.4 as well as 6.x). They are not all the same as the failure described here, but some of them are. The latest one was in a BWC test but non-BWC tests also failed in this fashion.

The build link is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+bwc-tests/1/console

The reproduction line follows, but does not reproduce for me:

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0#oldClusterTestRunner \
  -Dtests.seed=A6F6BC1F82877D71 \
  -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT \
  -Dtests.method="testWatcherRestart" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-VE \
  -Dtests.timezone=Etc/GMT+4

The output follows.

java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
    at __randomizedtesting.SeedInfo.seed([A6F6BC1F82877D71:33AEB290C99BA428]:0)
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
    at org.junit.Assert.assertThat(Assert.java:956)
    at org.junit.Assert.assertThat(Assert.java:923)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:830)
    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:804)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$12(WatchBackwardsCompatibilityIT.java:297)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:273)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.ensureWatcherStopped(WatchBackwardsCompatibilityIT.java:297)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.testWatcherRestart(WatchBackwardsCompatibilityIT.java:184)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"starting\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"starting\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:301)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"starting\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"starting\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:301)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"-34h6dtFQ2SsSYIJ_n0LKw\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"lKzxpI34R-qXm3EvTdeyAQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"Wj_9z6zFQ8yo8NSFko-K-A\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
        ... 42 more
elasticmachine commented 6 years ago

Pinging @elastic/es-core-infra

dliappis commented 6 years ago

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/111/consoleText

danielmitterdorfer commented 6 years ago

Looking at the failure output in the original comment, we see that on one node (node id -34h6dtFQ2SsSYIJ_n0LKw), Watcher is in the state starting and then moves to started over time. So maybe the problem is that before we attempt to stop Watcher in:

https://github.com/elastic/elasticsearch/blob/230cd10772f6fec217a6e9ae95ea6a76b2edd75b/x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/WatchBackwardsCompatibilityIT.java#L183

we'd actually need to ensure that all watches are started?

@spinscale / @hub-cap can one of you please have a look at this one?

andrershov commented 6 years ago

Occurred again https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=debian/58/console

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:without-system-key:v6.4.3-SNAPSHOT#oldClusterTestRunner \
  -Dtests.seed=EA5213BE918AF24A \
  -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT \
  -Dtests.method="testWatcherRestart" \
  -Dtests.security.manager=true \
  -Dtests.locale=el \
  -Dtests.timezone=Africa/Lagos \
  -Dcompiler.java=11 \
  -Druntime.java=8
jaymode commented 6 years ago

Another instance: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java8,nodes=virtual&&linux/56/console

albertzaharovits commented 5 years ago

And another one: https://groups.google.com/a/elastic.co/d/msg/build-elasticsearch/LXzJGUAL0WQ/wD4F1v4TBAAJ

java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
    at __randomizedtesting.SeedInfo.seed([7AADDEF1EDDF9F81:EFF5D07EA6C346D8]:0)
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
    at org.junit.Assert.assertThat(Assert.java:956)
    at org.junit.Assert.assertThat(Assert.java:923)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$12(WatchBackwardsCompatibilityIT.java:297)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:273)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.ensureWatcherStopped(WatchBackwardsCompatibilityIT.java:297)
    at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.testWatcherRestart(WatchBackwardsCompatibilityIT.java:184)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
        ... 42 more
    Suppressed: java.lang.AssertionError: 
Expected: not a string containing "\"watcher_state\":\"started\""
     but: was "{\"_nodes\":{\"total\":3,\"successful\":3,\"failed\":0},\"cluster_name\":\"rolling-upgrade\",\"manually_stopped\":true,\"stats\":[{\"node_id\":\"Wkj71lhKTKK6_WCWEm8iZA\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"1zPF1T65Tliq3BaItKtk4w\",\"watcher_state\":\"started\",\"watch_count\":1,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}},{\"node_id\":\"d8qRlV2HSC6uetFHniBKZQ\",\"watcher_state\":\"stopped\",\"watch_count\":0,\"execution_thread_pool\":{\"queue_size\":0,\"max_size\":0}}]}"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$ensureWatcherStopped$11(WatchBackwardsCompatibilityIT.java:302)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
cbuescher commented 5 years ago

And another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/369/console

cbuescher commented 5 years ago

And another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/357/console

cbuescher commented 5 years ago

I start to think we should mute this, especially since there seems to be a PR on the way to fix: https://github.com/elastic/elasticsearch/pull/35243

cbuescher commented 5 years ago

And another one. This should definitely be muted IMHO: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+bwc-tests/96/console

cbuescher commented 5 years ago

I muted this on 6.x with c847953f8aa166c7e1bd0b1580ff56bbbc1c5ff4 and on 6.5 with 15632d7d264dab52b590440443855f6678223977.

I haven't seen any failures on master since this seems to be a bwc thing, also there are three other similarly named WatchBackwardsCompatibilityIT in other qa modules which I haven't muted so far since the failures I've seen so far only were in rolling-upgrade:without-system-key

Tim-Brooks commented 5 years ago

There was a failure on 6.x for this test today:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/926/console

@pgomulka - Is this still an issue?

cbuescher commented 5 years ago

@tbrooks8 @pgomulka 7f20c0b apparently didn't solve the issue, I'll revert that commit on 6.x

cbuescher commented 5 years ago

Reverted in 633aac157d990944f9fe9c676dcd7fa1d454f084

jakelandis commented 5 years ago

I am closing this issue since this specific test upgrading TO 6.x, and as such was removed from 7.x (a long time ago). The test is currently muted in 6.x (non-existent in 7+) and I don't believe it worth effort to track down a possible root cause of the flakey-ness of this test since 7.0 has been released and we haven't had real world issues w/r/t Watcher and upgrading TO 6.x.