[CI] MonitoringWithWatcherRestIT.testThatLocalExporterAddsWatches failures

[cbuescher]

Build scan:

https://gradle-enterprise.elastic.co/s/7ozkt4tgixntq

Repro line:

./gradlew ':x-pack:plugin:watcher:qa:with-monitoring:integTestRunner' --tests "org.elasticsearch.smoketest.MonitoringWithWatcherRestIT.testThatLocalExporterAddsWatches" \
  -Dtests.seed=498D6AE9762D2D55 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-US \
  -Dtests.timezone=Pacific/Efate \
  -Druntime.java=11

Reproduces locally?:

Wasn't able to reproduce locally within a few runs

Error

java.lang.AssertionError: 
Expected: is <6>
     but: was <1>
    at __randomizedtesting.SeedInfo.seed([498D6AE9762D2D55:AE7A5AAF2E976FD1]:0)
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
    at org.junit.Assert.assertThat(Assert.java:956)
    at org.junit.Assert.assertThat(Assert.java:923)
    at org.elasticsearch.smoketest.MonitoringWithWatcherRestIT.lambda$assertTotalWatchCount$1(MonitoringWithWatcherRestIT.java:92)
    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:929)
    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:902)

Applicable branches:

Failure in 7.8, but found similar failures in the past on master, e.g. https://gradle-enterprise.elastic.co/s/wfknexh53ho5m

Failure history:

Looks like this failed occasionally (~ 6 times in June) some earlier in May... Latest one on master

https://gradle-enterprise.elastic.co/s/wfknexh53ho5m

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Watcher)

[dakrone]

Another failure: https://gradle-enterprise.elastic.co/s/dq3oxpmejx45a

[davidkyle]

A new failure on master: https://gradle-enterprise.elastic.co/s/fseuecrqhx37m

java.lang.AssertionError: 
Expected: is <6>
     but: was <1>
    at __randomizedtesting.SeedInfo.seed([5A9BC3BCD5B301F6:BD6CF3FA8D094372]:0)
    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
    at org.junit.Assert.assertThat(Assert.java:956)
    at org.junit.Assert.assertThat(Assert.java:923)
    at org.elasticsearch.smoketest.MonitoringWithWatcherRestIT.lambda$assertTotalWatchCount$2(MonitoringWithWatcherRestIT.java:109)

The test is failing once or twice a week on 7.10, 7.x and master

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(build.branch),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:testThatLocalExporterAddsWatches),sort:!(process.time-start,desc))

[davidkyle]

Muted master: 22b554e0f4d17b3f98fcaf806738218fcc7e4a97 7.x: ff5eb271d1924defdb2236bf6e609e35af2aa42d 7.10 : 51f8b0b8d2f7d4758f0f38534cf4846602021e7c

[jakelandis]

@jbaiera - Can this test be removed as superseded by some of the testing around the setting to enable this functionality ?

[jbaiera]

Can this test be removed as superseded by some of the testing around the setting to enable this functionality ?

Yes, Monitoring already tests this feature, this can be removed.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)