CI: WatcherRestartIT.testWatcherRestart

[costin]

The rolling-upgrades to 7.10 are failing due to a watcher test. Might be related to https://github.com/elastic/elasticsearch/pull/63437

Build scan: https://gradle-enterprise.elastic.co/s/zzfql5gpdpor6 https://gradle-enterprise.elastic.co/s/vkz3pnpybwg5o https://gradle-enterprise.elastic.co/s/euctqqukee7s2

Repro line:

Execution failed for task ':x-pack:qa:rolling-upgrade:v7.10.0#oneThirdUpgradedTest'.

REPRODUCE WITH: ./gradlew ':x-pack:qa:rolling-upgrade:v7.10.0#oneThirdUpgradedTest' --tests "org.elasticsearch.upgrades.WatcherRestartIT.testWatcherRestart" \
  -Dtests.seed=613F9E2C824179F \
  -Dtests.security.manager=true \
  -Dtests.locale=el-GR \
  -Dtests.timezone=Europe/Vaduz \
  -Dtests.distribution=default \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':x-pack:qa:rolling-upgrade:v7.10.0#oneThirdUpgradedTest' --tests "org.elasticsearch.upgrades.WatcherRestartIT.testWatcherRestart" \
  -Dtests.seed=613F9E2C824179F \
  -Dtests.security.manager=true \
  -Dtests.locale=el-GR \
  -Dtests.timezone=Europe/Vaduz \
  -Dtests.distribution=default \
  -Druntime.java=8

Reproduces locally?:

Wasn't able to due to some snafu with the rolling clusters.

Applicable branches: 7.10

Failure history: https://build-stats.elastic.co/goto/db39a4ded176a7bc638fc2ce0887ae1a

--> Failure excerpt:

12:27:52 org.elasticsearch.upgrades.WatcherRestartIT > testWatcherRestart FAILED
12:27:52     java.lang.AssertionError: 
12:27:52     Expected: is <200>
12:27:52          but: was <404>
12:27:52         at __randomizedtesting.SeedInfo.seed([613F9E2C824179F:934BF76D8338CEC6]:0)
12:27:52         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
12:27:52         at org.junit.Assert.assertThat(Assert.java:956)
12:27:52         at org.junit.Assert.assertThat(Assert.java:923)
12:27:52         at org.elasticsearch.upgrades.WatcherRestartIT.validateHistoryTemplate(WatcherRestartIT.java:51)
12:27:52         at org.elasticsearch.upgrades.WatcherRestartIT.testWatcherRestart(WatcherRestartIT.java:34)
12:27:52 REPRODUCE WITH: ./gradlew ':x-pack:qa:rolling-upgrade:v7.10.0#oneThirdUpgradedTest' --tests "org.elasticsearch.upgrades.WatcherRestartIT.testWatcherRestart" -Dtests.seed=613F9E2C824179F -Dtests.security.manager=true -Dtests.locale=el-GR -Dtests.timezone=Europe/Vaduz -Dtests.distribution=default -Druntime.java=8
12:27:52 
12:27:52 Suite: Test class org.elasticsearch.upgrades.WatcherRestartIT
12:27:52   1> [2020-11-20T11:27:50,806][INFO ][o.e.u.WatcherRestartIT   ] [testWatcherRestart] before test
12:27:52   1> [2020-11-20T11:27:51,101][INFO ][o.e.u.WatcherRestartIT   ] [testWatcherRestart] initializing REST clients against [http://[::1]:42242, http://127.0.0.1:39923, http://[::1]:41790, http://127.0.0.1:45618, http://[::1]:44734, http://127.0.0.1:43384]
12:27:52   2> Νοε 20, 2020 11:27:51 ΠΜ org.elasticsearch.client.RestClient logResponse
12:27:52   2> WARNING: request [HEAD http://[::1]:41790/_template/.watch-history-11?include_type_name=false] returned 1 warnings: [299 Elasticsearch-7.10.0-51e9d6f22758d0374a0f3f5c6e8f3a7997850f96 "[types removal] Specifying include_type_name in get index template requests is deprecated."]
12:27:52   1> [2020-11-20T11:27:51,966][INFO ][o.e.u.WatcherRestartIT   ] [testWatcherRestart] There are still tasks running after this test that might break subsequent tests [data_frame/transforms[c], xpack/ml/job[c], xpack/rollup/job[c]].
12:27:52   1> [2020-11-20T11:27:52,022][INFO ][o.e.u.WatcherRestartIT   ] [testWatcherRestart] after test
12:27:52   2> REPRODUCE WITH: ./gradlew ':x-pack:qa:rolling-upgrade:v7.10.0#oneThirdUpgradedTest' --tests "org.elasticsearch.upgrades.WatcherRestartIT.testWatcherRestart" -Dtests.seed=613F9E2C824179F -Dtests.security.manager=true -Dtests.locale=el-GR -Dtests.timezone=Europe/Vaduz -Dtests.distribution=default -Druntime.java=8
12:27:52   2> java.lang.AssertionError: 
12:27:52     Expected: is <200>
12:27:52          but: was <404>
12:27:52         at __randomizedtesting.SeedInfo.seed([613F9E2C824179F:934BF76D8338CEC6]:0)
12:27:52         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
12:27:52         at org.junit.Assert.assertThat(Assert.java:956)
12:27:52         at org.junit.Assert.assertThat(Assert.java:923)
12:27:52         at org.elasticsearch.upgrades.WatcherRestartIT.validateHistoryTemplate(WatcherRestartIT.java:51)
12:27:52         at org.elasticsearch.upgrades.WatcherRestartIT.testWatcherRestart(WatcherRestartIT.java:34)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[hendrikmuhs]

muted: https://github.com/elastic/elasticsearch/commit/1828a8b6cc98506329da68fba7997660a10ce3cc

[martijnvg]

This was fixed via #69918