DatafeedJobsIT#testDatafeedTimingStats_DatafeedRecreated fails

[javanna]

Build scan: https://gradle-enterprise.elastic.co/s/weggtbiofm7zw/console-log?task=:x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner

Repro line: ./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.testDatafeedTimingStats_DatafeedRecreated" -Dtests.seed=862B982F15848C7 -Dtests.security.manager=true -Dtests.locale=ro-RO -Dtests.timezone=Africa/Mogadishu -Druntime.java=8 -Dtests.fips.enabled=true

Reproduces locally?: no

Applicable branches: any

Failure history: This test has been failing every once in a while in the last months, hard to say when it started. Not sure why an issue was not logged, there seems to be #44335 but it's a very old one.

Failure excerpt:

    | org.elasticsearch.xpack.ml.integration.DatafeedJobsIT > testDatafeedTimingStats_DatafeedRecreated FAILED |     | java.lang.AssertionError: |     | Expected: a value greater than <0L> |     | but: <0L> was equal to <0L> |     | at __randomizedtesting.SeedInfo.seed([862B982F15848C7:CF35BA87409F0E03]:0) |     | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |     | at org.junit.Assert.assertThat(Assert.java:956) |     | at org.junit.Assert.assertThat(Assert.java:923) |     | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.assertDatafeedStats(DatafeedJobsIT.java:272) |     | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.lambda$testDatafeedTimingStats_DatafeedRecreated$5(DatafeedJobsIT.java:178) |     | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:951) |     | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.lambda$testDatafeedTimingStats_DatafeedRecreated$6(DatafeedJobsIT.java:175) |     | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.testDatafeedTimingStats_DatafeedRecreated(DatafeedJobsIT.java:185)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

@przemekwitek please could you take a look at this one?

I see at the end of the test it calls openAndRunJob twice: https://github.com/elastic/elasticsearch/blob/0bc307f865d62c4be09d1c2fa5e17a0aacb1972a/x-pack/plugin/ml/qa/native-multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/ml/integration/DatafeedJobsIT.java#L181-L182

The comment // Datafeed processed numDocs documents so search_count must be greater than 0. in that lambda is only true for the first run of the datafeed. The datafeed gets deleted and recreated but the data counts are from the job which doesn't get deleted and recreated. So second time through it's expected that the datafeed finds no documents. I can see that it ought to do a search to determine that there are no extra documents, but maybe there's uncertainty about how quickly it will do this if it already appears to be up-to-date (if the results of the first invocation are within the last query_delay seconds)?

[przemekwitek]

I'm taking a look at this issue now...

[droberts195]

Assigning medium-risk due to loss of test coverage from muting.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

This issue has been closed because it has been open for too long with no activity.

Any muted tests that were associated with this issue have been unmuted.

If the tests begin failing again, a new issue will be opened, and they may be muted again.

[elasticsearchmachine]

This issue is getting re-opened because there are still AwaitsFix mutes for the given test. It will likely be closed again in the future.