search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.19k stars 24.85k forks source link

[CI] DocsClientYamlTestSuiteIT test {yaml=reference/eql/eql/line_338} failing #95975

Closed iverase closed 1 year ago

iverase commented 1 year ago

Build scan: https://gradle-enterprise.elastic.co/s/7juphmg57mk2y/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Feql%2Feql%2Fline_338%7D

Reproduction line:

./gradlew ':docs:yamlRestTest' --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/eql/eql/line_338}" -Dtests.seed=206E917A70AC9100 -Dtests.locale=ga-IE -Dtests.timezone=Asia/Damascus -Druntime.java=20

Applicable branches: 8.8

Reproduces locally?: No

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT&tests.test=test%20%7Byaml%3Dreference/eql/eql/line_338%7D

Failure excerpt:

java.lang.AssertionError: Failure at [reference/eql/eql:747]: Expected a map containing
      hits: a map containing
       total: a map containing
         value: <2>
      relation: "eq"
   sequences: a list containing
             0: a map containing
       join_keys: a list containing
                 0: "doom"
          events: a list containing
                 0: a map containing
              _index: "my-index-000001"
             _source: a map containing
            @timestamp: "1234567897"
     @timestamp_pretty: "17-12-2022"
            event_type: "failure"
                  bool: <true>
                    os: "redhat"
                  port: <1234>
       missing_keyword: "yyy"
                  host: "doom"
                    id: <17>
               version: "20.2.0"
                uptime: <15>
                 _id: "7"
                 1: a map containing
              _index: "my-index-000001"
             _source: a map containing
            @timestamp: expected "1234567891" but was "1234568896"
     @timestamp_pretty: expected "12-12-2022" but was "17-12-2022"
             type_test: expected "abc" but was <missing>
            event_type: "alert"
                    os: expected "win10" but was "redhat"
                  port: expected <1234> but was <65123>
       missing_keyword: expected "test" but was <missing>
                    ip: expected "10.0.0.1" but was <missing>
                  host: "doom"
                    id: expected <11> but was <16>
               version: expected "1.0.0" but was "20.10.0"
                uptime: expected <0> but was <missing>
                  bool: <unexpected> but was <true>
                 _id: expected "1" but was "6"
                 2: a map containing
              _index: expected "my-index-000001" but was "my-index-000002"
             _source: a map containing
            @timestamp: expected "1234568896" but was "1234568996"
     @timestamp_pretty: expected "17-12-2022" but was "2022-12-17"
            event_type: "alert"
                  bool: <true>
                    os: expected "redhat" but was <missing>
                  port: <65123>
                  host: "doom"
                    id: expected <16> but was <26>
               version: expected "20.10.0" but was <missing>
                    ip: <unexpected> but was "10.0.0.5"
                op_sys: <unexpected> but was "redhat"
                 _id: "6"
             1: a map containing
       join_keys: a list containing
                 0: "farcry"
          events: a list containing
                 0: a map containing
              _index: "my-index-000001"
             _source: a map containing
            @timestamp: "1234567893"
     @timestamp_pretty: "12-12-2022"
            event_type: "alert"
             type_test: "abc"
                  bool: <false>
                    os: "win10"
                  port: <1234>
                  host: "farcry"
                    id: <13>
               version: "2.0.0"
                uptime: <1>
                 _id: "3"
                 1: a map containing
              _index: "my-index-000001"
             _source: a map containing
            @timestamp: "1234567893"
            event_type: "alert"
                  bool: <true>
                    os: "win10"
                  port: <1234>
       missing_keyword: null
                    ip: "10.0.0.5"
                  host: "farcry"
                    id: <110>
               version: "1.2.3"
                uptime: <1>
                 _id: "10"
                 2: a map containing
              _index: "my-index-000003"
             _source: a map containing
            @timestamp: "1334567893"
            event_type: "alert"
                  bool: <true>
                    os: "win10"
                  host: "farcry"
                    id: <33>
                 _id: "3"
is_running: <false>
      took: <61>
is_partial: <false>
 timed_out: <false>

  at __randomizedtesting.SeedInfo.seed([206E917A70AC9100:A83AAEA0DE50FCF8]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:572)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:524)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

  Caused by: java.lang.AssertionError: Expected a map containing
        hits: a map containing
         total: a map containing
           value: <2>
        relation: "eq"
     sequences: a list containing
               0: a map containing
         join_keys: a list containing
                   0: "doom"
            events: a list containing
                   0: a map containing
                _index: "my-index-000001"
               _source: a map containing
              @timestamp: "1234567897"
       @timestamp_pretty: "17-12-2022"
              event_type: "failure"
                    bool: <true>
                      os: "redhat"
                    port: <1234>
         missing_keyword: "yyy"
                    host: "doom"
                      id: <17>
                 version: "20.2.0"
                  uptime: <15>
                   _id: "7"
                   1: a map containing
                _index: "my-index-000001"
               _source: a map containing
              @timestamp: expected "1234567891" but was "1234568896"
       @timestamp_pretty: expected "12-12-2022" but was "17-12-2022"
               type_test: expected "abc" but was <missing>
              event_type: "alert"
                      os: expected "win10" but was "redhat"
                    port: expected <1234> but was <65123>
         missing_keyword: expected "test" but was <missing>
                      ip: expected "10.0.0.1" but was <missing>
                    host: "doom"
                      id: expected <11> but was <16>
                 version: expected "1.0.0" but was "20.10.0"
                  uptime: expected <0> but was <missing>
                    bool: <unexpected> but was <true>
                   _id: expected "1" but was "6"
                   2: a map containing
                _index: expected "my-index-000001" but was "my-index-000002"
               _source: a map containing
              @timestamp: expected "1234568896" but was "1234568996"
       @timestamp_pretty: expected "17-12-2022" but was "2022-12-17"
              event_type: "alert"
                    bool: <true>
                      os: expected "redhat" but was <missing>
                    port: <65123>
                    host: "doom"
                      id: expected <16> but was <26>
                 version: expected "20.10.0" but was <missing>
                      ip: <unexpected> but was "10.0.0.5"
                  op_sys: <unexpected> but was "redhat"
                   _id: "6"
               1: a map containing
         join_keys: a list containing
                   0: "farcry"
            events: a list containing
                   0: a map containing
                _index: "my-index-000001"
               _source: a map containing
              @timestamp: "1234567893"
       @timestamp_pretty: "12-12-2022"
              event_type: "alert"
               type_test: "abc"
                    bool: <false>
                      os: "win10"
                    port: <1234>
                    host: "farcry"
                      id: <13>
                 version: "2.0.0"
                  uptime: <1>
                   _id: "3"
                   1: a map containing
                _index: "my-index-000001"
               _source: a map containing
              @timestamp: "1234567893"
              event_type: "alert"
                    bool: <true>
                      os: "win10"
                    port: <1234>
         missing_keyword: null
                      ip: "10.0.0.5"
                    host: "farcry"
                      id: <110>
                 version: "1.2.3"
                  uptime: <1>
                   _id: "10"
                   2: a map containing
                _index: "my-index-000003"
               _source: a map containing
              @timestamp: "1334567893"
              event_type: "alert"
                    bool: <true>
                      os: "win10"
                    host: "farcry"
                      id: <33>
                   _id: "3"
  is_running: <false>
        took: <61>
  is_partial: <false>
   timed_out: <false>

    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:82)
    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:65)
    at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:94)
    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:552)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:524)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke(Method.java:578)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1623)
elasticsearchmachine commented 1 year ago

Pinging @elastic/es-ql (Team:QL)

pxsalehi commented 1 year ago

Failed again and it seems to fail often: https://gradle-enterprise.elastic.co/s/hlp2qifkiikiy/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Feql%2Feql%2Fline_338%7D?top-execution=1

ywangd commented 1 year ago

Another one https://gradle-enterprise.elastic.co/s/il6dilkcyroty/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Feql%2Feql%2Fline_338%7D?top-execution=1

astefan commented 1 year ago

@abdonpijpelink can you, please, have a look at this test failing? It might be because of which document is retrieved first (based on the number of shards of the index maybe; not sure how many shards each index is created with).