search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.59k stars 24.63k forks source link

[CI] FeatureStateResetApiIT testResetSystemIndices and feature-migration/line_47 failing #97780

Closed kingherc closed 1 year ago

kingherc commented 1 year ago

So far I notice this test failure goes hand in hand with the following test failure:

REPRODUCE WITH: ./gradlew ':docs:yamlRestTest' --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/migration/apis/feature-migration/line_47}" -Dtests.seed=ED495A645B8EA9A0 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=fi -Dtests.timezone=Europe/Istanbul -Druntime.java=20    
org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT > test {yaml=reference/migration/apis/feature-migration/line_47} FAILED   
    java.lang.AssertionError: Failure at [reference/migration/apis/feature-migration:15]: Expected a map containing 
             features: a list containing    
                      0: a map containing   
                  indices: an empty list    
             feature_name: "async_search"   
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      1: a map containing   
                  indices: an empty list    
             feature_name: "enrich" 
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      2: a map containing   
                  indices: an empty list    
             feature_name: "ent_search" 
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      3: a map containing   
                  indices: an empty list    
             feature_name: "fleet"  
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      4: a map containing   
                  indices: an empty list    
             feature_name: "geoip"  
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      5: a map containing   
                  indices: an empty list    
             feature_name: "kibana" 
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      6: a map containing   
                  indices: an empty list    
             feature_name: "logstash_management"    
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      7: a map containing   
                  indices: an empty list    
             feature_name: "machine_learning"   
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      8: a map containing   
                  indices: an empty list    
             feature_name: "searchable_snapshots"   
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                      9: a map containing   
                  indices: an empty list    
             feature_name: "security"   
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                     10: a map containing   
                  indices: an empty list    
             feature_name: expected "synonyms" but was "tasks"  
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                     11: a map containing   
                  indices: an empty list    
             feature_name: expected "tasks" but was "transform" 
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                     12: a map containing   
                  indices: an empty list    
             feature_name: expected "transform" but was "watcher"   
         migration_status: "NO_MIGRATION_NEEDED"    
    minimum_index_version: "8.9.0"  
                     13: expected a map but was <missing>   
     migration_status: "NO_MIGRATION_NEEDED"    
        at __randomizedtesting.SeedInfo.seed([ED495A645B8EA9A0:651D65BEF572C458]:0) 
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:572)    
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:524)  
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)    
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)   
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)   
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)    
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)    
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)   
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)   
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) 
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)   
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) 
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) 
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)  
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)  
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) 
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)    
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)    
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)    
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)   
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) 
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)   
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) 
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) 
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)   
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)   
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) 
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) 
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) 
        at java.base/java.lang.Thread.run(Thread.java:1623)

Build scan: https://gradle-enterprise.elastic.co/s/qs5kpprbtirdq/tests/:server:internalClusterTest/org.elasticsearch.snapshots.FeatureStateResetApiIT/testResetSystemIndices

Reproduction line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.snapshots.FeatureStateResetApiIT.testResetSystemIndices" -Dtests.seed=90E6E28A2194E5A1 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=bg-BG -Dtests.timezone=Africa/Windhoek -Druntime.java=20

Applicable branches: main, 8.9

Reproduces locally?: Didn't try

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.snapshots.FeatureStateResetApiIT&tests.test=testResetSystemIndices

Failure excerpt:

java.lang.AssertionError: 
Expected: iterable with items [<ResetFeatureStateStatus{featureName='SystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='SecondSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='EvilSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='tasks', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='synonyms', status=SUCCESS, exception='null'}>] in any order
     but: no item matches: <ResetFeatureStateStatus{featureName='synonyms', status=SUCCESS, exception='null'}> in [<ResetFeatureStateStatus{featureName='EvilSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='SecondSystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='SystemIndexTestPlugin', status=SUCCESS, exception='null'}>, <ResetFeatureStateStatus{featureName='tasks', status=SUCCESS, exception='null'}>]

  at __randomizedtesting.SeedInfo.seed([90E6E28A2194E5A1:3E3A135F1F5D364A]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.snapshots.FeatureStateResetApiIT.testResetSystemIndices(FeatureStateResetApiIT.java:74)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)
elasticsearchmachine commented 1 year ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

kingherc commented 1 year ago

Appears also several other times on main and 8.9 so will try to mute.

rjernst commented 1 year ago

@williamrandolph Can you take a look here?

williamrandolph commented 1 year ago

The problem in both cases looks to be that the synonyms feature is missing. There's a feature flag for the synonyms API; if the feature is disabled the system index descriptor doesn't get loaded at runtime. What we're seeing is consistent with the synonyms API being disable for these tests. I've been poking around to try to figure out how these flags get set and unset, but I'm not having any luck today.

rjernst commented 1 year ago

It seems odd the tasks and synonyms system indices are checked at all in this test. The test is checking feature reset with artificial system indices. It looks like synonyms was moved into server with the synonyms api work in #95895.

IMO those feature descriptors should be overridden completely by the test. I think this can be done by moving the definition of the server features to Node so that they can be merged with those from plugins before calling the SystemIndices ctor, and having it overrideable by MockNode. Then whether a particular real feature is available would be irrelevant to the test.

williamrandolph commented 1 year ago

I see now that -Dbuild.snapshot=false is a condition for this test failure, but to reproduce it locally we have to add -Dlicense.key=x-pack/plugin/core/src/test/resources/public.key on the command line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.snapshots.FeatureStateResetApiIT.testResetSystemIndices" -Dtests.seed=90E6E28A2194E5A1 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=bg-BG -Dtests.timezone=Africa/Windhoek -Druntime.java=20 -Dlicense.key=x-pack/plugin/core/src/test/resources/public.key
benwtrent commented 1 year ago

@carlosdelest take a gander dude, something funky with release builds. Maybe a feature flag thing???

carlosdelest commented 1 year ago

Thanks for the ping! Solving in https://github.com/elastic/elasticsearch/pull/98129

carlosdelest commented 1 year ago

Fixed by https://github.com/elastic/elasticsearch/pull/98129