elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.92k stars 24.73k forks source link

[CI] XDocsClientYamlTestSuiteIT test {yaml=en/rest-api/security/oidc-prepare-authentication-api/line_72} failing #85918

Closed mark-vieira closed 2 years ago

mark-vieira commented 2 years ago

This reproduces on Java 19. It seems the query params in the redirect url are ordered differently. This might be some implementation detail in the update JDK. Not sure if we can make this test a bit more robust.

Build scan: https://gradle-enterprise.elastic.co/s/j7rlne5j7yuri/tests/:x-pack:docs:integTest/org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT/test%20%7Byaml=en%2Frest-api%2Fsecurity%2Foidc-prepare-authentication-api%2Fline_72%7D

Reproduction line: ./gradlew ':x-pack:docs:integTest' --tests "org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT.test {yaml=en/rest-api/security/oidc-prepare-authentication-api/line_72}" -Dtests.seed=B47F42789F9BE697 -Dtests.locale=es-EC -Dtests.timezone=America/Lower_Princes -Druntime.java=19

Applicable branches: master

Reproduces locally?: Yes

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT&tests.test=test%20%7Byaml%3Den/rest-api/security/oidc-prepare-authentication-api/line_72%7D

Failure excerpt:

java.lang.AssertionError: Failure at [en/rest-api/security/oidc-prepare-authentication-api:19]: Expected a map containing
redirect: expected "http://127.0.0.1:8080/c2id-login?scope=openid&response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp" but was "http://127.0.0.1:8080/c2id-login?response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp&scope=openid"
   realm: "oidc1"
   state: "XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY"
   nonce: "sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4"

  at __randomizedtesting.SeedInfo.seed([B47F42789F9BE697:3C2B7DA231678B6F]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:493)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:828)

  Caused by: java.lang.AssertionError: Expected a map containing
  redirect: expected "http://127.0.0.1:8080/c2id-login?scope=openid&response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp" but was "http://127.0.0.1:8080/c2id-login?response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp&scope=openid"
     realm: "oidc1"
     state: "XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY"
     nonce: "sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4"

    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:82)
    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:65)
    at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:94)
    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:482)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke(Method.java:578)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
    at java.lang.Thread.run(Thread.java:828)
elasticmachine commented 2 years ago

Pinging @elastic/es-security (Team:Security)

grcevski commented 2 years ago

Happened again here:

https://gradle-enterprise.elastic.co/s/smiwt6ofog6bk/console-log?task=:x-pack:docs:integTest

mark-vieira commented 2 years ago

@Luegg We've added Java 19 testing to the 7.17 branch as well. Do you mind backporting https://github.com/elastic/elasticsearch/pull/86035 to 7.17 to address these failures?

dimitris-athanasiou commented 2 years ago

And another one here: https://gradle-enterprise.elastic.co/s/b32fonqbgov6w

fcofdez commented 2 years ago

There are a couple of related failures:

justincr-elastic commented 2 years ago

What version of Java 19 is being used?

This is from the Gradle scan for the CI test failure.

=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 4.15.0-1098-gcp (amd64)
  Runtime JDK Version   : 19-ea (Oracle)
  Runtime java.home     : /var/lib/jenkins/.java/openjdk19
  Gradle JDK Version    : 17.0.2 (Oracle)
  Gradle java.home      : /var/lib/jenkins/.java/openjdk-17.0.2+8-linux
  Random Testing Seed   : B47F42789F9BE697
  In FIPS 140 mode      : false
=======================================

I ran the reproduce command from that CI failure, but I used OpenJDK EA 19 b32 and the test passed.

./gradlew ':x-pack:docs:yamlRestTest' --tests "org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT.test {yaml=en/rest-api/security/oidc-prepare-authentication-api/line_72}" -Dtests.seed=B47F42789F9BE697 -Dtests.locale=es-EC -Dtests.timezone=America/Lower_Princes -Druntime.java=19

This is from the Gradle scan for my local build which passed.

======================================= |  
| Elasticsearch Build Hamster says Hello! |  
| Gradle Version        : 7.5 |  
| OS Info               : Linux 5.15.0-41-generic (amd64) |  
| Runtime JDK Version   : 19-ea (Oracle, 19-ea+32-2220) |  
| Runtime java.home     : /home/q/.jdks/openjdk-19b32 |  
| Gradle JDK Version    : 18.0.2 (Oracle) |  
| Gradle java.home      : /home/q/.jdks/openjdk-18.0.2 |  
| Random Testing Seed   : B47F42789F9BE697 |  
| In FIPS 140 mode      : false |  
| =======================================
BUILD SUCCESSFUL in 51s
grcevski commented 2 years ago

Based on the raw logs from the run, it appears to be Oracle Corporation 19-ea [OpenJDK 64-Bit Server VM 19-ea+32-2220]. Looks the same... maybe intermittent issue.

justincr-elastic commented 2 years ago

I don't see b32 mentioned in any of the raw logs for the CI failure links provided here:

I only see b32 mentioned in the raw log for my local build:

I am wondering if CI tests are using jdk-19-b0, hence no build number? If so, I wonder if updating to jdk-19-b32 could resolve these issues. OpenJDK 19 EA release download page only seems to provide the latest b32 build, so I would need to try and find an earlier copy of the b0 build.

mark-vieira commented 2 years ago

@justincr-elastic as mentioned in this comment this issue was addressed in main but not backported to 7.17. The failures happening now are only on that branch. For example: https://gradle-enterprise.elastic.co/s/2vmaxj63zozeu

justincr-elastic commented 2 years ago

88868 backported a fix from 8.x to 7.x to fix this CI failure.