[CI] SqlCompatIT testCreateCursorWithFormatTxtOnNewNode failing

[danielmitterdorfer]

Build scan: https://gradle-enterprise.elastic.co/s/eputgzf7edauq/tests/:x-pack:plugin:sql:qa:mixed-node:v8.11.2%23mixedClusterTest/org.elasticsearch.xpack.sql.qa.mixed_node.SqlCompatIT/testCreateCursorWithFormatTxtOnNewNode Reproduction line:

./gradlew ':x-pack:plugin:sql:qa:mixed-node:v8.11.2#mixedClusterTest' -Dtests.class="org.elasticsearch.xpack.sql.qa.mixed_node.SqlCompatIT" -Dtests.method="testCreateCursorWithFormatTxtOnNewNode" -Dtests.seed=4511623034E378D5 -Dtests.bwc=true -Dtests.locale=it-IT -Dtests.timezone=Pacific/Marquesas -Druntime.java=21

Applicable branches: main

Reproduces locally?: No

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.sql.qa.mixed_node.SqlCompatIT&tests.test=testCreateCursorWithFormatTxtOnNewNode

Failure excerpt:

org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:40695], URI [_sql?format=txt], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"verification_exception","reason":"Found 1 problem\nline 1:8: Unknown column [foo]"}],"type":"verification_exception","reason":"Found 1 problem\nline 1:8: Unknown column [foo]"},"status":400}

  at __randomizedtesting.SeedInfo.seed([4511623034E378D5:F59A16B7A97B687F]:0)
  at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
  at org.elasticsearch.xpack.sql.qa.mixed_node.SqlCompatIT.testCreateCursorWithFormatTxt(SqlCompatIT.java:200)
  at org.elasticsearch.xpack.sql.qa.mixed_node.SqlCompatIT.testCreateCursorWithFormatTxtOnNewNode(SqlCompatIT.java:177)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

[elasticsearchmachine]

Pinging @elastic/es-ql (Team:QL)

[alex-spies]

Labelling as low risk: Looks like only the test cluster is somehow misconfigured.

I get ~a reproduction~ a different error when trying to reproduce locally; seems like the test is run differently in CI than locally.

Locally, looks like the BWC cluster is somehow borked. I do not get the specific ValidationException from the build scan, instead I get

org.elasticsearch.xpack.sql.qa.mixed_node.SqlCompatIT > testCreateCursorWithFormatTxtOnNewNode FAILED
    java.lang.IllegalArgumentException: hosts must not be null nor empty
        at __randomizedtesting.SeedInfo.seed([F29B2803D747C007:42105C844ADFD0AD]:0)
        at org.elasticsearch.client.RestClient.builder(RestClient.java:212)
        at org.elasticsearch.test.rest.ESRestTestCase.buildClient(ESRestTestCase.java:1371)
        at org.elasticsearch.xpack.sql.qa.mixed_node.SqlCompatIT.initBwcClients(SqlCompatIT.java:58)

And this happens because we're expecting nodes on version 8.11.2 but find 8.11.4.

[alex-spies]

In the build scan I can see that, in fact, the bwc nodes were on 8.11.2-SNAPSHOT, so the IAE from the previous remark only happens when trying to reproduce locally, unfortunately.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[kingherc]

One more at elasticsearch / periodic / 8.12 / 8.11.4 / bwc

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

This issue has been closed because it has been open for too long with no activity.

Any muted tests that were associated with this issue have been unmuted.

If the tests begin failing again, a new issue will be opened, and they may be muted again.