[BUG] CrossClusterSearchIT fails when run against a remote cluster with security plugin turned on

acarbonetto commented 1 year ago

What is the bug? CrossClusterSearchIT fails when the security plugin is enabled. We should update the cross cluster search IT tests so that they work when the security plugin is working on the cluster.

How can one reproduce the bug? Create a multicluster with the security plugin enabled. CrossClusterSearchIT will be unable to hit the client due to SSL

  2> java.lang.IllegalStateException: Failed to perform request
        at __randomizedtesting.SeedInfo.seed([2B9C4ECD5AA96CAA:1BF8BCC4ED529521]:0)
        at org.opensearch.sql.legacy.TestUtils.isIndexExist(TestUtils.java:89)
        at org.opensearch.sql.legacy.SQLIntegTestCase.loadIndex(SQLIntegTestCase.java:201)
        at org.opensearch.sql.ppl.CrossClusterSearchIT.init(CrossClusterSearchIT.java:38)

        Caused by:
        java.io.IOException: Unrecognized SSL message, plaintext connection?
            at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:958)
            at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
            at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
            at org.opensearch.sql.legacy.TestUtils.isIndexExist(TestUtils.java:86)
            ... 2 more

            Caused by:
            javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
                at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:145)
                at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:64)
                at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:612)
                at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
                at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
                at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
                at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:278)
                at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:332)
                at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:547)
                at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
                at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
                at java.base/java.lang.Thread.run(Thread.java:833)

REPRODUCE WITH: ./gradlew ':integ-test:integTest' --tests "org.opensearch.sql.ppl.CrossClusterSearchIT" -Dtests.seed=2B9C4ECD5AA96CAA -Dtests.security.manager=false -Dtests.locale=en-IN -Dtests.timezone=America/El_Salvador -Druntime.java=17

AUTOCUT ticket: https://github.com/opensearch-project/sql/issues/1700
AUTOCUT release logs pass without security plugin: stdout.txt
AUTOCUT release logs fail with security plugin: stdout.txt
Note: failures not seen in 2.8 release: https://github.com/opensearch-project/sql/actions/runs/5126820875/jobs/9222179630

Do you have any screenshots? N/A

Do you have any additional context?

Maybe follow how AbstractMultiClustersTestCase sets up their test cases and how their clients deal with the security plugin.
Cross-cluster replication has these tests running with the security plugin enabled. See: https://github.com/opensearch-project/cross-cluster-replication/blob/main/build.gradle#L376

Yury-Fridlyand commented 1 year ago

I think we also need to run all ITs with all plugins and with Security on all release branches (x.y) on every push/PR.

acarbonetto commented 1 year ago

I think we also need to run all ITs with all plugins and with Security on all release branches (x.y) on every push/PR.

This ticket is to clean up the cross cluster search test engine so that it works with security plugin turned on

opensearch-project / sql

[BUG] CrossClusterSearchIT fails when run against a remote cluster with security plugin turned on #1707