[BUG] Unable to spin up security cluster with SSL config

[jmazanec15]

Describe the bug In k-NN plugin, I am working on adding a gradle task to be able to run gradle cluster with security plugin installed: https://github.com/opensearch-project/k-NN/pull/1307. The main goal is to remove dependency on staging docker image in our CI to run security tests.

It is close to working, but running into an error when running the waitforallconditions check: https://github.com/jmazanec15/k-NN-1/blob/issue-901/build.gradle#L383. This code can be found here: https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java#L569-L581

I get the following error:

./gradlew run -Dsecurity.enabled=1
...
* What went wrong:
Execution failed for task ':run'.
> `cluster{::integTest}` failed to wait for cluster health yellow after 40 SECONDS
    IO error while waiting cluster
    401 Unauthorized

When I comment this out, the cluster is able to run and Im able to hit it with the demo certs using ssl:

        //         getClusters().forEach { cluster ->
        //             cluster.waitForAllConditions()
        // }

I did notice that SQL was able to add this: https://github.com/opensearch-project/sql/pull/1986, but I did notice that they do not use ssl for their cluster and they do not have this cluster.waitForAllConditions check.

To Reproduce Refer here: https://github.com/opensearch-project/k-NN/pull/1307#issue-1992036288

Expected behavior Expect cluster.waitForAllConditions() to be able to run health check when ssl is enabled.

Plugins

1. opensearch-knn
2. opensearch-security

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: mac
• Version main

[andrross]

[Triage - attendees 1 2 3] @jmazanec15 Thanks for filing this issue. Looks like the PR is stalled. How can we make progress here?