Updates integTest behavior to run against local and remote cluster

[DarshitChanpura]

• related to: https://github.com/opensearch-project/index-management/issues/1086#issuecomment-1927084271

Recent CI build failed with:

* Where:
Build file '/tmp/tmp19lh39y9/index-management/build.gradle' line: 441

* What went wrong:
Execution failed for task ':integTest'.
> `cluster{::integTest}` failed to wait for cluster health yellow after 40 SECONDS
    Unexpected end of file from server

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 6m 23s

This is because the previous PR: https://github.com/opensearch-project/index-management/pull/1076/ assumed that the password will always be admin for integTest task since a custom configuration is applied for setup. However, that is not the case when integTest is triggered via opensearch-build as it uses a remote-cluster. Hence, this PR defaults integTest task user password to admin for local cluster and conditionally accepts the password if running against remote cluster.

CheckList:

• [x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (ba24f86) 74.91% compared to head (be40dc9) 74.93%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1088 +/- ## ============================================ + Coverage 74.91% 74.93% +0.02% Complexity 2813 2813 ============================================ Files 367 367 Lines 16522 16522 Branches 2363 2363 ============================================ + Hits 12377 12381 +4 + Misses 2843 2840 -3 + Partials 1302 1301 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[r1walz]

Hi, @DarshitChanpura. QQ: As per #1064, there an effort to reduce usage of "admin" or other custom strings for providing password inputs and it's recommended to use OPENSEARCH_INITIAL_ADMIN_PASSWORD env/system variable. Should we incorporate that here as well? I'm unsure how this flow works. Please, provide clarification.

Moreover, I see that logs for jenkins/index-management/with-security is filled with following errors:

[2024-02-05T13:31:20,070][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node_name_9200] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f636c75737465722f6865616c74683f776169745f666f725f6e6f6465733d3e3d3126776169745f666f725f7374617475733d79656c6c6f7720485454502f312e310d0a417574686f72697a6174696f6e3a20426173696320595752746157343662586c5464484a76626d645159584e7a643239795a4445794d79453d0d0a557365722d4167656e743a204a6176612f32312e302e310d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a

Could this be the issue affecting test setup?

Thanks.

[DarshitChanpura]

Hi, @DarshitChanpura. QQ: As per https://github.com/opensearch-project/index-management/issues/1064, there an effort to reduce usage of "admin" or other custom strings for providing password inputs and it's recommended to use OPENSEARCH_INITIAL_ADMIN_PASSWORD env/system variable. Should we incorporate that here as well? I'm unsure how this flow works. Please, provide clarification.

This is because the previous PR: https://github.com/opensearch-project/index-management/pull/1076 assumed that the password will always be admin for integTest task since a custom configuration is applied for setup. However, that is not the case when integTest is triggered via opensearch-build it uses a remote-cluster. OPENSEARCH_INITIAL_ADMIN_PASSWORD is irrelevant here since the demo configuration is not used to setup security. Instead, a custom configuration is applied. Hence, this PR defaults integTest task user password to admin for local cluster and conditionally accepts the password if running against remote cluster. I have mentioned the same in PR description as well

Moreover, I see that logs for jenkins/index-management/with-security is filled with following errors: Could this be the issue affecting test setup?

Its because SSL verification failed. -k option bypasses SSL verification when using cURL. Yes, this has to do with test setup, however it is not a deal-breaker.

Please note, I'm not a maintainer here so I have limited knowledge about workflows. I'm helping out to ensure that CI is unblocked due to changes in default admin credentials that were introduced in security plugin. If you have any questions about workflow setup, the maintainers would be able to help answer them the best.

[r1walz]

Thanks for detailing my understanding. LGTM 👍 @DarshitChanpura ty