Introduce new env var for adding/overriding jvm options in CLI tools

[Swarnim-singhal]

Is your feature request related to a problem? Please describe.

OpenSearch currently provides the following CLI tools

• opensearch-keystore
• opensearch-node
• opensearch-shard
• opensearch-plugin

All these CLI tools are invoked with default jvm args as -Xms4m -Xmx64m -XX:+UseSerialGC ${OPENSEARCH_JAVA_OPTS} which was added by this PR to close this issue.

With existing config, the same OPENSEARCH_JAVA_OPTS which are used to start the opensearch process, are also being used by CLI commands. This leads to below problems :

• If user overrides the heap sizes in OPENSEARCH_JAVA_OPTS in order to assign more heap to the opensearch process, the cli-tools also run with assigned heap sizes, resulting in increased cpu cycles.
• If user overrides the GC algo for their opensearch process, the cli command will fail since JVM will fail to initialize with Multiple garbage collectors selected error.
  • Setting GC in OPENSEARCH_JAVA_OPTS also causes a failure line in opensearch startup as the script invokes the opensearch-keystore cli tool for performing a keystore check.(code reference)
  • To get around the issue, user has to explicitly set -XX:-UseSerialGC in OPENSEARCH_JAVA_OPTS.

Describe the solution you'd like Addition of a new env variable like OPENSEARCH_CLI_JAVA_OPTS in opensearch-cli script

Describe alternatives you've considered Removal of default settings : The current default settings do make sense since we don't want to have large heap-sizes or sophisticated GC setup for simple and lightweight cli tools

Note The opensearch-cli tools mentioned here are not the ones which come with the opensearch-cli client tool, documented here

[dblock]

This is a good idea, PRs welcome!