search

opensearch-project / security

šŸ” Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
189 stars 271 forks source link

Ensure admin password changes work with TAR distribution #3634

Closed DarshitChanpura closed 8 months ago

DarshitChanpura commented 10 months ago

Description: The changes related to the admin password need to be thoroughly tested for all supported distributions. To achieve this, a systematic approach is required. We will initiate the testing process with one distribution, ensuring end-to-end verification to guarantee the intended functionality of the changes.

Initially, the decision was made to proceed with Docker for testing purposes. However, it was realized that changes to Docker have a broader impact on various components and repositories (such as k8s-operator, helm charts, etc.). Following discussions with the @opensearch-project/engineering-effectiveness team offline, it has been determined that testing these changes on the TAR distribution is more feasible. The TAR distribution is known for its customizability and is relatively simple to test.

Acceptance Criteria:

stephen-crawford commented 10 months ago

[Triage] Hi @DarshitChanpura, thank you for filing this issue. Looks good.

derek-ho commented 9 months ago
  1. Run ./opensearch-tar-install.sh without no initialAdminPassword environment variable set, no initialAdminPassword.txt. a. Verify a secure password is generated b. Verify that admin:admin fails authentication c. Verify that admin:${Generated Password} succeeds authentication
  2. Run ./opensearch-tar-install.sh initialAdminPassword environment variable set a. Verify that a "weak" password fails and stops script execution in a clean way b. Verify that a "strong" password succeeds i. Verify that admin:admin fails authentication ii. Verify that admin:${initialAdminPassword} succeeds authentication
  3. Run ./opensearch-tar-install.sh no initialAdminPassword environment variable set, initialAdminPassword.txt present. a. Verify that a "weak" password fails and stops script execution in a clean way b. Verify that a "strong" password succeeds i. Verify that admin:admin fails authentication ii. Verify that admin:${initialAdminPassword.txt contents} succeeds authentication

  1. Works

    Screenshot 2023-11-29 at 3 58 38 PM Screenshot 2023-11-29 at 3 59 05 PM

  2. Works

    Screenshot 2023-11-29 at 3 24 10 PM Screenshot 2023-11-29 at 3 52 22 PM Screenshot 2023-11-29 at 3 53 15 PM

  3. Works

    Screenshot 2023-11-29 at 4 02 47 PM
Screenshot 2023-11-29 at 4 03 55 PM Screenshot 2023-11-29 at 4 05 09 PM
DarshitChanpura commented 8 months ago

TAR distribution works as expected and satisfies all exit criteria.

Expand to see the logs of all tested scenarios: (without || exit 1) * no password supplied: ``` āžœ opensearch-2.12.0 ./opensearch-tar-install.sh ### OpenSearch Security Demo Installer ### ** Warning: Do not use on production or public reachable systems ** OpenSearch install type: .tar.gz on Mac OS X 14.2.1 aarch64 OpenSearch config dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/ OpenSearch config file: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch.yml OpenSearch bin dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/bin/ OpenSearch plugins dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/plugins/ OpenSearch lib dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/ Detected OpenSearch Version: 2.12.0 Detected OpenSearch Security Version: 2.12.0.0 No custom admin password found. Please provide a password. done security done plugins k-NN libraries not found in JAVA_LIBRARY_PATH. Updating path to: :/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/plugins/opensearch-knn/lib. Starting OpenSearch WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/opensearch-2.12.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/opensearch-2.12.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2024-01-02T16:32:52,978][INFO ][o.o.n.Node ] [3c06300b34da.ant.amazon.com] version[2.12.0], pid[95427], build[tar/56eb55dd2ab143a91fbbb4ead0ea9b15853861ab/2024-01-02T01:36:41.303539435Z], OS[Mac OS X/14.2.1/aarch64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/19.0.1/19.0.1+10-21] [2024-01-02T16:32:52,979][INFO ][o.o.n.Node ] [3c06300b34da.ant.amazon.com] JVM home [/Library/Java/JavaVirtualMachines/jdk-19.jdk/Contents/Home], using bundled JDK/JRE [false] [2024-01-02T16:32:52,980][INFO ][o.o.n.Node ] [3c06300b34da.ant.amazon.com] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/folders/bx/rb4zggps1qx7p5l3n0clr6bc0000gr/T/opensearch-6954462452246767185, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0, -Dopensearch.path.conf=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2024-01-02T16:32:53,680][INFO ][o.o.s.s.t.SSLConfig ] [3c06300b34da.ant.amazon.com] SSL dual mode is disabled [2024-01-02T16:32:53,680][INFO ][o.o.s.OpenSearchSecurityPlugin] [3c06300b34da.ant.amazon.com] OpenSearch Config path is /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config [2024-01-02T16:32:53,804][INFO ][o.o.s.s.DefaultSecurityKeyStore] [3c06300b34da.ant.amazon.com] JVM supports TLSv1.3 [2024-01-02T16:32:53,805][INFO ][o.o.s.s.DefaultSecurityKeyStore] [3c06300b34da.ant.amazon.com] Config directory is /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/, from there the key- and truststore files are resolved relatively [2024-01-02T16:32:53,812][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [3c06300b34da.ant.amazon.com] uncaught exception in thread [main] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.12.0.jar:2.12.0] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.12.0.jar:2.12.0] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:791) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:482) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:409) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.12.0.jar:2.12.0] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:482) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:409) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.12.0.jar:2.12.0] ... 6 more Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested. at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:487) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) ~[?:?] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) ~[?:?] at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:294) ~[?:?] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:482) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:409) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.12.0.jar:2.12.0] ... 6 more uncaught exception in thread [main] java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:487) at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:294) at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484) at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) at org.opensearch.plugins.PluginsService.(PluginsService.java:195) at org.opensearch.node.Node.(Node.java:482) at org.opensearch.node.Node.(Node.java:409) at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) For complete error details, refer to the log at /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/logs/opensearch.log āžœ ``` * weak password supplied ``` āžœ opensearch-2.12.0 export OPENSEARCH_INITIAL_ADMIN_PASSWORD=admin āžœ opensearch-2.12.0 echo $OPENSEARCH_INITIAL_ADMIN_PASSWORD admin āžœ opensearch-2.12.0 ./opensearch-tar-install.sh ### OpenSearch Security Demo Installer ### ** Warning: Do not use on production or public reachable systems ** OpenSearch install type: .tar.gz on Mac OS X 14.2.1 aarch64 OpenSearch config dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/ OpenSearch config file: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch.yml OpenSearch bin dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/bin/ OpenSearch plugins dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/plugins/ OpenSearch lib dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/ Detected OpenSearch Version: 2.12.0 Detected OpenSearch Security Version: 2.12.0.0 Password admin is weak. Please re-try with a stronger password. done security done plugins k-NN libraries not found in JAVA_LIBRARY_PATH. Updating path to: :/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/plugins/opensearch-knn/lib. Starting OpenSearch WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/opensearch-2.12.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/opensearch-2.12.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2024-01-02T16:34:55,914][INFO ][o.o.n.Node ] [3c06300b34da.ant.amazon.com] version[2.12.0], pid[95872], build[tar/56eb55dd2ab143a91fbbb4ead0ea9b15853861ab/2024-01-02T01:36:41.303539435Z], OS[Mac OS X/14.2.1/aarch64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/19.0.1/19.0.1+10-21] [2024-01-02T16:34:55,915][INFO ][o.o.n.Node ] [3c06300b34da.ant.amazon.com] JVM home [/Library/Java/JavaVirtualMachines/jdk-19.jdk/Contents/Home], using bundled JDK/JRE [false] [2024-01-02T16:34:55,916][INFO ][o.o.n.Node ] [3c06300b34da.ant.amazon.com] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/folders/bx/rb4zggps1qx7p5l3n0clr6bc0000gr/T/opensearch-13100606043528776111, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0, -Dopensearch.path.conf=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2024-01-02T16:34:56,608][INFO ][o.o.s.s.t.SSLConfig ] [3c06300b34da.ant.amazon.com] SSL dual mode is disabled [2024-01-02T16:34:56,609][INFO ][o.o.s.OpenSearchSecurityPlugin] [3c06300b34da.ant.amazon.com] OpenSearch Config path is /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config [2024-01-02T16:34:56,746][INFO ][o.o.s.s.DefaultSecurityKeyStore] [3c06300b34da.ant.amazon.com] JVM supports TLSv1.3 [2024-01-02T16:34:56,748][INFO ][o.o.s.s.DefaultSecurityKeyStore] [3c06300b34da.ant.amazon.com] Config directory is /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/, from there the key- and truststore files are resolved relatively [2024-01-02T16:34:56,756][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [3c06300b34da.ant.amazon.com] uncaught exception in thread [main] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.12.0.jar:2.12.0] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.12.0.jar:2.12.0] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:791) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:482) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:409) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.12.0.jar:2.12.0] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:482) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:409) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.12.0.jar:2.12.0] ... 6 more Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested. at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:487) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?] at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) ~[?:?] at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) ~[?:?] at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:294) ~[?:?] at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:482) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.node.Node.(Node.java:409) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.12.0.jar:2.12.0] ... 6 more uncaught exception in thread [main] java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.] at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:487) at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:204) at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:235) at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:294) at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484) at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) at org.opensearch.plugins.PluginsService.(PluginsService.java:195) at org.opensearch.node.Node.(Node.java:482) at org.opensearch.node.Node.(Node.java:409) at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) For complete error details, refer to the log at /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/logs/opensearch.log āžœ ``` * strong password supplied ``` āžœ opensearch-2.12.0 export OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123! āžœ opensearch-2.12.0 echo $OPENSEARCH_INITIAL_ADMIN_PASSWORD myStrongPassword123! āžœ opensearch-2.12.0 ./opensearch-tar-install.sh ### OpenSearch Security Demo Installer ### ** Warning: Do not use on production or public reachable systems ** OpenSearch install type: .tar.gz on Mac OS X 14.2.1 aarch64 OpenSearch config dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/ OpenSearch config file: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch.yml OpenSearch bin dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/bin/ OpenSearch plugins dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/plugins/ OpenSearch lib dir: /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/ Detected OpenSearch Version: 2.12.0 Detected OpenSearch Security Version: 2.12.0.0 Admin password set successfully. ### Success ### Execute this script now on all your nodes and then start all nodes ### OpenSearch Security will be automatically initialized. ### If you like to change the runtime configuration ### change the files in ../../../config/opensearch-security and execute: sudo "/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/plugins/opensearch-security/tools/securityadmin.sh" -cd "/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security" -icl -key "/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/kirk-key.pem" -cert "/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/kirk.pem" -cacert "/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/root-ca.pem" -nhnv ### or run ./securityadmin_demo.sh ### To use the Security Plugin ConfigurationGUI ### To access your secured cluster open https://: and log in with admin/. ### (Ignore the SSL certificate warning because we installed self-signed demo certificates) done security done plugins k-NN libraries not found in JAVA_LIBRARY_PATH. Updating path to: :/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/plugins/opensearch-knn/lib. Starting OpenSearch WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/opensearch-2.12.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch WARNING: System::setSecurityManager will be removed in a future release WARNING: A terminally deprecated method in java.lang.System has been called WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/lib/opensearch-2.12.0.jar) WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security WARNING: System::setSecurityManager will be removed in a future release [2024-01-02T16:35:55,664][INFO ][o.o.n.Node ] [smoketestnode] version[2.12.0], pid[96029], build[tar/56eb55dd2ab143a91fbbb4ead0ea9b15853861ab/2024-01-02T01:36:41.303539435Z], OS[Mac OS X/14.2.1/aarch64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/19.0.1/19.0.1+10-21] [2024-01-02T16:35:55,666][INFO ][o.o.n.Node ] [smoketestnode] JVM home [/Library/Java/JavaVirtualMachines/jdk-19.jdk/Contents/Home], using bundled JDK/JRE [false] [2024-01-02T16:35:55,666][INFO ][o.o.n.Node ] [smoketestnode] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/folders/bx/rb4zggps1qx7p5l3n0clr6bc0000gr/T/opensearch-16223761979105106139, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0, -Dopensearch.path.conf=/Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2024-01-02T16:35:56,369][INFO ][o.o.s.s.t.SSLConfig ] [smoketestnode] SSL dual mode is disabled [2024-01-02T16:35:56,369][INFO ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] OpenSearch Config path is /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config [2024-01-02T16:35:56,483][INFO ][o.o.s.s.DefaultSecurityKeyStore] [smoketestnode] JVM supports TLSv1.3 [2024-01-02T16:35:56,485][INFO ][o.o.s.s.DefaultSecurityKeyStore] [smoketestnode] Config directory is /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/, from there the key- and truststore files are resolved relatively [2024-01-02T16:35:56,730][INFO ][o.o.s.s.DefaultSecurityKeyStore] [smoketestnode] TLS Transport Client Provider : JDK [2024-01-02T16:35:56,730][INFO ][o.o.s.s.DefaultSecurityKeyStore] [smoketestnode] TLS Transport Server Provider : JDK [2024-01-02T16:35:56,730][INFO ][o.o.s.s.DefaultSecurityKeyStore] [smoketestnode] TLS HTTP Provider : JDK [2024-01-02T16:35:56,731][INFO ][o.o.s.s.DefaultSecurityKeyStore] [smoketestnode] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2] [2024-01-02T16:35:56,731][INFO ][o.o.s.s.DefaultSecurityKeyStore] [smoketestnode] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2] [2024-01-02T16:35:56,748][INFO ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] Clustername: opensearch [2024-01-02T16:35:56,751][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] Directory /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config has insecure file permissions (should be 0700) [2024-01-02T16:35:56,752][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] File /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/kirk.pem has insecure file permissions (should be 0600) [2024-01-02T16:35:56,752][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] File /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/esnode-key.pem has insecure file permissions (should be 0600) [2024-01-02T16:35:56,752][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] File /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/root-ca.pem has insecure file permissions (should be 0600) [2024-01-02T16:35:56,752][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] File /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/kirk-key.pem has insecure file permissions (should be 0600) [2024-01-02T16:35:56,752][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] File /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/internal_users.yml has insecure file permissions (should be 0600) [2024-01-02T16:35:56,752][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] File /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/esnode.pem has insecure file permissions (should be 0600) [2024-01-02T16:35:56,752][WARN ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] File /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/securityadmin_demo.sh has insecure file permissions (should be 0600) [2024-01-02T16:35:57,089][INFO ][o.o.i.r.ReindexPlugin ] [smoketestnode] ReindexPlugin reloadSPI called [2024-01-02T16:35:57,090][INFO ][o.o.i.r.ReindexPlugin ] [smoketestnode] Unable to find any implementation for RemoteReindexExtension [2024-01-02T16:35:57,109][INFO ][o.o.j.JobSchedulerPlugin ] [smoketestnode] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource [2024-01-02T16:35:57,109][INFO ][o.o.j.JobSchedulerPlugin ] [smoketestnode] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job [2024-01-02T16:35:57,112][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [aggs-matrix-stats] [2024-01-02T16:35:57,112][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [analysis-common] [2024-01-02T16:35:57,112][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [geo] [2024-01-02T16:35:57,112][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [ingest-common] [2024-01-02T16:35:57,112][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [ingest-geoip] [2024-01-02T16:35:57,112][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [ingest-user-agent] [2024-01-02T16:35:57,112][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [lang-expression] [2024-01-02T16:35:57,113][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [lang-mustache] [2024-01-02T16:35:57,113][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [lang-painless] [2024-01-02T16:35:57,113][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [mapper-extras] [2024-01-02T16:35:57,113][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [opensearch-dashboards] [2024-01-02T16:35:57,113][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [parent-join] [2024-01-02T16:35:57,113][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [percolator] [2024-01-02T16:35:57,115][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [rank-eval] [2024-01-02T16:35:57,115][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [reindex] [2024-01-02T16:35:57,115][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [repository-url] [2024-01-02T16:35:57,115][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [search-pipeline-common] [2024-01-02T16:35:57,115][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [systemd] [2024-01-02T16:35:57,115][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded module [transport-netty4] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-custom-codecs] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-flow-framework] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-geospatial] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-job-scheduler] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-knn] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-ml] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-neural-search] [2024-01-02T16:35:57,116][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-notifications] [2024-01-02T16:35:57,117][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-notifications-core] [2024-01-02T16:35:57,117][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-security] [2024-01-02T16:35:57,117][INFO ][o.o.p.PluginsService ] [smoketestnode] loaded plugin [opensearch-security-analytics] [2024-01-02T16:35:57,144][INFO ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml [2024-01-02T16:35:57,145][INFO ][o.o.e.ExtensionsManager ] [smoketestnode] ExtensionsManager initialized [2024-01-02T16:35:57,155][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [smoketestnode] Using MemorySegmentIndexInput with Java 19; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false [2024-01-02T16:35:57,160][INFO ][o.o.e.NodeEnvironment ] [smoketestnode] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk3s5)]], net usable_space [48.3gb], net total_space [228.2gb], types [apfs] [2024-01-02T16:35:57,160][INFO ][o.o.e.NodeEnvironment ] [smoketestnode] heap size [1gb], compressed ordinary object pointers [true] [2024-01-02T16:35:57,178][INFO ][o.o.n.Node ] [smoketestnode] node name [smoketestnode], node ID [klBwL_2lSfu8UhHYVvMN8g], cluster name [opensearch], roles [ingest, remote_cluster_client, data, cluster_manager] [2024-01-02T16:35:58,399][INFO ][o.o.n.p.NeuralSearch ] [smoketestnode] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled] [2024-01-02T16:35:58,617][WARN ][o.o.s.c.Salt ] [smoketestnode] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes [2024-01-02T16:35:58,643][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Message routing enabled: true [2024-01-02T16:35:58,667][INFO ][o.o.s.f.SecurityFilter ] [smoketestnode] indices are made immutable. [2024-01-02T16:35:58,763][INFO ][o.o.m.b.MLCircuitBreakerService] [smoketestnode] Registered ML memory breaker. [2024-01-02T16:35:58,764][INFO ][o.o.m.b.MLCircuitBreakerService] [smoketestnode] Registered ML disk breaker. [2024-01-02T16:35:58,764][INFO ][o.o.m.b.MLCircuitBreakerService] [smoketestnode] Registered ML native memory breaker. [2024-01-02T16:35:58,814][INFO ][o.r.Reflections ] [smoketestnode] Reflections took 24 ms to scan 1 urls, producing 21 keys and 58 values [2024-01-02T16:35:58,970][INFO ][o.o.t.NettyAllocator ] [smoketestnode] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}] [2024-01-02T16:35:58,973][INFO ][o.o.s.s.t.SSLConfig ] [smoketestnode] SSL dual mode is disabled [2024-01-02T16:35:59,029][INFO ][o.o.d.DiscoveryModule ] [smoketestnode] using discovery type [zen] and seed hosts providers [settings] [2024-01-02T16:35:59,206][WARN ][o.o.g.DanglingIndicesState] [smoketestnode] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2024-01-02T16:35:59,412][INFO ][o.o.n.Node ] [smoketestnode] initialized [2024-01-02T16:35:59,412][INFO ][o.o.n.Node ] [smoketestnode] starting ... [2024-01-02T16:35:59,427][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [windows_logtype.json] log type [2024-01-02T16:35:59,428][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [waf_logtype.json] log type [2024-01-02T16:35:59,428][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [vpcflow_logtype.json] log type [2024-01-02T16:35:59,428][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [test_windows_logtype.json] log type [2024-01-02T16:35:59,429][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [s3_logtype.json] log type [2024-01-02T16:35:59,429][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [others_web_logtype.json] log type [2024-01-02T16:35:59,430][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [others_proxy_logtype.json] log type [2024-01-02T16:35:59,430][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [others_macos_logtype.json] log type [2024-01-02T16:35:59,431][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [others_compliance_logtype.json] log type [2024-01-02T16:35:59,431][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [others_cloud_logtype.json] log type [2024-01-02T16:35:59,431][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [others_apt_logtype.json] log type [2024-01-02T16:35:59,432][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [others_application_logtype.json] log type [2024-01-02T16:35:59,432][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [okta_logtype.json] log type [2024-01-02T16:35:59,432][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [network_logtype.json] log type [2024-01-02T16:35:59,433][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [netflow_logtype.json] log type [2024-01-02T16:35:59,433][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [m365_logtype.json] log type [2024-01-02T16:35:59,434][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [linux_logtype.json] log type [2024-01-02T16:35:59,434][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [gworkspace_logtype.json] log type [2024-01-02T16:35:59,434][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [github_logtype.json] log type [2024-01-02T16:35:59,435][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [dns_logtype.json] log type [2024-01-02T16:35:59,437][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [cloudtrail_logtype.json] log type [2024-01-02T16:35:59,438][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [azure_logtype.json] log type [2024-01-02T16:35:59,438][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [apache_access_logtype.json] log type [2024-01-02T16:35:59,439][INFO ][o.o.s.l.BuiltinLogTypeLoader] [smoketestnode] Loaded [ad_ldap_logtype.json] log type [2024-01-02T16:35:59,492][INFO ][o.o.t.TransportService ] [smoketestnode] publish_address {11.128.76.37:9300}, bound_addresses {[::]:9300} [2024-01-02T16:35:59,493][INFO ][o.o.t.TransportService ] [smoketestnode] Remote clusters initialized successfully. [2024-01-02T16:35:59,610][INFO ][o.o.b.BootstrapChecks ] [smoketestnode] bound or publishing to a non-loopback address, enforcing bootstrap checks [2024-01-02T16:35:59,617][INFO ][o.o.c.c.Coordinator ] [smoketestnode] setting initial configuration to VotingConfiguration{klBwL_2lSfu8UhHYVvMN8g} [2024-01-02T16:35:59,717][INFO ][o.o.c.s.MasterService ] [smoketestnode] elected-as-cluster-manager ([1] nodes joined)[{smoketestnode}{klBwL_2lSfu8UhHYVvMN8g}{ko-9beiLRcaXWs6jhGsyYA}{11.128.76.37}{11.128.76.37:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{smoketestnode}{klBwL_2lSfu8UhHYVvMN8g}{ko-9beiLRcaXWs6jhGsyYA}{11.128.76.37}{11.128.76.37:9300}{dimr}{shard_indexing_pressure_enabled=true}]} [2024-01-02T16:35:59,746][INFO ][o.o.c.c.CoordinationState] [smoketestnode] cluster UUID set to [WK2VjA-jS8C2j3ZtBxvJPg] [2024-01-02T16:35:59,782][INFO ][o.o.c.s.ClusterApplierService] [smoketestnode] cluster-manager node changed {previous [], current [{smoketestnode}{klBwL_2lSfu8UhHYVvMN8g}{ko-9beiLRcaXWs6jhGsyYA}{11.128.76.37}{11.128.76.37:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2024-01-02T16:35:59,788][INFO ][o.o.d.PeerFinder ] [smoketestnode] setting findPeersInterval to [1s] as node commission status = [true] for local node [{smoketestnode}{klBwL_2lSfu8UhHYVvMN8g}{ko-9beiLRcaXWs6jhGsyYA}{11.128.76.37}{11.128.76.37:9300}{dimr}{shard_indexing_pressure_enabled=true}] [2024-01-02T16:35:59,791][INFO ][o.o.h.AbstractHttpServerTransport] [smoketestnode] publish_address {11.128.76.37:9201}, bound_addresses {[::]:9201} [2024-01-02T16:35:59,791][INFO ][o.o.n.Node ] [smoketestnode] started [2024-01-02T16:35:59,791][INFO ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] Node started [2024-01-02T16:35:59,792][INFO ][o.o.s.c.ConfigurationRepository] [smoketestnode] Will attempt to create index .opendistro_security and default configs if they are absent [2024-01-02T16:35:59,792][INFO ][o.o.s.c.ConfigurationRepository] [smoketestnode] Background init thread started. Install default config?: true [2024-01-02T16:35:59,792][INFO ][o.o.s.c.ConfigurationRepository] [smoketestnode] Wait for cluster to be available ... [2024-01-02T16:35:59,792][INFO ][o.o.s.OpenSearchSecurityPlugin] [smoketestnode] 0 OpenSearch Security modules loaded so far: [] [2024-01-02T16:35:59,810][INFO ][o.o.g.GatewayService ] [smoketestnode] recovered [0] indices into cluster_state [2024-01-02T16:35:59,819][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/g7WXQcSXSXmeIN9VlSHfdQ] [2024-01-02T16:35:59,873][INFO ][o.o.c.m.MetadataCreateIndexService] [smoketestnode] [.opensearch-sap-log-types-config] creating index, cause [auto(sap-logtype api)], templates [], shards [1]/[1] [2024-01-02T16:35:59,876][INFO ][o.o.c.r.a.AllocationService] [smoketestnode] updating number_of_replicas to [0] for indices [.opensearch-sap-log-types-config] [2024-01-02T16:35:59,921][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/g7WXQcSXSXmeIN9VlSHfdQ] [2024-01-02T16:36:00,068][INFO ][o.o.c.r.a.AllocationService] [smoketestnode] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opensearch-sap-log-types-config][0]]]). [2024-01-02T16:36:00,108][INFO ][o.o.s.l.LogTypeService ] [smoketestnode] Loading builtin types! [2024-01-02T16:36:00,111][INFO ][o.o.s.l.LogTypeService ] [smoketestnode] Indexing [429] fieldMappingDocs from logTypes: 24 [2024-01-02T16:36:00,147][INFO ][o.o.s.l.LogTypeService ] [smoketestnode] Indexing [429] fieldMappingDocs [2024-01-02T16:36:00,168][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/g7WXQcSXSXmeIN9VlSHfdQ] [2024-01-02T16:36:00,174][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opensearch-sap-log-types-config/g7WXQcSXSXmeIN9VlSHfdQ] update_mapping [_doc] [2024-01-02T16:36:00,216][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/g7WXQcSXSXmeIN9VlSHfdQ] [2024-01-02T16:36:00,220][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opensearch-sap-log-types-config/g7WXQcSXSXmeIN9VlSHfdQ] update_mapping [_doc] [2024-01-02T16:36:00,372][INFO ][o.o.s.l.LogTypeService ] [smoketestnode] Loaded [429] field mapping docs successfully! [2024-01-02T16:36:00,380][INFO ][o.o.s.l.LogTypeService ] [smoketestnode] Indexing [23] customLogTypes [2024-01-02T16:36:00,408][INFO ][o.o.s.l.LogTypeService ] [smoketestnode] Loaded [23] customLogType docs successfully! [2024-01-02T16:36:00,409][INFO ][o.o.s.SecurityAnalyticsPlugin] [smoketestnode] LogType config index successfully created and builtin log types loaded [2024-01-02T16:36:00,796][INFO ][o.o.s.i.DetectorIndexManagementService] [smoketestnode] No Old Correlation Indices to delete [2024-01-02T16:36:00,800][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:00,801][INFO ][o.o.c.m.MetadataCreateIndexService] [smoketestnode] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1] [2024-01-02T16:36:00,802][INFO ][o.o.c.r.a.AllocationService] [smoketestnode] updating number_of_replicas to [0] for indices [.opendistro_security] [2024-01-02T16:36:00,825][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:00,852][INFO ][o.o.s.i.DetectorIndexManagementService] [smoketestnode] No Old Alert Indices to delete [2024-01-02T16:36:00,852][INFO ][o.o.s.i.DetectorIndexManagementService] [smoketestnode] No Old Finding Indices to delete [2024-01-02T16:36:00,922][INFO ][o.o.c.r.a.AllocationService] [smoketestnode] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]). [2024-01-02T16:36:00,965][INFO ][o.o.s.c.ConfigurationRepository] [smoketestnode] Index .opendistro_security created?: true [2024-01-02T16:36:00,966][INFO ][o.o.s.c.ConfigurationRepository] [smoketestnode] Node started, try to initialize it. Wait for at least yellow cluster state.... [2024-01-02T16:36:00,968][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'config' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-01-02T16:36:00,990][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:00,992][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] create_mapping [2024-01-02T16:36:01,035][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'config' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,039][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'roles' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-01-02T16:36:01,045][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,047][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,098][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'roles' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,098][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'rolesmapping' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-01-02T16:36:01,103][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,107][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,168][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'rolesmapping' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,168][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'internalusers' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-01-02T16:36:01,177][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,180][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,225][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'internalusers' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,225][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'actiongroups' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-01-02T16:36:01,228][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,231][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,282][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'actiongroups' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,282][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'tenants' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-01-02T16:36:01,288][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,293][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,349][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'tenants' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,349][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'nodesdn' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2024-01-02T16:36:01,352][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,355][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,409][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'nodesdn' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,409][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'whitelist' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2024-01-02T16:36:01,415][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,419][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,481][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'whitelist' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,481][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'allowlist' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true [2024-01-02T16:36:01,484][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,487][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,546][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'allowlist' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,549][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Will update 'audit' with /Users/dchanp/Documents/amazon/testing-playground/test/admin-admin/rc/tar/opensearch-2.12.0/config/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false [2024-01-02T16:36:01,565][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] [2024-01-02T16:36:01,568][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [.opendistro_security/_i0AuffSQdOwnrMj2o5_HQ] update_mapping [_doc] [2024-01-02T16:36:01,613][INFO ][o.o.s.s.ConfigHelper ] [smoketestnode] Doc with id 'audit' and version 2 is updated in .opendistro_security index. [2024-01-02T16:36:01,666][INFO ][stdout ] [smoketestnode] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl [2024-01-02T16:36:01,667][INFO ][stdout ] [smoketestnode] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing on REST API is enabled. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing on Transport API is enabled. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing of request body is enabled. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Bulk requests resolution is disabled during request auditing. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Index resolution is enabled during request auditing. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Sensitive headers auditing is enabled. [2024-01-02T16:36:01,667][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing requests from kibanaserver users is disabled. [2024-01-02T16:36:01,668][WARN ][o.o.s.a.r.AuditMessageRouter] [smoketestnode] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint [2024-01-02T16:36:01,668][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing of external configuration is disabled. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing of internal configuration is enabled. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing only metadata information for read request is enabled. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing will watch {} for read requests. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing read operation requests from kibanaserver users is disabled. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing only metadata information for write request is enabled. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing diffs for write requests is disabled. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing write operation requests from kibanaserver users is disabled. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Auditing will watch for write requests. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] .opendistro_security is used as internal security index. [2024-01-02T16:36:01,669][INFO ][o.o.s.a.i.AuditLogImpl ] [smoketestnode] Internal index used for posting audit logs is null [2024-01-02T16:36:01,669][INFO ][o.o.s.c.ConfigurationRepository] [smoketestnode] Hot-reloading of audit configuration is enabled [2024-01-02T16:36:01,669][INFO ][o.o.s.c.ConfigurationRepository] [smoketestnode] Node 'smoketestnode' initialized [2024-01-02T16:36:09,808][ERROR][o.o.m.a.MLModelAutoReDeployer] [smoketestnode] Failed to query need auto redeploy models, no action will be performed, addedNodes are: [klBwL_2lSfu8UhHYVvMN8g] org.opensearch.index.IndexNotFoundException: no such index [.plugins-ml-model] at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1064) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1001) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:957) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:255) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:231) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:1041) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1081) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:925) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$4(TransportSearchAction.java:570) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.12.0.jar:2.12.0] at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:530) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.12.0.jar:2.12.0] at org.opensearch.search.pipeline.Pipeline.transformRequest(Pipeline.java:125) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:34) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:537) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:387) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:129) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.ml.autoredeploy.MLModelAutoReDeployer.queryRunningModels(MLModelAutoReDeployer.java:245) [opensearch-ml-2.12.0.0.jar:2.12.0.0] at org.opensearch.ml.autoredeploy.MLModelAutoReDeployer.triggerAutoDeployModels(MLModelAutoReDeployer.java:191) [opensearch-ml-2.12.0.0.jar:2.12.0.0] at org.opensearch.ml.autoredeploy.MLModelAutoReDeployer.buildAutoReloadArrangement(MLModelAutoReDeployer.java:144) [opensearch-ml-2.12.0.0.jar:2.12.0.0] at org.opensearch.ml.cluster.MLCommonsClusterManagerEventListener.lambda$onClusterManager$3(MLCommonsClusterManagerEventListener.java:86) [opensearch-ml-2.12.0.0.jar:2.12.0.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:852) [opensearch-2.12.0.jar:2.12.0] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] at java.base/java.lang.Thread.run(Thread.java:1589) [?:?] [2024-01-02T16:36:09,815][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [smoketestnode] Starting ML sync up job... [2024-01-02T16:36:19,828][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.plugins-ml-config/k6dm47T5Qeuwbm55O6_9Fg] [2024-01-02T16:36:19,835][INFO ][o.o.c.m.MetadataCreateIndexService] [smoketestnode] [.plugins-ml-config] creating index, cause [api], templates [], shards [1]/[1] [2024-01-02T16:36:19,836][INFO ][o.o.c.r.a.AllocationService] [smoketestnode] updating number_of_replicas to [0] for indices [.plugins-ml-config] [2024-01-02T16:36:19,883][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[.plugins-ml-config/k6dm47T5Qeuwbm55O6_9Fg] [2024-01-02T16:36:19,975][INFO ][o.o.c.r.a.AllocationService] [smoketestnode] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-config][0]]]). [2024-01-02T16:36:20,011][INFO ][o.o.m.e.i.MLIndicesHandler] [smoketestnode] create index:.plugins-ml-config [2024-01-02T16:36:20,043][INFO ][o.o.m.c.MLSyncUpCron ] [smoketestnode] ML configuration initialized successfully [2024-01-02T16:39:36,974][WARN ][o.o.s.a.BackendRegistry ] [smoketestnode] Authentication finally failed for admin from [::1]:59808 [2024-01-02T16:39:36,982][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[security-auditlog-2024.01.02/3IJfnIeTRkumG2RUY3AE0Q] [2024-01-02T16:39:37,038][INFO ][o.o.c.m.MetadataCreateIndexService] [smoketestnode] [security-auditlog-2024.01.02] creating index, cause [auto(bulk api)], templates [], shards [1]/[1] [2024-01-02T16:39:37,079][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[security-auditlog-2024.01.02/3IJfnIeTRkumG2RUY3AE0Q] [2024-01-02T16:39:37,233][INFO ][o.o.p.PluginsService ] [smoketestnode] PluginService:onIndexModule index:[security-auditlog-2024.01.02/3IJfnIeTRkumG2RUY3AE0Q] [2024-01-02T16:39:37,240][INFO ][o.o.c.m.MetadataMappingService] [smoketestnode] [security-auditlog-2024.01.02/3IJfnIeTRkumG2RUY3AE0Q] create_mapping ``` (verified with curl in a separate terminal window) ``` āžœ ~ curl -XGET https://localhost:9200/ -k -u admin:admin āžœ ~ curl -XGET https://localhost:9200/ -k -u admin:myStrongPassword123! { "name" : "smoketestnode", "cluster_name" : "opensearch", "cluster_uuid" : "WK2VjA-jS8C2j3ZtBxvJPg", "version" : { "distribution" : "opensearch", "number" : "2.12.0", "build_type" : "tar", "build_hash" : "56eb55dd2ab143a91fbbb4ead0ea9b15853861ab", "build_date" : "2024-01-02T01:36:41.303539435Z", "build_snapshot" : false, "lucene_version" : "9.8.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } ```