Closed q2dg closed 2 years ago
From what I understand looks like the file path config for SSL coming from Security plugin is not configured. I'll transfer this to opensearch-build.
cc: @peternied @peterzhuamazon
@q2dg I can't reproduce with docker, running docker container run --rm --name=puto -e "discovery.type=single-node" -p 9200:9200 opensearchproject/opensearch:latest
>curl --insecure https://admin:admin@localhost:9200
{
"name" : "55051492a18e",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "KMMSlu5NRQiECA1vdss2HA",
"version" : {
"distribution" : "opensearch",
"number" : "1.2.4",
"build_type" : "tar",
"build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
"build_date" : "2022-01-14T03:38:06.881862Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
>docker pull opensearchproject/opensearch:latest
latest: Pulling from opensearchproject/opensearch
3a461b3ae562: Already exists
281f706dd97a: Pull complete
4f11d5532c74: Pull complete
2f071fa29f89: Pull complete
906ed3bd695c: Pull complete
1682fbe93720: Pull complete
Digest: sha256:0b6c57dbbfbf941f159b469ee755ea46014cd3735c446f616f59daef0b45c7d5
Status: Downloaded newer image for opensearchproject/opensearch:latest
docker.io/opensearchproject/opensearch:latest
```
>docker container run --rm --name=puto -e "discovery.type=single-node" -p 9200:9200 opensearchproject/opensearch:latest
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-1.2.4
Detected OpenSearch Security Version: 1.2.4.0
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../securityconfig and execute:
"/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://
I am going to install podman
and give it a try. @q2dg Can you get the image id, and then could you attempt to pull it again and get the image id again?
>docker image ls opensearchproject/opensearch:latest
REPOSITORY TAG IMAGE ID CREATED SIZE
opensearchproject/opensearch latest 9ae13ad440d3 3 days ago 826MB
I setup an EC2 instance running the AMI ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20211129
after installing podman following the instructions on its getting started page I was able to run
$ podman container run --rm --name=puto -e "discovery.type=single-node" -p 9200:9200 opensearchproject/opensearch:latest
✔ docker.io/opensearchproject/opensearch:latest
Trying to pull docker.io/opensearchproject/opensearch:latest...
Getting image source signatures
Copying blob 3a461b3ae562 done
Copying blob 906ed3bd695c done
Copying blob 4f11d5532c74 done
Copying blob 1682fbe93720 done
Copying blob 281f706dd97a done
Copying blob 2f071fa29f89 done
Copying config 9ae13ad440 done
Writing manifest to image destination
Storing signatures
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
...
Image ID matches what I saw when I started up docker on my windows machine
$ podman image ls opensearchproject/opensearch:latest
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/opensearchproject/opensearch latest 9ae13ad440d3 3 days ago 834 MB
@q2dg we don't have much more action to take at this point as I cannot reproduce the issue you had. Please run those image ls ...
commands, also if you can copy the full docker output as text and paste it into these comments, it might help us figure out what went wrong better
Thanks a lot for you patience.
I'm running a last-minute updated Fedora 35 system (kernel v5.15.15, podman v3.4.4)
If I run podman container run --rm --name=puto -e "discovery.type=single-node" -p 9200:9200 opensearchproject/opensearch:latest I get:
Resolved "opensearchproject/opensearch" as an alias (/home/usuari/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/opensearchproject/opensearch:latest...
Getting image source signatures
Copying blob 906ed3bd695c done
Copying blob 1682fbe93720 done
Copying blob 4f11d5532c74 done
Copying blob 3a461b3ae562 done
Copying blob 281f706dd97a done
Copying blob 2f071fa29f89 done
Copying config 9ae13ad440 done
Writing manifest to image destination
Storing signatures
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-1.2.4
Detected OpenSearch Security Version: 1.2.4.0
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../securityconfig and execute:
"/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
Enabling OpenSearch Security Plugin
[2022-01-21T20:33:39,054][INFO ][o.o.n.Node ] [f69642b92937] version[1.2.4], pid[100], build[tar/e505b10357c03ae8d26d675172402f2f2144ef0f/2022-01-14T03:38:06.881862Z], OS[Linux/5.15.15-200.fc35.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-01-21T20:33:39,059][INFO ][o.o.n.Node ] [f69642b92937] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-01-21T20:33:39,073][INFO ][o.o.n.Node ] [f69642b92937] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-16540438191390975168, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -Dopensearch.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-01-21T20:33:40,971][INFO ][o.o.s.s.t.SSLConfig ] [f69642b92937] SSL dual mode is disabled
[2022-01-21T20:33:40,972][INFO ][o.o.s.OpenSearchSecurityPlugin] [f69642b92937] OpenSearch Config path is /usr/share/opensearch/config
[2022-01-21T20:33:41,468][INFO ][o.o.s.s.DefaultSecurityKeyStore] [f69642b92937] JVM supports TLSv1.3
[2022-01-21T20:33:41,470][INFO ][o.o.s.s.DefaultSecurityKeyStore] [f69642b92937] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-01-21T20:33:41,571][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [f69642b92937] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-1.2.4.jar:1.2.4]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-1.2.4.jar:1.2.4]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]
... 6 more
Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]
... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179)
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726)
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528)
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194)
at org.opensearch.node.Node.<init>(Node.java:396)
at org.opensearch.node.Node.<init>(Node.java:319)
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch.log
Killing performance analyzer process 101
OpenSearch exited with code 1
Performance analyzer exited with code 143
The output of podman image ls is:
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/opensearchproject/opensearch latest 9ae13ad440d3 3 days ago 834 MB
BUT if I run podman container run --rm --name=puto -e "discovery.type=single-node" -p 9200:9200 opensearchproject/opensearch:1.2.3 I get this, instead:
Resolved "opensearchproject/opensearch" as an alias (/home/usuari/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/opensearchproject/opensearch:1.2.3...
Getting image source signatures
Copying blob 8b8a142162d2 done
Copying blob 5d5ff2b51d5a done
Copying blob a478f063859a done
Copying blob f4a5866299c0 done
Copying blob fa411a7d2505 done
Copying blob 3099d63a1eeb done
Copying config e72207c604 done
Writing manifest to image destination
Storing signatures
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-1.2.3
Detected OpenSearch Security Version: 1.2.3.0
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../securityconfig and execute:
"/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
Enabling OpenSearch Security Plugin
[2022-01-21T20:36:55,927][INFO ][o.o.n.Node ] [755252261b9b] version[1.2.3], pid[98], build[tar/8a529d77c7432bc45b005ac1c4ba3b2741b57d4a/2021-12-21T01:36:21.407473Z], OS[Linux/5.15.15-200.fc35.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-01-21T20:36:55,931][INFO ][o.o.n.Node ] [755252261b9b] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-01-21T20:36:55,931][INFO ][o.o.n.Node ] [755252261b9b] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-14041432674666057114, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -Dopensearch.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-01-21T20:36:57,925][INFO ][o.o.s.s.t.SSLConfig ] [755252261b9b] SSL dual mode is disabled
[2022-01-21T20:36:57,926][INFO ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] OpenSearch Config path is /usr/share/opensearch/config
[2022-01-21T20:36:58,571][INFO ][o.o.s.s.DefaultSecurityKeyStore] [755252261b9b] JVM supports TLSv1.3
[2022-01-21T20:36:58,574][INFO ][o.o.s.s.DefaultSecurityKeyStore] [755252261b9b] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-01-21T20:36:59,663][INFO ][o.o.s.s.DefaultSecurityKeyStore] [755252261b9b] TLS Transport Client Provider : JDK
[2022-01-21T20:36:59,664][INFO ][o.o.s.s.DefaultSecurityKeyStore] [755252261b9b] TLS Transport Server Provider : JDK
[2022-01-21T20:36:59,665][INFO ][o.o.s.s.DefaultSecurityKeyStore] [755252261b9b] TLS HTTP Provider : JDK
[2022-01-21T20:36:59,665][INFO ][o.o.s.s.DefaultSecurityKeyStore] [755252261b9b] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2022-01-21T20:36:59,666][INFO ][o.o.s.s.DefaultSecurityKeyStore] [755252261b9b] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2022-01-21T20:37:00,218][INFO ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] Clustername: docker-cluster
[2022-01-21T20:37:00,232][WARN ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2022-01-21T20:37:00,234][WARN ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-01-21T20:37:00,235][WARN ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-01-21T20:37:00,235][WARN ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-01-21T20:37:00,236][WARN ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-01-21T20:37:00,237][WARN ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2022-01-21T20:37:00,527][INFO ][o.o.p.c.PluginSettings ] [755252261b9b] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-01-21T20:37:01,647][INFO ][o.o.i.r.ReindexPlugin ] [755252261b9b] ReindexPlugin reloadSPI called
[2022-01-21T20:37:01,649][INFO ][o.o.i.r.ReindexPlugin ] [755252261b9b] Unable to find any implementation for RemoteReindexExtension
[2022-01-21T20:37:01,681][INFO ][o.o.j.JobSchedulerPlugin ] [755252261b9b] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-01-21T20:37:01,722][INFO ][o.o.j.JobSchedulerPlugin ] [755252261b9b] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-01-21T20:37:01,734][INFO ][o.o.j.JobSchedulerPlugin ] [755252261b9b] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-01-21T20:37:01,745][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [aggs-matrix-stats]
[2022-01-21T20:37:01,746][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [analysis-common]
[2022-01-21T20:37:01,747][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [geo]
[2022-01-21T20:37:01,748][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [ingest-common]
[2022-01-21T20:37:01,749][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [ingest-geoip]
[2022-01-21T20:37:01,749][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [ingest-user-agent]
[2022-01-21T20:37:01,750][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [lang-expression]
[2022-01-21T20:37:01,751][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [lang-mustache]
[2022-01-21T20:37:01,752][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [lang-painless]
[2022-01-21T20:37:01,753][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [mapper-extras]
[2022-01-21T20:37:01,753][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [opensearch-dashboards]
[2022-01-21T20:37:01,754][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [parent-join]
[2022-01-21T20:37:01,754][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [percolator]
[2022-01-21T20:37:01,755][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [rank-eval]
[2022-01-21T20:37:01,755][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [reindex]
[2022-01-21T20:37:01,756][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [repository-url]
[2022-01-21T20:37:01,757][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded module [transport-netty4]
[2022-01-21T20:37:01,759][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-alerting]
[2022-01-21T20:37:01,760][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-anomaly-detection]
[2022-01-21T20:37:01,761][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-asynchronous-search]
[2022-01-21T20:37:01,761][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-cross-cluster-replication]
[2022-01-21T20:37:01,762][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-index-management]
[2022-01-21T20:37:01,762][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-job-scheduler]
[2022-01-21T20:37:01,763][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-knn]
[2022-01-21T20:37:01,763][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-observability]
[2022-01-21T20:37:01,766][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-performance-analyzer]
[2022-01-21T20:37:01,766][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-reports-scheduler]
[2022-01-21T20:37:01,767][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-security]
[2022-01-21T20:37:01,768][INFO ][o.o.p.PluginsService ] [755252261b9b] loaded plugin [opensearch-sql]
[2022-01-21T20:37:01,811][INFO ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-01-21T20:37:01,835][DEPRECATION][o.o.d.c.s.Settings ] [755252261b9b] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2022-01-21T20:37:01,853][INFO ][o.o.e.NodeEnvironment ] [755252261b9b] using [1] data paths, mounts [[/ (overlay)]], net usable_space [3.8gb], net total_space [13.3gb], types [overlay]
[2022-01-21T20:37:01,854][INFO ][o.o.e.NodeEnvironment ] [755252261b9b] heap size [1gb], compressed ordinary object pointers [true]
[2022-01-21T20:37:01,933][INFO ][o.o.n.Node ] [755252261b9b] node name [755252261b9b], node ID [uJQNvbsQQEa6QU0b13iy3g], cluster name [docker-cluster], roles [master, remote_cluster_client, data, ingest]
[2022-01-21T20:37:05,581][DEPRECATION][o.o.d.c.s.Settings ] [755252261b9b] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2022-01-21T20:37:07,724][WARN ][o.o.s.c.Salt ] [755252261b9b] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-01-21T20:37:07,774][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Message routing enabled: true
[2022-01-21T20:37:07,880][INFO ][o.o.s.f.SecurityFilter ] [755252261b9b] <NONE> indices are made immutable.
[2022-01-21T20:37:08,512][INFO ][o.o.a.b.ADCircuitBreakerService] [755252261b9b] Registered memory breaker.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/opensearch/plugins/opensearch-anomaly-detection/protostuff-runtime-1.7.4.jar) to field java.lang.Throwable.cause
WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2022-01-21T20:37:09,494][INFO ][o.o.t.NettyAllocator ] [755252261b9b] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2022-01-21T20:37:09,657][INFO ][o.o.d.DiscoveryModule ] [755252261b9b] using discovery type [single-node] and seed hosts providers [settings]
[2022-01-21T20:37:10,653][WARN ][o.o.g.DanglingIndicesState] [755252261b9b] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-01-21T20:37:11,211][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [755252261b9b] PerformanceAnalyzer Enabled: false
[2022-01-21T20:37:11,430][INFO ][o.o.n.Node ] [755252261b9b] initialized
[2022-01-21T20:37:11,431][INFO ][o.o.n.Node ] [755252261b9b] starting ...
[2022-01-21T20:37:11,697][INFO ][o.o.t.TransportService ] [755252261b9b] publish_address {10.0.2.100:9300}, bound_addresses {[::]:9300}
[2022-01-21T20:37:11,891][WARN ][o.o.b.BootstrapChecks ] [755252261b9b] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2022-01-21T20:37:11,911][INFO ][o.o.c.c.Coordinator ] [755252261b9b] setting initial configuration to VotingConfiguration{uJQNvbsQQEa6QU0b13iy3g}
[2022-01-21T20:37:12,135][INFO ][o.o.c.s.MasterService ] [755252261b9b] elected-as-master ([1] nodes joined)[{755252261b9b}{uJQNvbsQQEa6QU0b13iy3g}{TmZjDDaMRFK32BTvfY4XJg}{10.0.2.100}{10.0.2.100:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{755252261b9b}{uJQNvbsQQEa6QU0b13iy3g}{TmZjDDaMRFK32BTvfY4XJg}{10.0.2.100}{10.0.2.100:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
[2022-01-21T20:37:12,210][INFO ][o.o.c.c.CoordinationState] [755252261b9b] cluster UUID set to [oJr_PrzZS72xXdeYH6zdEw]
[2022-01-21T20:37:12,226][INFO ][o.o.c.s.ClusterApplierService] [755252261b9b] master node changed {previous [], current [{755252261b9b}{uJQNvbsQQEa6QU0b13iy3g}{TmZjDDaMRFK32BTvfY4XJg}{10.0.2.100}{10.0.2.100:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2022-01-21T20:37:12,262][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [755252261b9b] Config override setting update called with empty string. Ignoring.
[2022-01-21T20:37:12,265][INFO ][o.o.a.c.ADClusterEventListener] [755252261b9b] Cluster is not recovered yet.
[2022-01-21T20:37:12,311][INFO ][o.o.h.AbstractHttpServerTransport] [755252261b9b] publish_address {10.0.2.100:9200}, bound_addresses {[::]:9200}
[2022-01-21T20:37:12,312][INFO ][o.o.n.Node ] [755252261b9b] started
[2022-01-21T20:37:12,313][INFO ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] Node started
[2022-01-21T20:37:12,314][INFO ][o.o.s.c.ConfigurationRepository] [755252261b9b] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-01-21T20:37:12,316][INFO ][o.o.s.OpenSearchSecurityPlugin] [755252261b9b] 0 OpenSearch Security modules loaded so far: []
[2022-01-21T20:37:12,317][INFO ][o.o.s.c.ConfigurationRepository] [755252261b9b] Background init thread started. Install default config?: true
[2022-01-21T20:37:12,373][INFO ][o.o.a.c.HashRing ] [755252261b9b] Node added: [uJQNvbsQQEa6QU0b13iy3g]
[2022-01-21T20:37:12,397][INFO ][o.o.a.c.HashRing ] [755252261b9b] Add data node to AD version hash ring: uJQNvbsQQEa6QU0b13iy3g
[2022-01-21T20:37:12,417][INFO ][o.o.a.c.HashRing ] [755252261b9b] All nodes with known AD version: {uJQNvbsQQEa6QU0b13iy3g=ADNodeInfo{version=1.2.3, isEligibleDataNode=true}}
[2022-01-21T20:37:12,418][INFO ][o.o.a.c.HashRing ] [755252261b9b] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2022-01-21T20:37:12,419][INFO ][o.o.a.c.HashRing ] [755252261b9b] Build AD version hash ring successfully
[2022-01-21T20:37:12,421][INFO ][o.o.g.GatewayService ] [755252261b9b] recovered [0] indices into cluster_state
[2022-01-21T20:37:12,430][INFO ][o.o.a.c.ADDataMigrator ] [755252261b9b] Start migrating AD data
[2022-01-21T20:37:12,431][INFO ][o.o.a.c.ADDataMigrator ] [755252261b9b] AD job index doesn't exist, no need to migrate
[2022-01-21T20:37:12,432][INFO ][o.o.a.c.ADClusterEventListener] [755252261b9b] Init AD version hash ring successfully
[2022-01-21T20:37:12,691][INFO ][o.o.c.m.MetadataCreateIndexService] [755252261b9b] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2022-01-21T20:37:12,705][INFO ][o.o.c.r.a.AllocationService] [755252261b9b] updating number_of_replicas to [0] for indices [.opendistro_security]
[2022-01-21T20:37:12,990][INFO ][o.o.c.r.a.AllocationService] [755252261b9b] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2022-01-21T20:37:13,041][INFO ][o.o.s.c.ConfigurationRepository] [755252261b9b] Index .opendistro_security created?: true
[2022-01-21T20:37:13,042][INFO ][o.o.s.c.ConfigurationRepository] [755252261b9b] Node started, try to initialize it. Wait for at least yellow cluster state....
[2022-01-21T20:37:13,046][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-01-21T20:37:13,223][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] create_mapping [_doc]
[2022-01-21T20:37:13,386][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'config' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:13,386][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-01-21T20:37:13,421][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:13,482][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'roles' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:13,483][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-01-21T20:37:13,524][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:13,598][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'rolesmapping' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:13,599][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-01-21T20:37:13,624][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:13,689][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'internalusers' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:13,690][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-01-21T20:37:13,722][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:13,784][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'actiongroups' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:13,786][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-01-21T20:37:13,802][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:13,850][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'tenants' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:13,851][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2022-01-21T20:37:13,882][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:13,951][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'nodesdn' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:13,952][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2022-01-21T20:37:13,966][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:14,019][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'whitelist' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:14,020][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Will update 'audit' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2022-01-21T20:37:14,062][INFO ][o.o.c.m.MetadataMappingService] [755252261b9b] [.opendistro_security/hBXfg7FcTA6FmoO-qK9tIA] update_mapping [_doc]
[2022-01-21T20:37:14,129][INFO ][o.o.s.s.ConfigHelper ] [755252261b9b] Doc with id 'audit' and version 2 is updated in .opendistro_security index.
[2022-01-21T20:37:14,349][INFO ][stdout ] [755252261b9b] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2022-01-21T20:37:14,350][INFO ][stdout ] [755252261b9b] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2022-01-21T20:37:14,352][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing on REST API is enabled.
[2022-01-21T20:37:14,355][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2022-01-21T20:37:14,356][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing on Transport API is enabled.
[2022-01-21T20:37:14,356][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2022-01-21T20:37:14,357][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing of request body is enabled.
[2022-01-21T20:37:14,358][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Bulk requests resolution is disabled during request auditing.
[2022-01-21T20:37:14,358][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Index resolution is enabled during request auditing.
[2022-01-21T20:37:14,359][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Sensitive headers auditing is enabled.
[2022-01-21T20:37:14,359][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing requests from kibanaserver users is disabled.
[2022-01-21T20:37:14,371][WARN ][o.o.s.a.r.AuditMessageRouter] [755252261b9b] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2022-01-21T20:37:14,372][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing of external configuration is disabled.
[2022-01-21T20:37:14,372][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing of internal configuration is enabled.
[2022-01-21T20:37:14,373][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing only metadata information for read request is enabled.
[2022-01-21T20:37:14,373][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing will watch {} for read requests.
[2022-01-21T20:37:14,374][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing read operation requests from kibanaserver users is disabled.
[2022-01-21T20:37:14,375][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing only metadata information for write request is enabled.
[2022-01-21T20:37:14,376][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing diffs for write requests is disabled.
[2022-01-21T20:37:14,376][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing write operation requests from kibanaserver users is disabled.
[2022-01-21T20:37:14,377][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Auditing will watch <NONE> for write requests.
[2022-01-21T20:37:14,379][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] .opendistro_security is used as internal security index.
[2022-01-21T20:37:14,380][INFO ][o.o.s.a.i.AuditLogImpl ] [755252261b9b] Internal index used for posting audit logs is null
[2022-01-21T20:37:14,381][INFO ][o.o.s.c.ConfigurationRepository] [755252261b9b] Hot-reloading of audit configuration is enabled
[2022-01-21T20:37:14,382][INFO ][o.o.s.c.ConfigurationRepository] [755252261b9b] Node '755252261b9b' initialized
The output of podman image ls now is:
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/opensearchproject/opensearch latest 9ae13ad440d3 3 days ago 834 MB
docker.io/opensearchproject/opensearch 1.2.3 e72207c604f2 4 weeks ago 845 MB
Thanks a lot again!!
When look at at a diff between the different images outputs, after the line Enabling OpenSearch Security Plugin
, the whole of the output is catted and an string subsitution takes place, it looks like that is missing from your image.
The failure is on the left and the operational one on the right
I think you'll need to inspect if the cat / sed / tee commands are failing which is aborting the rest of the script execution.
Adding @peterzhuamazon in case you have any debugging advice
Well, you're true!
I've entered (by executing podman exec -it puto /bin/bash ) into my functional v1.2.3 container and I've run cat->sed->tee pipeline manually. The fact is that running just cat or cat->sed is the same: there's no difference in the shown output (that is, sed doesn't delete any line because there's no line with the "plugins.security.disabled" string in the original opensearch.yml file). So all it's all right... BUT when sed's output is piped into tee command, this output disappears!! There's no output when running entire pipeline...tee swallows it! In fact, it's worst, because tee overwrittes original "opensearch.yml" file thus it erases all its content (resulting in a void file). The fact is that if I indicate another name for the file written by tee, it's all right: output is shown through screen and is written in final file; but if this final file is the same than the original, its content is destroyed.
Anyway, this behaviour happens in my functional container, so I don't know if this is specifically the reason why v.1.2.4 doesn't works...
Thanks a lot again!!
Does that mean you can start the 1.2.4 container?
We did change how the container updates the config file as there were reported issues with sed reading while tee was writing, but we might not have fixed this at all. Checkout https://github.com/opensearch-project/opensearch-build/pull/1130 as there is more detail in that pull request.
Previous startup script: https://github.com/opensearch-project/opensearch-build/blob/780c28d4d70dca10a2a775b672e192fdacdcd97e/docker/release/config/opensearch/opensearch-docker-entrypoint.sh#L71-L72
No, no, I can't start the v1.2.4 container. My tests have been done in the v1.2.3 one. That's why I say that I don't know to what extent this may or may not be the source of the error, but it is still an interesting investigation, anyway.
In fact, I've tried the previous startup script and I can tell you that it does respect the content on opensearch.yml file (that is, it doesn't void it). If this can help...
Thanks again!
@q2dg I'm out for the weekend, I'll check with some of our other folks on Monday that might have a better idea how to A) reproduce this and B) how to resolve this issue. We might have some experimental docker images for you to try if you are willing.
https://github.com/opensearch-project/opensearch-build/pull/1458 This one fix an issue related to #1130 where tee is reportedly failing. cat->sed->tee in #1458 should run without omitting the output like sed->tee in #1130.
Are you running on a embbed system with busybox?
Which might not have all the functionalities of a normal sed and tee.
I have seen similar issue before on a custom linux with busybox.
Will see if I can reproduce on my Rpi.
I think I get confused this should not happen regardless of the host system. We use AL2 running the entrypoint so it should not have this issue anyway. I will test on this specific system later
I'm running a last-minute updated Fedora 35 system (kernel v5.15.15, podman v3.4.4)
Thanks.
I am using Fedora 35 and then update the kernel to the latest, since 5.15.15 is not available I am using latest 5.15.16.
Installed Packages
kernel.x86_64 5.14.10-300.fc35 @anaconda
kernel.x86_64 5.15.16-200.fc35 @updates
Available Packages
kernel.x86_64 5.14.10-300.fc35 fedora
kernel.x86_64 5.15.16-200.fc35 updates
podman version 3.4.4
Podman allows me to choose from these images:
Please select an image:
registry.fedoraproject.org/opensearchproject/opensearch:1.2.4
registry.access.redhat.com/opensearchproject/opensearch:1.2.4
â–¸ docker.io/opensearchproject/opensearch:1.2.4
quay.io/opensearchproject/opensearch:1.2.4
Fedora release 35 (Thirty Five)
NAME="Fedora Linux"
VERSION="35 (Server Edition)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server
Fedora release 35 (Thirty Five)
Fedora release 35 (Thirty Five)
I am not seeing any issues running this. Weird. @q2dg are you using HDD mechanical drive not SSD?
The only thing I can think of is tee somehow happens before cat, thus empty the file by default before cat can read anything. But then the line is cat then sed through then tee, so in theory it should not happen.
I am thinking about using > directly since we are not using sed to read anymore, which should have very similar behavior to tee just no output, and inode should not change as well.
I think I need more information on your setup @q2dg as of now I cannot reproduce your situation on exactly the same system.
Thanks.
@q2dg can you run this line and let me know what is the output?
$ podman container run --rm --name=puto -e "discovery.type=single-node" -p 9200:9200 opensearchproject/opensearch:latest /bin/bash -c "cat ~/opensearch-docker-entrypoint.sh"
I want to understand whether the latest
image you are using is actually the cache on your host.
It should completely match this file:
https://github.com/peterzhuamazon/opensearch-build/blob/b33273f69cd19ac5d365d54bd306282cf5541c21/docker/release/config/opensearch/opensearch-docker-entrypoint.sh
Thanks.
Sure!
What I get is this:
#!/bin/bash
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
# This script specify the entrypoint startup actions for opensearch
# It will start both opensearch and performance analyzer plugin cli
# If either process failed, the entire docker container will be removed
# in favor of a newly started container
# Export OpenSearch Home
export OPENSEARCH_HOME=/usr/share/opensearch
# Files created by OpenSearch should always be group writable too
umask 0002
if [[ "$(id -u)" == "0" ]]; then
echo "OpenSearch cannot run as root. Please start your container as another user."
exit 1
fi
# Parse Docker env vars to customize OpenSearch
#
# e.g. Setting the env var cluster.name=testcluster
#
# will cause OpenSearch to be invoked with -Ecluster.name=testcluster
declare -a opensearch_opts
while IFS='=' read -r envvar_key envvar_value
do
# OpenSearch settings need to have at least two dot separated lowercase
# words, e.g. `cluster.name`, except for `processors` which we handle
# specially
if [[ "$envvar_key" =~ ^[a-z0-9_]+\.[a-z0-9_]+ || "$envvar_key" == "processors" ]]; then
if [[ ! -z $envvar_value ]]; then
opensearch_opt="-E${envvar_key}=${envvar_value}"
opensearch_opts+=("${opensearch_opt}")
fi
fi
done < <(env)
# The virtual file /proc/self/cgroup should list the current cgroup
# membership. For each hierarchy, you can follow the cgroup path from
# this file to the cgroup filesystem (usually /sys/fs/cgroup/) and
# introspect the statistics for the cgroup for the given
# hierarchy. Alas, Docker breaks this by mounting the container
# statistics at the root while leaving the cgroup paths as the actual
# paths. Therefore, OpenSearch provides a mechanism to override
# reading the cgroup path from /proc/self/cgroup and instead uses the
# cgroup path defined the JVM system property
# opensearch.cgroups.hierarchy.override. Therefore, we set this value here so
# that cgroup statistics are available for the container this process
# will run in.
export OPENSEARCH_JAVA_OPTS="-Dopensearch.cgroups.hierarchy.override=/ $OPENSEARCH_JAVA_OPTS"
##Security Plugin
SECURITY_PLUGIN="opensearch-security"
if [ -d "$OPENSEARCH_HOME/plugins/$SECURITY_PLUGIN" ]; then
if [ "$DISABLE_INSTALL_DEMO_CONFIG" = "true" ]; then
echo "Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin"
else
echo "Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin"
bash $OPENSEARCH_HOME/plugins/$SECURITY_PLUGIN/tools/install_demo_configuration.sh -y -i -s
fi
if [ "$DISABLE_SECURITY_PLUGIN" = "true" ]; then
echo "Disabling OpenSearch Security Plugin"
cat $OPENSEARCH_HOME/config/opensearch.yml | sed "/plugins.security.disabled/d" | tee $OPENSEARCH_HOME/config/opensearch.yml
echo "plugins.security.disabled: true" >> $OPENSEARCH_HOME/config/opensearch.yml
else
echo "Enabling OpenSearch Security Plugin"
cat $OPENSEARCH_HOME/config/opensearch.yml | sed "/plugins.security.disabled/d" | tee $OPENSEARCH_HOME/config/opensearch.yml
fi
fi
# Start up the opensearch and performance analyzer agent processes.
# When either of them halts, this script exits, or we receive a SIGTERM or SIGINT signal then we want to kill both these processes.
function terminateProcesses {
if kill -0 $OPENSEARCH_PID >& /dev/null; then
echo "Killing opensearch process $OPENSEARCH_PID"
kill -TERM $OPENSEARCH_PID
wait $OPENSEARCH_PID
fi
if kill -0 $PA_PID >& /dev/null; then
echo "Killing performance analyzer process $PA_PID"
kill -TERM $PA_PID
wait $PA_PID
fi
}
# Enable job control so we receive SIGCHLD when a child process terminates
set -m
# Make sure we terminate the child processes in the event of us received TERM (e.g. "docker container stop"), INT (e.g. ctrl-C), EXIT (this script terminates for an unexpected reason), or CHLD (one of the processes terminated unexpectedly)
trap terminateProcesses TERM INT EXIT CHLD
# Start opensearch
$OPENSEARCH_HOME/bin/opensearch "${opensearch_opts[@]}" &
OPENSEARCH_PID=$!
# Start performance analyzer agent
$OPENSEARCH_HOME/bin/performance-analyzer-agent-cli > $OPENSEARCH_HOME/logs/performance-analyzer.log 2>&1 &
PA_PID=$!
# Wait for the child processes to terminate
wait $OPENSEARCH_PID
echo "OpenSearch exited with code $?"
wait $PA_PID
echo "Performance analyzer exited with code $?"
Doing a diff with reference file whose link you give, I get no output, so it seems both files are exactly equal.
My setup is a VirtualBox machine, with standard options (disk is a VDI one with a simulated SATA connection)
Sorry for disturbing so much, and thanks again
Interesting, I am running on virtualbox as well. I am having 8G Memory + 4CPU setups as we need at least 4G for OpenSearch and Dashboards to run without issues. I dont have VDI but standard VMDK.
Here is a screenshot of my setup.
And are you running on x86_64 or ARM64 host? I have not test the latter yet. Probably it is an issue specific to ARM64 version?
My host is a x86_64 machine (in fact, it's another Fedora 35 Workstation system) You're right, RAM has to be enough... My setup is this:
lscpu's output as seen from vm is this:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 1
On-line CPU(s) list: 0
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
CPU family: 6
Model: 142
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
Stepping: 10
BogoMIPS: 3983.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc
a cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall n
x rdtscp lm constant_tsc rep_good nopl xtopology nonsto
p_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3
cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave a
vx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_
single pti fsgsbase avx2 invpcid rdseed clflushopt
Virtualization features:
Hypervisor vendor: KVM
Virtualization type: full
Caches (sum of all):
L1d: 32 KiB (1 instance)
L1i: 32 KiB (1 instance)
L2: 256 KiB (1 instance)
L3: 8 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0
Vulnerabilities:
Itlb multihit: KVM: Mitigation: VMX unsupported
L1tf: Mitigation; PTE Inversion
Mds: Vulnerable: Clear CPU buffers attempted, no microcode;
SMT Host state unknown
Meltdown: Mitigation; PTI
Spec store bypass: Vulnerable
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer
sanitization
Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB
filling
Srbds: Unknown: Dependent on hypervisor status
Tsx async abort: Not affected
Thanks!
I will try the workstation version later as I only download the Server version. I dont think there are different behavior but I have never used the workstation version, so cannot comment whether it would affect anything.
I am experiencing the same problem and I cannot get 1.2.4 to start with the same security configuration error as in the original error report.
To try to debug I took a copy of the docker image that failed to start using docker commit
and started that up to inspect. Inside this image my opensearch.yml
file has been wiped and is now completely empty (despite it not being empty when first run). I believe that the cat->sed->tee command is wiping the contents of this file. Previous versions of the docker image have been fine.
This issue seems to be intermittent and sometimes it works fine, but most of the time it fails. I've not been able to tie it down to anything more concrete than that.
I'm running on Amazon Linux version 4.14.232-177.418.amzn2.x86_64 Docker version 20.10.7, build f0df350
I've reproduced my issue of the config being wiped in the Dockerfile below, which replicates the cat > sed > tee command:
# syntax=docker/dockerfile:1.3-labs
FROM opensearchproject/opensearch:1.2.4
RUN <<EOF
### CREATE A SIMPLE CONFIG
cat <<EOT > /usr/share/opensearch/config/opensearch.yml
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
EOT
### CHANGING THE ENTRYPOINT TO JUST RUN THE cat > sed > tee COMMAND
cat <<EOT > /usr/share/opensearch/opensearch-docker-entrypoint.sh
#!/bin/bash
echo "Before:"
cat /usr/share/opensearch/config/opensearch.yml
echo "Running cat > sed > tee"
cat /usr/share/opensearch/config/opensearch.yml | sed "/plugins.security.disabled/d" | tee /usr/share/opensearch/config/opensearch.yml
echo "After:"
cat /usr/share/opensearch/config/opensearch.yml
echo
EOT
chmod a+x /usr/share/opensearch/opensearch-docker-entrypoint.sh
EOF
Build with
DOCKER_BUILDKIT=1 docker build --tag test .
Here is the output running it 4 times. Note that on the 4th execution the config file was been wiped and the file is now empty.
[root@ip-10-0-1-1 opensearch]# docker run --rm test
Before:
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
Running cat > sed > tee
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
After:
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
[root@ip-10-0-1-1 opensearch]# docker run --rm test
Before:
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
Running cat > sed > tee
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
After:
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
[root@ip-10-0-1-1 opensearch]# docker run --rm test
Before:
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
Running cat > sed > tee
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
After:
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
[root@ip-10-0-1-1 opensearch]# docker run --rm test
Before:
# Test OpenSearch yml config
cluster.name: "docker-cluster"
network.host: 0.0.0.0
Running cat > sed > tee
After:
[root@ip-10-0-1-1 opensearch]#
I don't know if this is of any help?
Could it be that the order of the execution of the cat | sed | tee
is nondeterministic as detailed here https://askubuntu.com/a/719094
If it does get executed out of order then the config file will be wiped.
@peterzhuamazon What do you think about resolve these issues once and for all with a migration to python? I've created a separate task that we might be able to bang out relatively quickly to mitigate these issues.
Maybe I'm not fully understanding the issue, but isn't part of the problem that the entrypoint is directly modifying the original config file?
When a config file is mounted into the docker container, a user probably wouldn't expect that their config file on their local machine might get modified (or even deleted!). Moreover, the config file may be mounted as read only, meaning that the code will probably fail when it is unable to write to the config.
I'm not sure how rewriting in python addresses this issue. Maybe it would be better to take a copy of the config file, modify that and then run OpenSearch using that? That said I don't believe there is a way of specifying an alternative config path to OpenSearch.
Maybe I'm not fully understanding the issue, but isn't part of the problem that the entrypoint is directly modifying the original config file?
When a config file is mounted into the docker container, a user probably wouldn't expect that their config file on their local machine might get modified (or even deleted!). Moreover, the config file may be mounted as read only, meaning that the code will probably fail when it is unable to write to the config.
I'm not sure how rewriting in python addresses this issue. Maybe it would be better to take a copy of the config file, modify that and then run OpenSearch using that? That said I don't believe there is a way of specifying an alternative config path to OpenSearch.
Hi @jgough thanks for the investigation. The reason that we initially think about modifying the file is to allow env vars to define what config to change on the fly: https://github.com/opensearch-project/opensearch-build/tree/main/docker/release#disable-security-plugin-security-dashboards-plugin-security-demo-configurations-and-related-configurations
Seems like this is not an ideal approach at this point.
The reason we go all the way to use cat/sed/tee is due to sed -i creating new inode, and mounting in docker will disallow that behavior.
Would love to have some more opinions on how this can be changed and fixed to a better approach.
@unhipzippo You have helped us to identify the issue before, thanks for that, would like to see if you have any take on this?
Thanks.
Using sponge
as suggested in the post is adding additional packages, which we want to avoid for the time being.
I am still trying to find out if there is anyway to avoid this race condition.
Thanks.
We can, however, save the output in a var, then echo the var into the file in a second line, but that is quite a messy script going forward.
NEW_CONFIG=`cat /usr/share/opensearch/config/opensearch.yml | sed "/plugins.security.disabled/d"`
echo $NEW_CONFIG > /usr/share/opensearch/config/opensearch.yml
I think there are several ways we can think about resolving this issue:
FROM opensearchproject/opensearch:1.2.4
and add the fixes so people can build it themselvesopensearchstaging/opensearch
repo and ask people to test it
- We re-release the 1.2.4 version with confirmed fix (not determined)
Please note following sem-ver we cannot re-release. We would either need to wait for next release or do a patch release fixing this or have a work around. Thanks!
@unhipzippo You have helped us to identify the issue before, thanks for that, would like to see if you have any take on this?
Thinking about it more, I think I agree with @jgough -- As an end-user, I would generally expect that config files that I bind into the container are my config, and they won't be modified in any way by the program at runtime (unless the program has communicated this to users up front and received implicit buy-in).
I wonder whether a better solution wouldn't just be to update whatever code is consulting plugins.security.disabled from opensearch.yml and instead have it consult the environment variable instead -- Then set the environment variable on startup as needed.
i.e. You end up setting the environment variable based on the config file, rather than setting the config file based on the environment variable. :)
The code in opensearch-docker-entrypoint.sh could change to something like:
if [ -z "$DISABLE_SECURITY_PLUGIN" ] || [ "$DISABLE_SECURITY_PLUGIN" != "true" ]; then
# Environment var not set -- Set it based on
# config file, or default it to false.
DISABLE_SECURITY_PLUGIN_FILE_CONFIG=$(grep "^plugins.security.disabled: " $OPENSEARCH_HOME/config/opensearch.yml | awk '{print $2}')
if [ "$DISABLE_SECURITY_PLUGIN_FILE_CONFIG" = "true" ]; then
echo "Disabling OpenSearch Security Plugin"
DISABLE_SECURITY_PLUGIN="true"
else
echo "Enabling OpenSearch Security Plugin"
DISABLE_SECURITY_PLUGIN="false"
fi
else
# DISABLE_SECURITY_PLUGIN = "true"
echo "Disabling OpenSearch Security Plugin"
# Nothing to do - value is already set
fi
This would save you from needing to modify the config file at runtime at all.
The downside is that you now need to go through the code and find anywhere that directly consults plugins.security.disabled, and have it pay attention to the environment var instead.
Are we trying to overcomplicate this?
If I run opensearch -E "plugins.security.disabled=true"
it seems to override that setting in the opensearch.yml file. Can we just add this to the command string if DISABLE_SECURITY_PLUGIN
is explicitly set?
Yep -- that might be even simpler; I hadn't checked in the code to see that was a possibility.
@unhipzippo @jgough the entrypoint will try to figure out if any env var is presenting with the configuration settings and applying during startup: https://github.com/opensearch-project/opensearch-build/blob/main/docker/release/config/opensearch/opensearch-docker-entrypoint.sh#L80-L109
We can probably deprecated these made up variables and just ask people to directly use opensearch settings in ENV.
However, one of the reasons we introduce these new ENV VAR is due to some of them present in ODFE back in the days, so people still want to have that BC.
And especially for Dashboards you need to completely uninstall the security FE plugin and replace all the HTTPS to HTTP in config file, as it assume OpenSearch has security BE plugin installed by default.
Thus have all these kind of issues that we need to change config file on the fly.
I do, however, appreciate any ideas on how to have a way to resolve this without all the live patches to the file.
Will create a PR for temp fix based on https://github.com/opensearch-project/opensearch-build/issues/1529#issuecomment-1021590798.
@jgough @unhipzippo @qmonitoring @deng47 Here is the PR for the permanent fix on OpenSearch, and a temp fix for Dashboards: https://github.com/opensearch-project/opensearch-build/pull/1579
We have staging images here if you willing to try out, and let us know if the issue is fixed for now. We apologize for the inconvinience.
OpenSearch 1.2.4 testfix: docker pull opensearchstaging/opensearch:1.2.4-testfix
Dashboards 1.2.0 testfix: docker pull opensearchstaging/opensearch-dashboards:1.2.0-testfix
Thanks.
@peterzhuamazon I can confirm that I've upgraded a 1.2.3 OpenSearch cluster with the opensearchstaging/opensearch:1.2.4-testfix image successfully and that has fixed the issue we were having. I can't easily test the opensearch-dashboards image.
@peterzhuamazon - I can confirm from my side that none of the versions work for me, when I like to exchange the config (latest, 1.2.3, 1.2.4-testfix and so on). An exchange of the opensearch.yml
configuration is not possible and always results in the error from the initial comment 1529 - #1 (failed to load the Security-Plugin).
The opensearch.yml
on my host has the desired permission (660) and looks the same as the standard config (no changes were made from my side). Host system is an Ubuntu 20.04.3 with Docker Engine 20.10.9.
docker-compose.yml
and custom_opensearch.yml
located in the same directory.
The docker-compose looks like this:
version: '3'
services:
opensearch-node:
# image: opensearchproject/opensearch:latest
image: opensearchstaging/opensearch:1.2.4-testfix
container_name: opensearch-node
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node
- discovery.seed_hosts=opensearch-node
- cluster.initial_master_nodes=opensearch-node
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data:/usr/share/opensearch/data
- ./custom_opensearch.yml:/usr/share/opensearch/config/opensearch.yml
ports:
- 9200:9200
- 9600:9600
networks:
- opensearch-net
volumes:
opensearch-data:
networks:
opensearch-net:
Am I missing something? Thanks in advanced!
@peterzhuamazon - I can confirm from my side that none of the versions work for me, when I like to exchange the config (latest, 1.2.3, 1.2.4-testfix and so on). An exchange of the
opensearch.yml
configuration is not possible and always results in the error from the initial comment 1529 - #1 (failed to load the Security-Plugin).The
opensearch.yml
on my host has the desired permission (660) and looks the same as the standard config (no changes were made from my side). Host system is an Ubuntu 20.04.3 with Docker Engine 20.10.9.docker-compose.yml
andcustom_opensearch.yml
located in the same directory.The docker-compose looks like this:
version: '3' services: opensearch-node: # image: opensearchproject/opensearch:latest image: opensearchstaging/opensearch:1.2.4-testfix container_name: opensearch-node environment: - cluster.name=opensearch-cluster - node.name=opensearch-node - discovery.seed_hosts=opensearch-node - cluster.initial_master_nodes=opensearch-node - bootstrap.memory_lock=true - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 volumes: - opensearch-data:/usr/share/opensearch/data - ./custom_opensearch.yml:/usr/share/opensearch/config/opensearch.yml ports: - 9200:9200 - 9600:9600 networks: - opensearch-net volumes: opensearch-data: networks: opensearch-net:
Am I missing something? Thanks in advanced!
Hi @lerdt do you have any logs showing what is going on with your errors? 1.2.4-testfix will not make changes to opensearch.yml anymore if you dont use any env vars to disable security. In theory it should work out of the box because seems like your dockercompose file is just running the default behavior.
Thanks.
We have officially re-released 1.2.4 OpenSearch and 1.2.0 Dashboards with the above fixes as well as new OS level patches.
docker pull opensearchproject/opensearch:1.2.4
docker pull opensearchproject/opensearch-dashboards:1.2.0
@lerdt I will close this issue for now it has resolved majority of the issues that 1.2.3 can run but 1.2.4 cant. As for your specific issues would you mind open a new issue for tracking, as this issue thread is too broad for your case.
Thanks.
This command works
But this one doesn't
Errors are shown below