search

opensearch-project / helm-charts

:wheel_of_dharma: A community repository for Helm Charts of OpenSearch Project.
https://opensearch.org/docs/latest/opensearch/install/helm/
Apache License 2.0
172 stars 234 forks source link

[BUG][Opensearch does not start with default helm configuration.] #233

Closed JWebDev closed 2 years ago

JWebDev commented 2 years ago

Description helm install opensearch opensearch/opensearch -n opensearch -f defaultvalues.yaml I do as the instructions say. I tried both with the default tag and with 1.2.3 and 1.2.4

I don't want https so I disabled it.

defaultvalues.yaml

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "true"

image:
  repository: "opensearchproject/opensearch"
  # override image tag, which is .Chart.AppVersion by default
  tag: "1.2.3"
  pullPolicy: "IfNotPresent"

replicas: 1
minimumMasterNodes: 1

# if not set, falls back to parsing .Values.imageTag, then .Chart.appVersion.
majorVersion: "1"

config:
  opensearch.yml: |
    cluster.name: opensearch-cluster
    network.host: 0.0.0.0
    discovery.type: single-node
    plugins:
      security:
        ssl:
          transport: {}
          http:
            enabled: false
        allow_unsafe_democertificates: false
        allow_default_init_securityindex: true
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]

Error:

Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
Enabling OpenSearch Security Plugin
sed: cannot rename /usr/share/opensearch/config/sedhHAx7v: Device or resource busy
[2022-03-17T12:06:41,651][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[1.2.3], pid[11], build[tar/8a529d77c7432bc45b005ac1c4ba3b2741b57d4a/2021-12-21T01:36:21.407473Z], OS[Linux/3.10.0-1160.49.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-03-17T12:06:41,652][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-03-17T12:06:41,652][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-18367318079747187011, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-03-17T12:06:42,278][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2022-03-17T12:06:42,278][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2022-03-17T12:06:42,421][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] JVM supports TLSv1.3
[2022-03-17T12:06:42,422][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
uncaught exception in thread [main]
[2022-03-17T12:06:42,455][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-1.2.3.jar:1.2.3]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-1.2.3.jar:1.2.3]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.3.jar:1.2.3]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.3.jar:1.2.3]
... 6 more
Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.3.jar:1.2.3]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.3.jar:1.2.3]
... 6 more
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258)
at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179)
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726)
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528)
at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194)
at org.opensearch.node.Node.<init>(Node.java:396)
at org.opensearch.node.Node.<init>(Node.java:319)
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
at org.opensearch.cli.Command.main(Command.java:101)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log
Killing performance analyzer process 12
OpenSearch exited with code 1
Performance analyzer exited with code 143

If i starts default installation without changes, the same errors comes: helm install opensearch opensearch/opensearch -n opensearch

TheAlgo commented 2 years ago

What is the k8s setup you are using?

JWebDev commented 2 years ago 
kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:44:26Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Kubernetes: v1.21.5

helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}
JWebDev commented 2 years ago

If i completly disable security plugin then it starts. plugins.security.disabled=true

But I want to disable ssl only. And leave the possibility of login. In docker, I can start opensearch with these parameters without any problems. 

    environment:
      - cluster.name=opensearch-cluster
      - discovery.type=single-node
      - plugins.security.ssl.http.enabled=false
      - node.name=opensearch
      - bootstrap.memory_lock=true
TheAlgo commented 2 years ago 
kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:44:26Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Kubernetes: v1.21.5

helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}

I meant the cluster configurations.🙂 Sorry if I was not clear

TheAlgo commented 2 years ago

If i completly disable security plugin then it starts. plugins.security.disabled=true

But I want to disable ssl only. And leave the possibility of login. In docker, I can start opensearch with these parameters without any problems.

    environment:
      - cluster.name=opensearch-cluster
      - discovery.type=single-node
      - plugins.security.ssl.http.enabled=false
      - node.name=opensearch
      - bootstrap.memory_lock=true

The yaml you shared is not the same as this configurations right?

JWebDev commented 2 years ago
  • cluster.name=opensearch-cluster
    • discovery.type=single-node
    • plugins.security.ssl.http.enabled=false
    • node.name=opensearch
    • bootstrap.memory_lock=true

Thanks for the answer. Nope. The same error.

Just try this config with default settings: helm install opensearch opensearch/opensearch -n opensearch -f defaultvalues.yaml

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "true"
  - name: discovery.type
    value: single-node
  - name: cluster.name
    value: opensearch-cluster
  - name: plugins.security.ssl.http.enabled
    value: "false"
  - name: node.name
    value: opensearch
  - name: bootstrap.memory_lock
    value: "true"

replicas: 1
minimumMasterNodes: 1

majorVersion: "1"

config: 
  opensearch.yml: |
    cluster.name: opensearch-cluster
    network.host: 0.0.0.0
    discovery.type: single-node
JWebDev commented 2 years ago 
kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:44:26Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Kubernetes: v1.21.5

helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}

I meant the cluster configurations.🙂 Sorry if I was not clear

It is not cluster config problem. It is security-plugin configurations problem. As I sad, if I completly disable security plugin then it starts. plugins.security.disabled=true. But I want to disable ssl only and keep authentication.

TheAlgo commented 2 years ago

Looks like you are adding it in extraEnvs, any reason for it? Why not in config itself we have a placeholder for SSL plugin already present in the default values.yaml

# Start OpenSearch Security Demo Configuration
    # WARNING: revise all the lines below before you go into production
    plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
            enforce_hostname_verification: false
          http:
            enabled: false
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
        allow_unsafe_democertificates: true
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=kirk,OU=client,O=client,L=test,C=de
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true

Link : https://github.com/opensearch-project/helm-charts/blob/main/charts/opensearch/values.yaml#L68

JWebDev commented 2 years ago

@TheAlgo I tried with envs and without

image:
  repository: "opensearchproject/opensearch"
  tag: "1.3.0"
  pullPolicy: "IfNotPresent"
opensearchJavaOpts: "-Xmx5000M -Xms5000M"
resources:
  requests:
    cpu: "1000m"
    memory: "500Mi"

config:
  opensearch.yml: |
    plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
            enforce_hostname_verification: false
          http:
            enabled: false
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
        allow_unsafe_democertificates: true
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=kirk,OU=client,O=client,L=test,C=de
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true

The same error.

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
--
Sat, Mar 19 2022 20:14:06 | OpenSearch Security Demo Installer
Sat, Mar 19 2022 20:14:06 | ** Warning: Do not use on production or public reachable systems **
Sat, Mar 19 2022 20:14:06 | Basedir: /usr/share/opensearch
Sat, Mar 19 2022 20:14:06 | OpenSearch install type: rpm/deb on NAME="Amazon Linux"
Sat, Mar 19 2022 20:14:06 | OpenSearch config dir: /usr/share/opensearch/config
Sat, Mar 19 2022 20:14:06 | OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
Sat, Mar 19 2022 20:14:06 | OpenSearch bin dir: /usr/share/opensearch/bin
Sat, Mar 19 2022 20:14:06 | OpenSearch plugins dir: /usr/share/opensearch/plugins
Sat, Mar 19 2022 20:14:06 | OpenSearch lib dir: /usr/share/opensearch/lib
Sat, Mar 19 2022 20:14:06 | Detected OpenSearch Version: x-content-1.3.0
Sat, Mar 19 2022 20:14:06 | Detected OpenSearch Security Version: 1.3.0.0
Sat, Mar 19 2022 20:14:06 |  
Sat, Mar 19 2022 20:14:06 | tee: /usr/share/opensearch/config/opensearch.yml: Permission denied
Sat, Mar 19 2022 20:14:06 | Enabling OpenSearch Security Plugin
Sat, Mar 19 2022 20:14:07 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sat, Mar 19 2022 20:14:08 | [2022-03-19T19:14:08,823][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[1.3.0], pid[45], build[tar/e45991597c86ba1bbcc36ee1dfdc165197a913af/2022-03-15T19:07:30.455415Z], OS[Linux/3.10.0-1160.49.1.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.14.1/11.0.14.1+1]
Sat, Mar 19 2022 20:14:08 | [2022-03-19T19:14:08,825][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
Sat, Mar 19 2022 20:14:08 | [2022-03-19T19:14:08,825][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/opensearch-8731229463541717511, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -Dopensearch.cgroups.hierarchy.override=/, -Xmx5000M, -Xms5000M, -XX:MaxDirectMemorySize=2621440000, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
Sat, Mar 19 2022 20:14:09 | [2022-03-19T19:14:09,424][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Sat, Mar 19 2022 20:14:09 | [2022-03-19T19:14:09,424][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
Sat, Mar 19 2022 20:14:09 | [2022-03-19T19:14:09,424][WARN ][stderr ] [opensearch-cluster-master-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,143][INFO ][o.o.p.c.PluginSettings ] [opensearch-cluster-master-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,342][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-cluster-master-0] ReindexPlugin reloadSPI called
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,342][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-cluster-master-0] Unable to find any implementation for RemoteReindexExtension
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,354][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,371][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,372][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,374][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [aggs-matrix-stats]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,374][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [analysis-common]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [geo]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-common]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-geoip]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-user-agent]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-expression]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-mustache]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-painless]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [mapper-extras]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [opensearch-dashboards]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [parent-join]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [percolator]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [rank-eval]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [reindex]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [repository-url]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [transport-netty4]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-alerting]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,375][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-anomaly-detection]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-asynchronous-search]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-cross-cluster-replication]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-index-management]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-job-scheduler]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-knn]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-ml]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-observability]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-performance-analyzer]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-reports-scheduler]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-security]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,376][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-sql]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,396][INFO ][o.o.e.NodeEnvironment ] [opensearch-cluster-master-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/longhorn/pvc-0be5a9c7-3e3d-4b28-9728-380022ad5913)]], net usable_space [7.7gb], net total_space [7.8gb], types [ext4]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,397][INFO ][o.o.e.NodeEnvironment ] [opensearch-cluster-master-0] heap size [4.8gb], compressed ordinary object pointers [true]
Sat, Mar 19 2022 20:14:10 | [2022-03-19T19:14:10,445][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] node name [opensearch-cluster-master-0], node ID [YqAuJ5bWTrCLf5Bg60Zljg], cluster name [opensearch-cluster], roles [master, remote_cluster_client, data, ingest]
Sat, Mar 19 2022 20:14:12 | [2022-03-19T19:14:12,387][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-master-0] Registered memory breaker.
Sat, Mar 19 2022 20:14:12 | [2022-03-19T19:14:12,585][INFO ][o.o.m.c.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML memory breaker.
Sat, Mar 19 2022 20:14:12 | [2022-03-19T19:14:12,849][INFO ][o.o.t.NettyAllocator ] [opensearch-cluster-master-0] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
Sat, Mar 19 2022 20:14:12 | [2022-03-19T19:14:12,892][INFO ][o.o.d.DiscoveryModule ] [opensearch-cluster-master-0] using discovery type [zen] and seed hosts providers [settings]
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,142][WARN ][o.o.g.DanglingIndicesState] [opensearch-cluster-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,402][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-master-0] PerformanceAnalyzer Enabled: false
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,418][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] initialized
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,418][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] starting ...
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,502][INFO ][o.o.t.TransportService ] [opensearch-cluster-master-0] publish_address {10.42.1.43:9300}, bound_addresses {0.0.0.0:9300}
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,632][INFO ][o.o.b.BootstrapChecks ] [opensearch-cluster-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
Sat, Mar 19 2022 20:14:13 | ERROR: [1] bootstrap checks failed
Sat, Mar 19 2022 20:14:13 | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Sat, Mar 19 2022 20:14:13 | ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/opensearch-cluster.log
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,638][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] stopping ...
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,658][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] stopped
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,658][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] closing ...
Sat, Mar 19 2022 20:14:13 | [2022-03-19T19:14:13,668][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] closed
Sat, Mar 19 2022 20:14:14 | Killing performance analyzer process 46
Sat, Mar 19 2022 20:14:14 | OpenSearch exited with code 78
Sat, Mar 19 2022 20:14:14 | Performance analyzer exited with code 143
TheAlgo commented 2 years ago

Closing in favour of #235