[BUG][OpenSearch Dashboard]-CrashLoopBackOff Error

[daniel-dodoo]

CrashLoopBackOff Error I am deploying an OpenSearch cluster on my local machine, after successfully installing the pods with the helm command

I checked the status of my pods with kubectl get pods command but it returns a "CrashLoopBackOff" error

I found a solution on StackOverflow that requires increasing the initialDelaySeconds to 300s but that doesn't seem to work

To Reproduce Steps to reproduce the behavior:

1. Go to kubectl get pods
2. View the pods status of opensearch-cluster-client-0, opensearch-cluster-data-0, opensearch-cluster-master-0

Expected behavior The output shows READY 1/1 and status Running once the deployment succeeds.

Screenshots

Host/Environment (please complete the following information):

• Helm Version: "v3.9.1"
• Kubernetes Client Version: "v3.9.1"
• Kubernetes Server Version: "v1.24.1
• Local Machine: OS Name: Microsoft Windows 10 Home Version: 10.0.19044 Build 19044 System Model: HP ENVY x360 Convertible Processor: Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz, 2592 Mhz, 2 Core(s), 4 Logical Processor(s) BIOS Version/Date: Insyde F.1A, 7/25/2016

[TheAlgo]

@daniel-dodoo Can you share the values.yaml file you are using, also share the helm chart version you are using for spinning up the multi-cluster nodes. Thanks

[daniel-dodoo]

@TheAlgo Sure sure

helm version: "v3.9.1"

I tried to upload the .yaml files but github doesn't support the extension.

I also tried to copy in directly to the text, the formatting is difficult to read so I uploaded the .txt copies of the script

client_script.txt data_script.txt master_script.txt

[TheAlgo]

@daniel-dodoo I meant the chart version and not the helm version so that I can try reproducing the issue. Also you can check the logs of the pods using the command kubectl logs command, this will help us why the container is crashing as I can see the container was started and it crashed post that.

[daniel-dodoo]

@TheAlgo ,

Sorry for the delayed response

Helm Chart Version: 2.3.0

Kindly find the logs the pods attached: opensearch-cluster-client-0 logs.txt opensearch-cluster-master-0 logs.txt opensearch-cluster-data-0 logs.txt

[TheAlgo]

Looks like this is the error

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/opensearch-cluster.log

Also looks like you are following the blog and it is using chart versions for OpenSearch is 1.2.4 and 1.0.6 for OpenSearch Dashboards.

Can you add the following in all YAML(s) and retry?

sysctl:
  enabled: true

[daniel-dodoo]

Hello @TheAlgo

How do I view the Helm chart version for OpenSearch Dashboard?

I followed your instructions and made changes to the yaml files for master, data and client

I have the following error now: forbidden sysctl: "vm.max_map_count" not allowlisted

[

[TheAlgo]

You can follow this issue to fix it.

[daniel-dodoo]

Hello @TheAlgo ,

Many thanks

The pods are now in the running state, I have 2 errors which I have tried to debug to no avail

1. OpenSearch Security not Initialised when I run "curl -XGET https://localhost:9200 -u 'admin:admin' --insecure" command

2. Startup probe failed: dial tcp 172.17.0.3:9200: connect: connection refused when I describe master, data and client pods

[TheAlgo]

@daniel-dodoo Can you share the pod logs so that we can see what is the issue you are getting.

P:S : Alternatively just wanted to share if you are not aware of : if you don't want a dedicated master, data, client structure you can follow the simpler way of installation here as well.

[daniel-dodoo]

@TheAlgo

Kindly find the logs for master, data and client attached

Many thanks, I will definitely review the simpler documentation but I will first like to troubleshoot my errors in the current deployment

master.log data.log data.log

[TheAlgo]

Seeing this error log some permission issues in the cluster which needs to be fixed it seems You can use official documentation as well to fix security plugin issues in the cluster:

[2022-08-01T23:19:49,335][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-08-01T23:19:49,335][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-08-01T23:19:50,222][INFO ][o.o.p.c.PluginSettings   ] [opensearch-cluster-data-0] Config: metric

Also do you mind installing charts with the version mentioned in the blog? You can use the flag --version 6.7.4

OpenSearch is 1.2.4 and 1.0.6 for OpenSearch Dashboards

[daniel-dodoo]

@TheAlgo

"Also do you mind installing charts with the version mentioned in the blog? You can use the flag --version 6.7.4"

How do I install --version 6.7.4?

When I tried to specify the version in the helm install command but I have an error response

Eg. helm install opensearch-master opensearch/opensearch -f master.yaml --version 6.7.4

Error: INSTALLATION FAILED: failed to download "opensearch/opensearch" at version "6.7.4"

[daniel-dodoo]

@TheAlgo

After I run the helm repo update and helm search repo opensearch

The latest OpenSearch chart version was opensearch/opensearch: 2.3.0 and opensearch/opensearch-dashboards: 2.2.4

[daniel-dodoo]

@TheAlgo I run into an error when running the securityadmin.sh from the Documentation

./securityadmin.sh -cd ../../../config/opensearch-security/ -icl -nhnv \ -cacert ../../../config/root-ca.pem \ -cert ../../../config/kirk.pem \ -key ../../../config/kirk-key.pem

I receive the following error response :

 * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Kindly find the logs attached: securityadmin_diag_trace

[TheAlgo]

@TheAlgo

"Also do you mind installing charts with the version mentioned in the blog? You can use the flag --version 6.7.4"

How do I install --version 6.7.4?

When I tried to specify the version in the helm install command but I have an error response

Eg. helm install opensearch-master opensearch/opensearch -f master.yaml --version 6.7.4

Error: INSTALLATION FAILED: failed to download "opensearch/opensearch" at version "6.7.4"

@daniel-dodoo Sorry if I was not clear but I meant to install the version mentioned in the blog and it was just an example I gave. 1.2.4 is the version for OpenSearch and 1.0.6 is the version for Dashboards.