After creating the objects with opensearch-cluster.yaml the PODS don't go in running

[fdgDev]

Hi, I'm trying for the first time OpenSearch's installation on Kubernetes. i have followed all your instractions but after creating the objects with opensearch-cluster.yaml example file some PODS don't go in running mode, there is a startup failed " dial tcp 10.36.0.5:9200: connect: connection refused". This is PODs's status now:

AMESPACE NAME READY STATUS RESTARTS AGE default nginx-deployment-85996f8dbd-kh4dh 1/1 Running 0 15h default nginx-deployment-85996f8dbd-sf8z8 1/1 Running 0 15h kube-system coredns-787d4945fb-6rtqq 1/1 Running 36 (41h ago) 6d kube-system coredns-787d4945fb-kdlcb 1/1 Running 35 (41h ago) 6d kube-system etcd-dbl-pdt-as-l011 1/1 Running 315 (41h ago) 6d kube-system kube-apiserver-dbl-pdt-as-l011 1/1 Running 306 (41h ago) 6d kube-system kube-controller-manager-dbl-pdt-as-l011 1/1 Running 118 (41h ago) 6d kube-system kube-proxy-gwkdw 1/1 Running 6 5d18h kube-system kube-proxy-kd8n8 1/1 Running 0 15h kube-system kube-proxy-sn4zj 1/1 Running 0 3d kube-system kube-proxy-zqkds 1/1 Running 49 (41h ago) 5d21h kube-system kube-scheduler-dbl-pdt-as-l011 1/1 Running 336 (41h ago) 6d kube-system weave-net-5ck5l 2/2 Running 0 15h kube-system weave-net-rvl2d 2/2 Running 0 40m kube-system weave-net-tnvtq 2/2 Running 0 15h kube-system weave-net-zjxgs 2/2 Running 0 17h kubernetes-dashboard dashboard-metrics-scraper-7bc864c59-gvr2k 1/1 Running 0 5d17h kubernetes-dashboard kubernetes-dashboard-6c7ccbcf87-klbth 1/1 Running 0 5d17h local-path-storage local-path-provisioner-8bc8875b-vh775 1/1 Running 0 21h opensearch opensearch-operator-controller-manager-8568955f5c-9gk4n 2/2 Running 0 86m opensearch plus-cluster-bootstrap-0 0/1 CrashLoopBackOff 13 (2m31s ago) 44m opensearch plus-cluster-coordinators-0 0/1 Running 11 (5m59s ago) 44m opensearch plus-cluster-dashboards-6d7cc7f45-czpxk 0/1 CrashLoopBackOff 9 (2m54s ago) 44m opensearch plus-cluster-dashboards-6d7cc7f45-rgjmk 0/1 CrashLoopBackOff 9 (2m24s ago) 44m opensearch plus-cluster-masters-0 0/1 Running 12 (116s ago) 44m opensearch plus-cluster-nodes-0 0/1 CrashLoopBackOff 12 (32s ago) 44m

My kubernets version is: Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

Can you help me understand where is the error? Thanks

[fdgDev]

in POD's log I read this: no preferred addresses found; known addresses: []

[fdgDev]

logs cluster-coordinators:

-> Installing repository-s3 -> Downloading repository-s3 from opensearch @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

• java.lang.RuntimePermission accessDeclaredMembers
• java.lang.RuntimePermission getClassLoader
• java.lang.reflect.ReflectPermission suppressAccessChecks
• java.net.NetPermission setDefaultAuthenticator
• java.net.SocketPermission * connect,resolve
• java.util.PropertyPermission opensearch.allow_insecure_settings read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed repository-s3 with folder name repository-s3 -> Installing https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/1.3.0.0/prometheus-exporter-1.3.0.0.zip -> Downloading https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/1.3.0.0/prometheus-exporter-1.3.0.0.zip @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
• java.lang.RuntimePermission accessClassInPackage.sun.misc
• java.lang.RuntimePermission accessDeclaredMembers
• java.lang.reflect.ReflectPermission suppressAccessChecks See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed prometheus-exporter with folder name prometheus-exporter Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin OpenSearch Security Demo Installer Warning: Do not use on production or public reachable systems Basedir: /usr/share/opensearch OpenSearch install type: rpm/deb on NAME="Amazon Linux" OpenSearch config dir: /usr/share/opensearch/config OpenSearch config file: /usr/share/opensearch/config/opensearch.yml OpenSearch bin dir: /usr/share/opensearch/bin OpenSearch plugins dir: /usr/share/opensearch/plugins OpenSearch lib dir: /usr/share/opensearch/lib Detected OpenSearch Version: x-content-1.3.0 Detected OpenSearch Security Version: 1.3.0.0

  Success

  Execute this script now on all your nodes and then start all nodes

  OpenSearch Security will be automatically initialized.

  If you like to change the runtime configuration

  change the files in ../securityconfig and execute:

  "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv

  or run ./securityadmin_demo.sh

  To use the Security Plugin ConfigurationGUI

  To access your secured cluster open https://: and log in with admin/admin.

  (Ignore the SSL certificate warning because we installed self-signed demo certificates)

  Enabling OpenSearch Security Plugin

I have installed kubernetes on premise. Must I reinstall prometheus-exporter manualy ? Thanks

[idanl21]

Hey @fdgDev So your cluster is up but you cannot see any metrics on you Prometheus?

[idanl21]

@fdgDev Hey mate, do you need any help here ?

[swoehrl-mw]

Closing as there is no response from reporter.