Unable to retrieve version information from Elasticsearch nodes. socket hang up

[grigoryevandrey]

I am getting this error inside the kibana container, therefore ingress returns 503 error and container is never ready.

When i am doing curl to elasticsearch from inside the kibana container, it successfully returns a response.

So either something is wrong with documentation or with chart itself.

Did anyone figure out how to fix it?

Chart version: 7.17.3

elasticsearch-values.yaml

``` clusterName: "elasticsearch" nodeGroup: "master" createCert: false roles: master: "true" data: "true" ingest: "true" ml: "true" transform: "true" remote_cluster_client: "true" protocol: https replicas: 2 imageTag: "7.17.3" sysctlVmMaxMapCount: 262144 readinessProbe: failureThreshold: 3 initialDelaySeconds: 90 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 extraEnvs: - name: ELASTIC_PASSWORD valueFrom: secretKeyRef: name: elasticsearch-creds key: password - name: ELASTIC_USERNAME valueFrom: secretKeyRef: name: elasticsearch-creds key: username clusterHealthCheckParams: "wait_for_status=green&timeout=20s" antiAffinity: "soft" resources: requests: cpu: "100m" memory: "1Gi" limits: cpu: "1000m" memory: "1Gi" esJavaOpts: "-Xms512m -Xmx512m" volumeClaimTemplate: accessModes: ["ReadWriteOnce"] resources: requests: storage: 30Gi esConfig: elasticsearch.yml: | xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12 secretMounts: - name: elastic-certificates secretName: elastic-certificates path: /usr/share/elasticsearch/config/certs ```

kibana-values.yaml

``` elasticSearchHosts: "https://elasticsearch-master:9200" imageTag: "7.17.3" extraEnvs: - name: ELASTICSEARCH_USERNAME valueFrom: secretKeyRef: name: elasticsearch-creds key: username - name: ELASTICSEARCH_PASSWORD valueFrom: secretKeyRef: name: elasticsearch-creds key: password - name: KIBANA_ENCRYPTION_KEY valueFrom: secretKeyRef: name: encryption-key key: encryption_key kibanaConfig: kibana.yml: | server.ssl: enabled: true key: /usr/share/kibana/config/certs/elastic-certificate.pem certificate: /usr/share/kibana/config/certs/elastic-certificate.pem xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY} elasticsearch.ssl: certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem verificationMode: certificate protocol: https secretMounts: - name: elastic-certificate-pem secretName: elastic-certificate-pem path: /usr/share/kibana/config/certs ingress: enabled: true ingressClassName: nginx annotations: cert-manager.io/cluster-issuer: letsencrypt-issuer kubernetes.io/ingress.allow-http: 'false' paths: - path: / pathType: Prefix backend: serviceName: kibana servicePort: 5601 hosts: - host: mydomain.com paths: - path: / pathType: Prefix backend: serviceName: kibana servicePort: 5601 tls: - hosts: - mydomain.com secretName: mydomain.com ```

[grigoryevandrey]

UPD: tried it with other image version (8.4.1), nothing has changed, i am getting the same error. By the way, logstash is successfully shipping logs to this elasticsearch instance, so i think problem is in kibana.

[jmlrt]

👋🏻 Elasticsearch cluster requires an odd number of nodes. I see that you have defined only 2 replicas in your values.

Some manual requests may work but your cluster will never be in green state, and I think that's why Kibana can't connect to it as it's waiting that the cluster is in green state.

Can you retry with 3 replicas for Elasticsearch?

[grigoryevandrey]

Figured it out. It was a complete pain. I hope these tips will help others:

1. xpack.security.http.ssl.enabled should be set to false. I can't find another way around it, but if you do i'd be glad to hear any advices. As i see it, you don't need security for http layer since kibana connects to elastic via transport layer (correct me if i am wrong). Therefore xpack.security.transport.ssl.enabled should be still set to true, but xpack.security.http.ssl.enabled should be set to false. (don't forget to change your protocol field for readinessProbe to http, and also change protocol for elasticsearch in kibana chart to http.
2. ELASTIC_USERNAME env variable is pointless in elasticsearch chart, only password is used, user is always elastic
3. ELASTICSEARCH_USERNAME in kibana chart should be actually set to kibana_systems user with according password for that user

[grigoryevandrey]

👋🏻 Elasticsearch cluster requires an odd number of nodes. I see that you have defined only 2 replicas in your values.

Some manual requests may work but your cluster will never be in green state, and I think that's why Kibana can't connect to it as it's waiting that the cluster is in green state.

It was green all the time, so this is not the case

Can you retry with 3 replicas for Elasticsearch?

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.