melissajenner22
commented
4 years ago I deleted 7.7.1, installed 7.8.1. I got same error.
Error: Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
I use the chart, https://github.com/elastic/helm-charts, version 7.8.1 to install elasticsearch into kubernetes cluster. helm version: 2.16.10 Kubernetes version: 1.16 (EKS)
I use default settings, except changed the number of replicas from 3 to 1, changed minimumMasterNodes from 2 to 1 due to resource limitations.
git diff
--- a/elasticsearch/values.yaml
+++ b/elasticsearch/values.yaml
-replicas: 3
-minimumMasterNodes: 2
+replicas: 1
+minimumMasterNodes: 1
helm install --name elasticsearch ./elasticsearch --namespace elk
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 0/1 Running 0 8m22s
kubectl describe pod elasticsearch-master-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 45s default-scheduler Successfully assigned elk/elasticsearch-master-0 to ip-11-111-1-111.us-west-2.compute.internal
Normal SuccessfulAttachVolume 42s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-71b0adf3-3731-4891-97ef-83f4a192a929"
Normal Pulled 35s kubelet, ip-11-111-1-111.us-west-2.compute.internal Container image "docker.elastic.co/elasticsearch/elasticsearch:7.8.1" already present on machine
Normal Created 35s kubelet, ip-11-111-1-111.us-west-2.compute.internal Created container configure-sysctl
Normal Started 35s kubelet, ip-11-111-1-111.us-west-2.compute.internal Started container configure-sysctl
Normal Pulled 35s kubelet, ip-11-111-1-111.us-west-2.compute.internal Container image "docker.elastic.co/elasticsearch/elasticsearch:7.8.1" already present on machine
Normal Created 35s kubelet, ip-11-111-1-111.us-west-2.compute.internal Created container elasticsearch
Normal Started 34s kubelet, ip-11-111-1-111.us-west-2.compute.internal Started container elasticsearch
Warning Unhealthy 5s (x2 over 17s) kubelet, ip-11-111-1-111.us-west-2.compute.internal Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )
vikas4cloud
jaykishan007
iantheparker
nneul
adinhodovic
mykhailosmolynets
botelastic[bot]
obenziane2
sidhuko
byronmansfield
Nimesh36
IsaackRasmussen
vitalyrychkov
jmlrt
chance2021
dev852old
Chart version: 7.7.1 Kubernetes version: 1.16 Kubernetes provider: E.g. GKE (Google Kubernetes Engine) EKS Helm Version: 2.16.10
helm get releaseoutpute.g.
helm get elasticsearch(replaceelasticsearchwith the name of your helm release)Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting.
If you find some secrets in plain text in
helm get releaseoutput you should use Kubernetes Secrets to managed them is a secure way (see Security Example).Output of helm get release
``` $ helm get elasticsearch REVISION: 1 RELEASED: Fri Aug 14 14:15:21 2020 CHART: elasticsearch-7.7.1 USER-SUPPLIED VALUES: {} COMPUTED VALUES: antiAffinity: hard antiAffinityTopologyKey: kubernetes.io/hostname clusterHealthCheckParams: wait_for_status=green&timeout=1s clusterName: elasticsearch envFrom: [] esConfig: {} esJavaOpts: -Xmx1g -Xms1g esMajorVersion: "" extraContainers: [] extraEnvs: [] extraInitContainers: [] extraVolumeMounts: [] extraVolumes: [] fsGroup: "" fullnameOverride: "" httpPort: 9200 image: docker.elastic.co/elasticsearch/elasticsearch imagePullPolicy: IfNotPresent imagePullSecrets: [] imageTag: 7.7.1 ingress: annotations: {} enabled: false hosts: - chart-example.local path: / tls: [] initResources: {} keystore: [] labels: {} lifecycle: {} masterService: "" masterTerminationFix: false maxUnavailable: 1 minimumMasterNodes: 1 nameOverride: "" networkHost: 0.0.0.0 nodeAffinity: {} nodeGroup: master nodeSelector: {} persistence: annotations: {} enabled: true podAnnotations: {} podManagementPolicy: Parallel podSecurityContext: fsGroup: 1000 runAsUser: 1000 podSecurityPolicy: create: false name: "" spec: fsGroup: rule: RunAsAny privileged: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - secret - configMap - persistentVolumeClaim priorityClassName: "" protocol: http rbac: create: false serviceAccountName: "" readinessProbe: failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 3 timeoutSeconds: 5 replicas: 1 resources: limits: cpu: 1000m memory: 2Gi requests: cpu: 1000m memory: 2Gi roles: data: "true" ingest: "true" master: "true" schedulerName: "" secretMounts: [] securityContext: capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 service: annotations: {} httpPortName: http labels: {} labelsHeadless: {} loadBalancerIP: "" loadBalancerSourceRanges: [] nodePort: "" transportPortName: transport type: ClusterIP sidecarResources: {} sysctlInitContainer: enabled: true sysctlVmMaxMapCount: 262144 terminationGracePeriod: 120 tolerations: [] transportPort: 9300 updateStrategy: RollingUpdate volumeClaimTemplate: accessModes: - ReadWriteOnce resources: requests: storage: 30Gi HOOKS: --- # elasticsearch-lgztd-test apiVersion: v1 kind: Pod metadata: name: "elasticsearch-lgztd-test" annotations: "helm.sh/hook": test-success spec: securityContext: fsGroup: 1000 runAsUser: 1000 containers: - name: "elasticsearch-ctvif-test" image: "docker.elastic.co/elasticsearch/elasticsearch:7.7.1" command: - "sh" - "-c" - | #!/usr/bin/env bash -e curl -XGET --fail 'elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s' restartPolicy: Never MANIFEST: --- # Source: elasticsearch/templates/poddisruptionbudget.yaml apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: "elasticsearch-master-pdb" spec: maxUnavailable: 1 selector: matchLabels: app: "elasticsearch-master" --- # Source: elasticsearch/templates/service.yaml kind: Service apiVersion: v1 metadata: name: elasticsearch-master labels: heritage: "Tiller" release: "elasticsearch" chart: "elasticsearch" app: "elasticsearch-master" annotations: {} spec: type: ClusterIP selector: heritage: "Tiller" release: "elasticsearch" chart: "elasticsearch" app: "elasticsearch-master" ports: - name: http protocol: TCP port: 9200 - name: transport protocol: TCP port: 9300 --- # Source: elasticsearch/templates/service.yaml kind: Service apiVersion: v1 metadata: name: elasticsearch-master-headless labels: heritage: "Tiller" release: "elasticsearch" chart: "elasticsearch" app: "elasticsearch-master" annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" spec: clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve # Create endpoints also if the related pod isn't ready publishNotReadyAddresses: true selector: app: "elasticsearch-master" ports: - name: http port: 9200 - name: transport port: 9300 --- # Source: elasticsearch/templates/statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: elasticsearch-master labels: heritage: "Tiller" release: "elasticsearch" chart: "elasticsearch" app: "elasticsearch-master" annotations: esMajorVersion: "7" spec: serviceName: elasticsearch-master-headless selector: matchLabels: app: "elasticsearch-master" replicas: 1 podManagementPolicy: Parallel updateStrategy: type: RollingUpdate volumeClaimTemplates: - metadata: name: elasticsearch-master spec: accessModes: - ReadWriteOnce resources: requests: storage: 30Gi template: metadata: name: "elasticsearch-master" labels: heritage: "Tiller" release: "elasticsearch" chart: "elasticsearch" app: "elasticsearch-master" annotations: spec: securityContext: fsGroup: 1000 runAsUser: 1000 affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - "elasticsearch-master" topologyKey: kubernetes.io/hostname terminationGracePeriodSeconds: 120 volumes: initContainers: - name: configure-sysctl securityContext: runAsUser: 0 privileged: true image: "docker.elastic.co/elasticsearch/elasticsearch:7.7.1" imagePullPolicy: "IfNotPresent" command: ["sysctl", "-w", "vm.max_map_count=262144"] resources: {} containers: - name: "elasticsearch" securityContext: capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000 image: "docker.elastic.co/elasticsearch/elasticsearch:7.7.1" imagePullPolicy: "IfNotPresent" readinessProbe: exec: command: - sh - -c - | #!/usr/bin/env bash -e # If the node is starting up wait for the cluster to be ready (request params: "wait_for_status=green&timeout=1s" ) # Once it has started only check that the node itself is responding START_FILE=/tmp/.es_start_file http () { local path="${1}" local args="${2}" set -- -XGET -s if [ "$args" != "" ]; then set -- "$@" $args fi if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}" fi curl --output /dev/null -k "$@" "http://127.0.0.1:9200${path}" } if [ -f "${START_FILE}" ]; then echo 'Elasticsearch is already running, lets check the node is healthy' HTTP_CODE=$(http "/" "-w %{http_code}") RC=$? if [[ ${RC} -ne 0 ]]; then echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with RC ${RC}" exit ${RC} fi # ready if HTTP code 200, 503 is tolerable if ES version is 6.x if [[ ${HTTP_CODE} == "200" ]]; then exit 0 elif [[ ${HTTP_CODE} == "503" && "7" == "6" ]]; then exit 0 else echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with HTTP code ${HTTP_CODE}" exit 1 fi else echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )' if http "/_cluster/health?wait_for_status=green&timeout=1s" "--fail" ; then touch ${START_FILE} exit 0 else echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )' exit 1 fi fi failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 3 timeoutSeconds: 5 ports: - name: http containerPort: 9200 - name: transport containerPort: 9300 resources: limits: cpu: 1000m memory: 2Gi requests: cpu: 1000m memory: 2Gi env: - name: node.name valueFrom: fieldRef: fieldPath: metadata.name - name: cluster.initial_master_nodes value: "elasticsearch-master-0," - name: discovery.seed_hosts value: "elasticsearch-master-headless" - name: cluster.name value: "elasticsearch" - name: network.host value: "0.0.0.0" - name: ES_JAVA_OPTS value: "-Xmx1g -Xms1g" - name: node.data value: "true" - name: node.ingest value: "true" - name: node.master value: "true" volumeMounts: - name: "elasticsearch-master" mountPath: /usr/share/elasticsearch/data ```Describe the bug: