search

elastic / cloud-on-k8s

Elastic Cloud on Kubernetes
Other
2.56k stars 695 forks source link

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] #2853

Closed balusarakesh closed 4 years ago

balusarakesh commented 4 years ago

Bug Report

What did you do? Deploy elasticsearch using the operator

What did you expect to see? Working elasticsearch data and master nodes What did you see instead? Under which circumstances? Data nodes not able to start

Environment

1.0.1

I know that running the command sysctl -w vm.max_map_count=262144 will fix it and it actually fixed it, but the value for vm.max_map_count is already set to 262144

[root@elasticsearch-es-data-0 elasticsearch]# cat /proc/sys/vm/max_map_count
262144
sebgl commented 4 years ago

I know that running the command sysctl -w vm.max_map_count=262144 will fix it and it actually fixed it, but the value for vm.max_map_count is already set to 262144

Did you run that command on the host before Elasticsearch starts, or after the Pod has been created?

balusarakesh commented 4 years ago

I know that running the command sysctl -w vm.max_map_count=262144 will fix it and it actually fixed it, but the value for vm.max_map_count is already set to 262144

Did you run that command on the host before Elasticsearch starts, or after the Pod has been created? I ran it before the pod starts using initcontainer

- name: set-max-map-count
command:
- sh
- -c
- sysctl -w vm.max_map_count=262144
anyasabo commented 4 years ago

Can you share your full Elasticsearch manifest so we can try and reproduce?

david-kow commented 4 years ago

Closing due to inactivity - feel free to reopen if the issue is still there.

balusarakesh commented 4 years ago

@anyasabo sorry for the late reply here's the manifest

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  annotations:
    common.k8s.elastic.co/controller-version: 1.0.1
    elasticsearch.k8s.elastic.co/cluster-uuid: xVkiLb4RRnmBhqXCXt4d-A
  creationTimestamp: "2020-04-08T21:40:39Z"
  generation: 17
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 7.6.1
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
    elasticsearch.k8s.elastic.co/statefulset-name: elasticsearch-es-data
    helm.sh/chart: elasticsearch-1.0.1
  name: elasticsearch
  namespace: elastic-system
  resourceVersion: "16413186"
  selfLink: /apis/elasticsearch.k8s.elastic.co/v1/namespaces/elastic-system/elasticsearches/elasticsearch
  uid: 7b8deb0a-eb3a-4db0-b2a2-5115dfdd52e5
spec:
  http:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
    tls:
      certificate: {}
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - config:
      cluster.remote.connect: false
      index.store.type: mmapfs
      node.data: false
      node.ingest: false
      node.master: true
      xpack.monitoring.collection.enabled: true
      xpack.monitoring.exporters:
        my_local:
          index.name.time_format: YYYY-MM
          type: local
      xpack.security.authc.realms:
        native:
          native1:
            order: 1
    count: 3
    name: masters
    podTemplate:
      metadata:
        annotations:
          ad.datadoghq.com/elasticsearch.check_names: '["elastic"]'
          ad.datadoghq.com/elasticsearch.init_configs: '[{}]'
      spec:
        containers:
        - name: elasticsearch
          resources:
            limits:
              cpu: 500m
              memory: 2Gi
            requests:
              cpu: 500m
              memory: 2Gi
        initContainers:
        - command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-s3
            echo "y" | bin/elasticsearch-keystore create
          name: install-plugins
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-masters
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  - config:
      cluster.remote.connect: false
      index.store.type: mmapfs
      indices.recovery.max_bytes_per_sec: 500mb
      node.data: true
      node.ingest: true
      node.master: false
      xpack.monitoring.collection.enabled: true
      xpack.monitoring.exporters:
        my_local:
          index.name.time_format: YYYY-MM
          type: local
      xpack.security.authc.realms:
        native:
          native1:
            order: 1
    count: 2
    name: data
    podTemplate:
      metadata:
        annotations:
          ad.datadoghq.com/elasticsearch.check_names: '["elastic"]'
          ad.datadoghq.com/elasticsearch.init_configs: '[{}]'
      spec:
        containers:
        - env:
          - name: ES_JAVA_OPTS
            value: -Xms1g -Xmx1g
          name: elasticsearch
          resources:
            limits:
              cpu: "2"
              memory: 2Gi
            requests:
              cpu: "2"
              memory: 2Gi
        initContainers:
        - command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-s3
            echo "y" | bin/elasticsearch-keystore create
          name: install-plugins
        - command:
          - sh
          - -c
          - sysctl -w vm.max_map_count=262144
          name: set-max-map-count
          securityContext:
            privileged: true
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 400Gi
  updateStrategy:
    changeBudget: {}
  version: 7.6.1
status:
  availableNodes: 5
  health: green
  phase: Ready
sebgl commented 4 years ago

I cannot reproduce this. Is it still happening @balusarakesh? Could there be something that resets the kernel setting behind your back on the hosts?

I noticed you only specify the init container for data nodes, not for master nodes: is that on purpose?

balusarakesh commented 4 years ago

I cannot reproduce this. Is it still happening @balusarakesh? Could there be something that resets the kernel setting behind your back on the hosts?

I noticed you only specify the init container for data nodes, not for master nodes: is that on purpose?

sorry for the late reply, yes that is on purpose. I'm not aware of any resets happening behind our backs. We are moving to ECK operator 1.1.0 soon, maybe then we won't see this issue. I'll keep you updated

michaelschem commented 2 years ago

Anyone solve this one? Trying this.

michaelschem commented 2 years ago

Yea, the link above worked.

kuisathaverat commented 1 year ago

Virtual memory

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-stack
spec:
  version: 8.4.3
  nodeSets:
  - name: default
    count: 3
    config:
      logger.org.elasticsearch: info
    podTemplate:
      spec:
        initContainers:
          - name: sysctl
            securityContext:
              privileged: true
              runAsUser: 0
            command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
michaelschem commented 1 year ago

@kuisathaverat thanks! Confirmed this worked on my new cluster.

stolario commented 1 year ago

How to set this in docker-compose? this fails https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html