Closed balusarakesh closed 4 years ago
I know that running the command sysctl -w vm.max_map_count=262144 will fix it and it actually fixed it, but the value for vm.max_map_count is already set to 262144
Did you run that command on the host before Elasticsearch starts, or after the Pod has been created?
I know that running the command sysctl -w vm.max_map_count=262144 will fix it and it actually fixed it, but the value for vm.max_map_count is already set to 262144
Did you run that command on the host before Elasticsearch starts, or after the Pod has been created? I ran it before the pod starts using initcontainer
- name: set-max-map-count command: - sh - -c - sysctl -w vm.max_map_count=262144
Can you share your full Elasticsearch manifest so we can try and reproduce?
Closing due to inactivity - feel free to reopen if the issue is still there.
@anyasabo sorry for the late reply here's the manifest
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
annotations:
common.k8s.elastic.co/controller-version: 1.0.1
elasticsearch.k8s.elastic.co/cluster-uuid: xVkiLb4RRnmBhqXCXt4d-A
creationTimestamp: "2020-04-08T21:40:39Z"
generation: 17
labels:
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/version: 7.6.1
common.k8s.elastic.co/type: elasticsearch
elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
elasticsearch.k8s.elastic.co/statefulset-name: elasticsearch-es-data
helm.sh/chart: elasticsearch-1.0.1
name: elasticsearch
namespace: elastic-system
resourceVersion: "16413186"
selfLink: /apis/elasticsearch.k8s.elastic.co/v1/namespaces/elastic-system/elasticsearches/elasticsearch
uid: 7b8deb0a-eb3a-4db0-b2a2-5115dfdd52e5
spec:
http:
service:
metadata:
creationTimestamp: null
spec: {}
tls:
certificate: {}
selfSignedCertificate:
disabled: true
nodeSets:
- config:
cluster.remote.connect: false
index.store.type: mmapfs
node.data: false
node.ingest: false
node.master: true
xpack.monitoring.collection.enabled: true
xpack.monitoring.exporters:
my_local:
index.name.time_format: YYYY-MM
type: local
xpack.security.authc.realms:
native:
native1:
order: 1
count: 3
name: masters
podTemplate:
metadata:
annotations:
ad.datadoghq.com/elasticsearch.check_names: '["elastic"]'
ad.datadoghq.com/elasticsearch.init_configs: '[{}]'
spec:
containers:
- name: elasticsearch
resources:
limits:
cpu: 500m
memory: 2Gi
requests:
cpu: 500m
memory: 2Gi
initContainers:
- command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch repository-s3
echo "y" | bin/elasticsearch-keystore create
name: install-plugins
volumeClaimTemplates:
- metadata:
name: elasticsearch-masters
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- config:
cluster.remote.connect: false
index.store.type: mmapfs
indices.recovery.max_bytes_per_sec: 500mb
node.data: true
node.ingest: true
node.master: false
xpack.monitoring.collection.enabled: true
xpack.monitoring.exporters:
my_local:
index.name.time_format: YYYY-MM
type: local
xpack.security.authc.realms:
native:
native1:
order: 1
count: 2
name: data
podTemplate:
metadata:
annotations:
ad.datadoghq.com/elasticsearch.check_names: '["elastic"]'
ad.datadoghq.com/elasticsearch.init_configs: '[{}]'
spec:
containers:
- env:
- name: ES_JAVA_OPTS
value: -Xms1g -Xmx1g
name: elasticsearch
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: "2"
memory: 2Gi
initContainers:
- command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch repository-s3
echo "y" | bin/elasticsearch-keystore create
name: install-plugins
- command:
- sh
- -c
- sysctl -w vm.max_map_count=262144
name: set-max-map-count
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 400Gi
updateStrategy:
changeBudget: {}
version: 7.6.1
status:
availableNodes: 5
health: green
phase: Ready
I cannot reproduce this. Is it still happening @balusarakesh? Could there be something that resets the kernel setting behind your back on the hosts?
I noticed you only specify the init container for data nodes, not for master nodes: is that on purpose?
I cannot reproduce this. Is it still happening @balusarakesh? Could there be something that resets the kernel setting behind your back on the hosts?
I noticed you only specify the init container for data nodes, not for master nodes: is that on purpose?
sorry for the late reply, yes that is on purpose. I'm not aware of any resets happening behind our backs. We are moving to ECK operator 1.1.0 soon, maybe then we won't see this issue. I'll keep you updated
Anyone solve this one? Trying this.
Yea, the link above worked.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elastic-stack
spec:
version: 8.4.3
nodeSets:
- name: default
count: 3
config:
logger.org.elasticsearch: info
podTemplate:
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
runAsUser: 0
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
@kuisathaverat thanks! Confirmed this worked on my new cluster.
How to set this in docker-compose? this fails https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
Bug Report
What did you do? Deploy elasticsearch using the operator
What did you expect to see? Working elasticsearch data and master nodes What did you see instead? Under which circumstances? Data nodes not able to start
Environment
1.0.1
Kubernetes information: EKS We are using the publicly available AMI
amazon-eks-node-1.15-v20200228 (ami-0582e4c984a1e848a)
Logs:
I know that running the command
sysctl -w vm.max_map_count=262144
will fix it and it actually fixed it, but the value forvm.max_map_count
is already set to262144