search

pires / kubernetes-elasticsearch-cluster

Elasticsearch cluster on top of Kubernetes made easy.
Apache License 2.0
1.51k stars 690 forks source link

Out of memory issue for master and client nodes #202

Open mootezbessifi opened 6 years ago

mootezbessifi commented 6 years ago

Hi, i am running a ES cluster on top of k8s (2 es-client, 3 es-master and 3 es-data). the cluster is used for efk stack. i configured before the heap size for each one as following:

the cluster was working fine until today when i tried to change the storage with Glusterfs for the es-data and restart all ES deployments.

es-masters and es-clients refuse to run and generate exception OutOfMemory exception. i was going for es-client beyond 5GB and i still got the same issue

i need help please

rewt commented 6 years ago

Try increasing es-data nodes to at least 4 GB.

mootezbessifi commented 6 years ago

@rewt thank you very much for your response when i tried to fo this, all es-data pods generated Out of memory issue. i want to know i the case of huge traffic from fluentd, pushing logs to ES, what is the best configuration tu use (number of es-masters/client/data and what is the memory heap size for each one) thank you

mootezbessifi commented 6 years ago

the strange think here is that when it was es-data heap = 1G like es-master and increasing it, it was working fine, but when increasing its heap to 4GB as you recommended @rewt , it returns out of memory issue !!!!

rewt commented 6 years ago

@mootezbessifi In my experience, I was giving java heap more RAM than container was able to provide, so while pod worked fine for a while, as time went on, Java heap would attempt to allocate more RAM than available - hence java heap errors.

I modified resource requests in stateful yaml to include memory req and limits and error, configured java_opts within those limits, and errors resolved.

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/

mootezbessifi commented 6 years ago

@rewt in my case i am running on top of machines (k8s cluster nodes) with 16GB memory. and as i mentioned before i have 2 es-clients, 3 es-masters and 3 es-data. what is the most suitable memory heap configuration for each es-{type}.

thank you

rewt commented 6 years ago

@mootezbessifi That is essentially your call and depends what kind of data ES needs to handle.

To resolve java heap out of memory errors, start by checking how much RAM the container is currently consuming:

kubectl top pod <data node name>

Then be sure ES_JAVA_OPTS does not exceed how much RAM the pod is currently consuming.

From there you can set limits for the container based on link above.