search

Waziup / WaziCloud

WAZIUP Cloud and local platform
31 stars 28 forks source link

Elasticsearch is down #152

Closed sabrine2020 closed 7 years ago

sabrine2020 commented 7 years ago

image

ChaDup commented 7 years ago

We saw yesterday an out of memory error. Before running elasticsearch, we must increase the limits of virtual memory : https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html There is also the jvm heap size : https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

Any other important configuration are described here : https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config.html

cdupont commented 7 years ago

Ok I found the kubernetes files, they where in the manifest folder... Psss There is this line: https://github.com/Waziup/Platform/blob/master/visualisation/ELS-kibana-logstash/manifests/elasticsearch.yaml#L20 Now the heap size is 1g. Is it enough? Our VM is 8G, so 1G seems already big for one app...

Regarding the swap, we had a problem today with the kubernetes VM going extremely slow. It was swapping heavily; and the "java" process was consuming 120% CPU. Maybe it's coming from elastic search?

Elastic search seems to work: http://elasticsearch.waziup.io/ But Kibana still displays red...

sabrine2020 commented 7 years ago

Here is some note about the use of ELS in production: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_notes_for_production_use_and_defaults Concerning swapping, it mention that it needs to be disabled for performance and node stability.

In this link they recommand using 50% of the machine RAM :sss https://www.elastic.co/guide/en/elasticsearch/reference/master/heap-size.html

The cluster status is red: http://elasticsearch.waziup.io/_cluster/health

cdupont commented 7 years ago

@bures any opinion on this? Should we raise the heap?

cdupont commented 7 years ago

I raised the heap to 2G, but the cluster is still red. Is it a configuration problem?

sabrine2020 commented 7 years ago

The status is yellow now and kibana is working.

cdupont commented 7 years ago

What does yellow means?

ChaDup commented 7 years ago

We also have yellow in our internal instance at EGM. I think it is because the configuration is node optimum (for exampe not several nodes) but working

sabrine2020 commented 7 years ago

I set replication factor of all the indices to 0 since we have one node: http://stackoverflow.com/questions/29192835/elasticsearchhow-to-change-cluster-health-from-yellow-to-green

the status is green now.

cdupont commented 7 years ago

Great! Should I close the issue?

sabrine2020 commented 7 years ago

yes I think the problem was solved.