search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.52k stars 24.9k forks source link

Document that Transparent Huge Pages should be disabled on Linux #26551

Open jakommo opened 7 years ago

jakommo commented 7 years ago

There seems to be a Kernel issue https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1644056 that causes a Kernel crash during high load. Also reported in discuss: https://discuss.elastic.co/t/elasticsearch-5-4-2-process-periodically-dying-with-high-cpu-load-and-kernel-message-pgtable-generic-c-33-bad-pmd/92239

It seems like this can be worked around by disabling THP on linux, e.g. echo -n never > /sys/kernel/mm/transparent_hugepage/enabled.

I had a chat with @jasontedor and we should recommend to disable THP in general, not just with the effected Kernel versions. Seems like this is enabled by default on at least Ubuntu 14/16.04 and RHEL 6/7.

I think Important System Configuration would be a good place for this.

jakommo commented 7 years ago

I talked to a user and they still experienced the above kernel bug after disabling THP, a lot less frequent though. What seems to have solved it for them now is to also disable NUMA, so maybe we can add this as well (if there are no objections from dev side).

jasontedor commented 7 years ago

As I mentioned in another channel, we should make the THP recommendation independent of any kernel bugs that may or may not be present.

As far as NUMA, our recommendations here require gathering more data and running some experiments. I don’t think we should base our recommendations on the basis of one data point (that might be fixed in different kernel versions).

elasticmachine commented 6 years ago

Pinging @elastic/es-core-infra

dliappis commented 6 years ago

One additional data point here, me and @danielmitterdorfer are working on a) evaluating the stability and b) performance behavior of Elasticsearch with and without THP. To be more exact, tuning involves not only testing THP enabled/false but also the defrag THP option, which as of kernel 4.6.1 offers new defrag strategies.

So far in our nightly benchmarking environment we have discovered that disabling THP (which in newer kernels is usually done by setting /sys/kernel/mm/transparent_hugepage/{defrag,enabled} to madvise) causes a performance drop in Elasticsearch&_a=(description:'',filters:!(),fullScreenMode:!f,options:(darkTheme:!f,useMargins:!f),panels:!((gridData:(h:30,i:'0',w:48,x:0,y:0),id:'18612537-523b-45b5-acdc-35e5ccb17646',panelIndex:'0',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'1',w:24,x:24,y:30),id:fd965200-c648-476d-9f57-b2d8d26a134e,panelIndex:'1',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'2',w:24,x:0,y:30),id:'992882c6-8720-491b-9bcf-48adb07d318f',panelIndex:'2',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'3',w:24,x:24,y:60),id:b7135960-d90e-4be3-9346-4a1d8a53b742,panelIndex:'3',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'4',w:24,x:0,y:60),id:dd636312-c49d-4fad-adb7-b81a0f9ffaef,panelIndex:'4',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'5',w:24,x:24,y:90),id:c49159f0-569c-4195-8f15-137cf74ad08b,panelIndex:'5',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'6',w:24,x:0,y:90),id:'519d2ef9-c73b-4cd6-8204-d6b055874d52',panelIndex:'6',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'7',w:24,x:24,y:120),id:'7982a99a-b766-46f6-9842-e0b5e521bb07',panelIndex:'7',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'8',w:24,x:0,y:120),id:ae597c7d-1387-4df1-b18e-9d2f7e7aacad,panelIndex:'8',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'9',w:24,x:24,y:150),id:'8f978d31-645e-4926-80e8-f0281b24be9d',panelIndex:'9',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'10',w:24,x:0,y:150),id:'70cff7d1-e920-4590-8cd3-04d74413cd68',panelIndex:'10',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'11',w:24,x:24,y:180),id:'05a50b2d-2b0e-4e1c-a99c-7be64a1aa0f3',panelIndex:'11',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'12',w:24,x:0,y:180),id:'2756a599-fdc6-43bc-8765-d38a1f0d7959',panelIndex:'12',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'13',w:24,x:24,y:210),id:b9deb64e-cd59-4879-8196-bc83cb1884b3,panelIndex:'13',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'14',w:24,x:0,y:210),id:d4812f5f-aef9-4a86-9bab-7ce4f1138e44,panelIndex:'14',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'15',w:24,x:24,y:240),id:aa8eba1e-0eb1-4947-8c6b-49c0f0121989,panelIndex:'15',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'16',w:24,x:0,y:240),id:'6790df80-16b0-4340-a9e3-f080949575a7',panelIndex:'16',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'17',w:24,x:24,y:270),id:'491e1dd2-24fb-454a-b141-212aa096c509',panelIndex:'17',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'18',w:24,x:0,y:270),id:'4b4cc39a-8dd0-4df0-a8b3-968e28fb071a',panelIndex:'18',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'19',w:24,x:24,y:300),id:d08fd81b-e24f-4001-997c-74eefc99b979,panelIndex:'19',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'20',w:24,x:0,y:300),id:e8c2e05e-e785-411c-a943-206459b01a7f,panelIndex:'20',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'21',w:24,x:24,y:330),id:'41dea9d4-efe5-42e1-8ad0-98b993af8066',panelIndex:'21',type:visualization,version:'6.3.2'),(gridData:(h:30,i:'22',w:24,x:0,y:330),id:b918cd55-d690-4e3c-8b6e-2302ae0fc938,panelIndex:'22',type:visualization,version:'6.3.2'),(gridData:(h:36,i:'23',w:47,x:0,y:360),id:'52a2cf86-ac01-42f2-8c71-8dc7acfe81ff',panelIndex:'23',type:visualization,version:'6.3.2'),(embeddableConfig:(),gridData:(h:34,i:'24',w:24,x:0,y:396),id:'00ff5004-2599-412b-8a0b-3e1c000f4b17',panelIndex:'24',type:visualization,version:'6.3.2'),(embeddableConfig:(),gridData:(h:34,i:'25',w:24,x:24,y:396),id:'424976e5-7fe9-45fb-939d-91d023dd5058',panelIndex:'25',type:visualization,version:'6.3.2'),(embeddableConfig:(),gridData:(h:27,i:'26',w:24,x:0,y:430),id:'84984a78-4131-4730-a6a1-41c326f6361f',panelIndex:'26',type:visualization,version:'6.3.2'),(embeddableConfig:(),gridData:(h:27,i:'27',w:24,x:24,y:430),id:'75bcc0c6-d9bc-44b5-9e74-697553f5ba13',panelIndex:'27',type:visualization,version:'6.3.2')),query:(language:lucene,query:(query_string:(analyze_wildcard:!t,query:'*'))),timeRestore:!f,title:nightly-oss-geonames,viewMode:view)).

Additionally, more recent versions of the Ubuntu kernel (starting with 4.12.2) are now setting THP to madvise from the enabled which used to be the default, which is how we became aware of the performance regression in the first place. madvise will also be the default setting in the upcoming Ubuntu 18.04 LTS release in a few days.

We will be providing more details when the necessary longrunning benchmarks have finished, backed by enough CI runs plus sufficient benchmarking data for a THP suggestion.

alexander-marquardt commented 6 years ago

MongoDB recommends against THP in the following document, so the same logic might apply to Elasticsearch if our access patterns are similar: https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/

pmoust commented 5 years ago

@dliappis did you get the chance to test more / dig deeper on this?

dliappis commented 5 years ago

@pmoust unfortunately the deep dive work has been paused due to other activities. However, since my last comment some things haven't been documented so I'll summarize:

jasontedor commented 5 years ago

I wouldn't expect stability issues with always, just long-tail latency spikes from transparent huge page defragging.

jrodewig commented 5 years ago

[docs issue triage]

elasticmachine commented 3 years ago

Pinging @elastic/es-perf (Team:Performance)

zez3 commented 3 years ago

@dliappis Is there somewhere a detailed description or the Cluster nodes used in elasticsearch-ci.elastic.co testing? I am most interested in JVM heap sizes and per node memory usage during the tests. Did the nodes used swap? How much swap was used?

lockewritesdocs commented 2 years ago

This issue hasn’t been updated in 3+ years so I’m closing it. We can revisit if needed.

pmoust commented 2 years ago

I'd like us to have a look again into this @tomcallahan @dliappis, if you agree. It's unclear to me what the exact recommendation is nowadays.

okwute419 commented 1 year ago

I'd like us to have a look again into this @tomcallahan @dliappis, if you agree. It's unclear to me what the exact recommendation is nowadays.

This issue hasn't been updated in almost 1yr, can we close this?

fholzer commented 1 year ago

Give the previous statement about CA. 7% performance degradation under certain circumstances I would assume it would be beneficial to get an officially recommendation on this topic an add the same.to the official documentation.

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-docs (Team:Docs)

geekpete commented 5 months ago

Will the recent enhancements to Lucene 9.11.0 around madvise have any bearing on this issue once a stack release containing Lucene 9.11.0 is released? (8.14.0 uses Lucene 9.10.0)?

https://lucene.apache.org/core/9_11_0/changes/Changes.html#v9.11.0.new_features