elasticsearch 6.4.2: High CPU Usage when Idle

[mfza]

Hello there,

i followed the instructions for RHEL 7.4 in this Wikipage: https://github.com/linux-on-ibm-z/docs/wiki/Building-Elasticsearch

We are using ClefOS 7.4 ( CentOS for s390x ) on a z14 Machine. We are experimenting with elasticsearch in our zVM Environment and got stuck with elasticsearch.

The Instance of elasticsearch is running and accepting data, but keeps both vCPUs on 100% after startup. I build the version 6.4.2 from GitHub with the AdoptOpenJDK 10+13. We are running elasticsearch as a service via systemd with a seperate user.

Hot-Threads says:

[root@<cluster>elasticsearch]# curl -X GET "localhost:9200/_nodes/hot_threads"
::: {<cluster>}{q9dhMl3ZReavOBR9J1EF2g}{EeILzSGFQUS4ZejbFA64Rw}{<ip>}{<ip>:9300}{xpack.installed=true}
   Hot threads at 2018-11-08T08:48:04.324, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

    0.0% (6.9micros out of 500ms) cpu usage by thread 'elasticsearch[<cluster>][[timer]]'
     10/10 snapshots sharing following 2 elements
       java.base@10.0.2-adoptopenjdk/java.lang.Thread.sleep(Native Method)
       app//org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:543)

[root@<cluster>elasticsearch]# curl -X GET "localhost:9200/_nodes/hot_threads"
::: {<cluster>}{q9dhMl3ZReavOBR9J1EF2g}{EeILzSGFQUS4ZejbFA64Rw}{<ip>}{<ip>:9300}{xpack.installed=true}
   Hot threads at 2018-11-08T08:48:07.260, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

    0.0% (9.3micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
     10/10 snapshots sharing following 4 elements
       java.base@10.0.2-adoptopenjdk/java.lang.Thread.sleep(Native Method)
       io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
       io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
       java.base@10.0.2-adoptopenjdk/java.lang.Thread.run(Thread.java:844)

Our elasticsearch configuration: see files attached elasticsearch.zip

Can someone help us? We can not explain why a plain elasticsearch installation is blasting the vCPUs.

[cwsolee]

We tried to build on z14 RHEL 7.4 but we didn't observe this.

[mfza]

We will try another OS on the zVM. Maybe there ist something special with the thrid party builded Version of CentOS. I keep you updated.