Periodic node crash. JVM crashes with SIGSEGV every few days.

Elasticsearch version: 2.2.0

JVM version: java version "1.8.0_74" Java(TM) SE Runtime Environment (build 1.8.0_74-b02) Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)

OS version: Ubuntu 14.04.4 LTS, Linux 4.2.0-30-generic #36~14.04.1-Ubuntu SMP Fri Feb 26 18:49:23 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: During normal operation, one of my nodes will randomly crash every few days with a JVM segfault

Steps to reproduce: None, problem is intermittent during normal operation. Workload is constant indexing (log data), very infrequent searches.

Provide logs (if relevant): See attached files and below kaiju-v4.txt hs_err_pid21172.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  [thread 140246293530368 also had an error]
[thread 140241095403264 also had an error]
[thread 140236838332160 also had an error]
SIGSEGV (0xb) at pc=0x00007f8e650b3cd7, pid=21172, tid=140246295635712
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x436cd7]  ciObjectFactory::create_new_metadata(Metadata*)+0x327
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /mnt/elastic/elasticsearch-2.2.0/hs_err_pid21172.log

[error occurred during error reporting , id 0xb]

#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)

elastic / elasticsearch

Periodic node crash. JVM crashes with SIGSEGV every few days. #17371