bubo77
commented
8 years ago i have the same issue https://github.com/elastic/elasticsearch/issues/18406
Closed fxh closed 8 years ago
bubo77
commented
8 years ago i have the same issue https://github.com/elastic/elasticsearch/issues/18406
clintongormley
commented
8 years ago This looks like a JNA problem. See http://mail-archives.apache.org/mod_mbox/cassandra-user/201405.mbox/%3Cpan.2014.05.27.15.14.54@googlemail.com%3E
Try just setting the jna tmpdir to a tmp folder without noexec.
bubo77
commented
8 years ago @clintongormley if you read this ticket and mine again you can see that we tried to point the ES config to a new directory with appropriate rights and it still doesn't work.
ls -alZ /var/lib/elasticsearch/ drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 . drwxr-xr-x. root root unconfined_u:object_r:usr_t:s0 .. drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 graylog2 drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 tmp
clintongormley
commented
8 years ago 
jasontedor
commented
8 years ago I've tried to reproduce this on Centos 6.7 and Fedora 23. It does not reproduce for me.
I notice that both error reports show death trying to register the C library. This the first step before we can even make any native calls. Thus, we are immediately dying here and it's not that we are interacting poorly with other native calls (e.g., mlockall or seccomp).
Is there additional configuration at play here that might be relevant? What else can you share about the system? What kernel do you have? I tested on 2.6.32-573.12.1.el6.x86_64 (Centos 6.7) and 4.2.3-300.fc23.x86_64 (Fedora 23). What's the output of ldd --version? What happens if you add -Djna.nosys=false to the ES_JAVA_OPTS?
fxh
commented
8 years ago Kernel: 2.6.32-573.7.1.el6.x86_64 CentOS release 6.7 (Final)
$ ldd --version
ldd (GNU libc) 2.12
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.Adding -Djna.nosys=false did not help:
Starting elasticsearch: /usr/share/elasticsearch/bin/elasticsearch: line 155: 31386 Aborted (core dumped) exec "$JAVA" $JAVA_OPTS $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH" org.elasticsearch.bootstrap.Elasticsearch start "$@" 0>&-CentOS Linux release 7.2.1511 (Core) Kernel: 3.10.0-327.10.1.el7.x86_64
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich DrepperMy tmp dir is on an LVM disk but it doesn't matter if the tmp dir is a symlink or not.
ls -alZ /export/apps/elasticsearch/
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 .
drwxr-xr-x. root root unconfined_u:object_r:usr_t:s0 ..
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 graylog2
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0 tmp@fxh @bubo77 Thanks. Nothing alarming there. Can you add these system properties to ES_JAVA_OPTS (leave off the -Djna.nosys=false):
-Djna.debug_load.jna=true -Djna.debug_load=true -Djna.tmpdir=/path/to/tmp -Djava.io.tmpdir=/path/to/tmpand start Elasticsearch (only the first two should be new). This should produce output like:
Looking in classpath from sun.misc.Launcher$AppClassLoader@5c647e05 for /com/sun/jna/linux-x86-64/libjnidispatch.so
Found library resource at jar:file:/home/jason/elasticsearch/elasticsearch-5.0.0-SNAPSHOT/lib/jna-4.1.0.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
Trying /home/jason/elasticsearch/elasticsearch-5.0.0-SNAPSHOT/tmp/jna6458947537802950020.tmp
Found jnidispatch at /home/jason/elasticsearch/elasticsearch-5.0.0-SNAPSHOT/tmp/jna6458947537802950020.tmpCan you share that and any additional JNA debugging output?
jasontedor
commented
8 years ago Also, can you share your SELinux configuration via sestatus and getsebool -a? If you're willing to, can you completely disable SELinux and try as well (preferably through /etc/selinux/config and a reboot)? If that starts up successfully, that will at least confirm my current hunch that it's SELinux and we just need to find the right configuration to let JNA do it's thing.
bubo77
commented
8 years ago sestatus:
tmp]# sestatus -v
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
Process contexts:
Current context: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
Init context: system_u:system_r:init_t:s0
/usr/sbin/sshd system_u:system_r:sshd_t:s0-s0:c0.c1023
File contexts:
Controlling terminal: unconfined_u:object_r:user_devpts_t:s0
/etc/passwd system_u:object_r:passwd_file_t:s0
/etc/shadow system_u:object_r:shadow_t:s0
/bin/bash system_u:object_r:shell_exec_t:s0
/bin/login system_u:object_r:login_exec_t:s0
/bin/sh system_u:object_r:bin_t:s0 -> system_u:object_r:shell_exec_t:s0
/sbin/agetty system_u:object_r:getty_exec_t:s0
/sbin/init system_u:object_r:bin_t:s0 -> system_u:object_r:init_exec_t:s0
/usr/sbin/sshd system_u:object_r:sshd_exec_t:s0getsebool
abrt_anon_write --> off
abrt_handle_event --> off
abrt_upload_watch_anon_write --> on
antivirus_can_scan_system --> on
antivirus_use_jit --> off
auditadm_exec_content --> on
authlogin_nsswitch_use_ldap --> off
authlogin_radius --> off
authlogin_yubikey --> off
awstats_purge_apache_log_files --> off
boinc_execmem --> on
cdrecord_read_content --> off
cluster_can_network_connect --> off
cluster_manage_all_files --> off
cluster_use_execmem --> off
cobbler_anon_write --> off
cobbler_can_network_connect --> off
cobbler_use_cifs --> off
cobbler_use_nfs --> off
collectd_tcp_network_connect --> off
condor_tcp_network_connect --> off
conman_can_network --> off
cron_can_relabel --> off
cron_system_cronjob_use_shares --> off
cron_userdomain_transition --> on
cups_execmem --> off
cvs_read_shadow --> off
daemons_dump_core --> off
daemons_enable_cluster_mode --> off
daemons_use_tcp_wrapper --> off
daemons_use_tty --> off
dbadm_exec_content --> on
dbadm_manage_user_files --> off
dbadm_read_user_files --> off
deny_execmem --> off
deny_ptrace --> off
dhcpc_exec_iptables --> off
dhcpd_use_ldap --> off
domain_fd_use --> on
domain_kernel_load_modules --> off
entropyd_use_audio --> on
exim_can_connect_db --> off
exim_manage_user_files --> off
exim_read_user_files --> off
fcron_crond --> off
fenced_can_network_connect --> off
fenced_can_ssh --> off
fips_mode --> on
ftp_home_dir --> off
ftpd_anon_write --> off
ftpd_connect_all_unreserved --> off
ftpd_connect_db --> off
ftpd_full_access --> off
ftpd_use_cifs --> off
ftpd_use_fusefs --> off
ftpd_use_nfs --> off
ftpd_use_passive_mode --> off
git_cgi_enable_homedirs --> off
git_cgi_use_cifs --> off
git_cgi_use_nfs --> off
git_session_bind_all_unreserved_ports --> off
git_session_users --> off
git_system_enable_homedirs --> off
git_system_use_cifs --> off
git_system_use_nfs --> off
gitosis_can_sendmail --> off
glance_api_can_network --> off
glance_use_execmem --> off
glance_use_fusefs --> off
global_ssp --> off
gluster_anon_write --> off
gluster_export_all_ro --> off
gluster_export_all_rw --> on
gpg_web_anon_write --> off
gssd_read_tmp --> on
guest_exec_content --> on
haproxy_connect_any --> off
httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_dbus_sssd --> off
httpd_dontaudit_search_dirs --> off
httpd_enable_cgi --> on
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_graceful_shutdown --> on
httpd_manage_ipa --> off
httpd_mod_auth_ntlm_winbind --> off
httpd_mod_auth_pam --> off
httpd_read_user_content --> off
httpd_run_ipa --> off
httpd_run_preupgrade --> off
httpd_run_stickshift --> off
httpd_serve_cobbler_files --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_sys_script_anon_write --> off
httpd_tmp_exec --> off
httpd_tty_comm --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_fusefs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
httpd_use_openstack --> off
httpd_use_sasl --> off
httpd_verify_dns --> off
icecast_use_any_tcp_ports --> off
irc_use_any_tcp_ports --> off
irssi_use_full_network --> off
kdumpgui_run_bootloader --> off
kerberos_enabled --> on
ksmtuned_use_cifs --> off
ksmtuned_use_nfs --> off
logadm_exec_content --> on
logging_syslogd_can_sendmail --> off
logging_syslogd_run_nagios_plugins --> off
logging_syslogd_use_tty --> on
login_console_enabled --> on
logrotate_use_nfs --> off
logwatch_can_network_connect_mail --> off
lsmd_plugin_connect_any --> off
mailman_use_fusefs --> off
mcelog_client --> off
mcelog_exec_scripts --> on
mcelog_foreground --> off
mcelog_server --> off
minidlna_read_generic_user_content --> off
mmap_low_allowed --> off
mock_enable_homedirs --> off
mount_anyfile --> on
mozilla_plugin_bind_unreserved_ports --> off
mozilla_plugin_can_network_connect --> off
mozilla_plugin_use_bluejeans --> off
mozilla_plugin_use_gps --> off
mozilla_plugin_use_spice --> off
mozilla_read_content --> off
mpd_enable_homedirs --> off
mpd_use_cifs --> off
mpd_use_nfs --> off
mplayer_execstack --> off
mysql_connect_any --> off
nagios_run_pnp4nagios --> off
nagios_run_sudo --> off
named_tcp_bind_http_port --> off
named_write_master_zones --> off
neutron_can_network --> off
nfs_export_all_ro --> on
nfs_export_all_rw --> on
nfsd_anon_write --> off
nis_enabled --> off
nscd_use_shm --> on
openshift_use_nfs --> off
openvpn_can_network_connect --> on
openvpn_enable_homedirs --> on
openvpn_run_unconfined --> off
pcp_bind_all_unreserved_ports --> off
pcp_read_generic_logs --> off
piranha_lvs_can_network_connect --> off
polipo_connect_all_unreserved --> off
polipo_session_bind_all_unreserved_ports --> off
polipo_session_users --> off
polipo_use_cifs --> off
polipo_use_nfs --> off
polyinstantiation_enabled --> off
postfix_local_write_mail_spool --> on
postgresql_can_rsync --> off
postgresql_selinux_transmit_client_label --> off
postgresql_selinux_unconfined_dbadm --> on
postgresql_selinux_users_ddl --> on
pppd_can_insmod --> off
pppd_for_user --> off
privoxy_connect_any --> on
prosody_bind_http_port --> off
puppetagent_manage_all_files --> off
puppetmaster_use_db --> off
racoon_read_shadow --> off
rsync_anon_write --> off
rsync_client --> off
rsync_export_all_ro --> off
rsync_full_access --> off
samba_create_home_dirs --> off
samba_domain_controller --> off
samba_enable_home_dirs --> off
samba_export_all_ro --> off
samba_export_all_rw --> off
samba_load_libgfapi --> off
samba_portmapper --> off
samba_run_unconfined --> off
samba_share_fusefs --> off
samba_share_nfs --> off
sanlock_use_fusefs --> off
sanlock_use_nfs --> off
sanlock_use_samba --> off
saslauthd_read_shadow --> off
secadm_exec_content --> on
secure_mode --> off
secure_mode_insmod --> off
secure_mode_policyload --> off
selinuxuser_direct_dri_enabled --> on
selinuxuser_execheap --> off
selinuxuser_execmod --> on
selinuxuser_execstack --> on
selinuxuser_mysql_connect_enabled --> off
selinuxuser_ping --> on
selinuxuser_postgresql_connect_enabled --> off
selinuxuser_rw_noexattrfile --> on
selinuxuser_share_music --> off
selinuxuser_tcp_server --> off
selinuxuser_udp_server --> off
selinuxuser_use_ssh_chroot --> off
sftpd_anon_write --> off
sftpd_enable_homedirs --> off
sftpd_full_access --> off
sftpd_write_ssh_home --> off
sge_domain_can_network_connect --> off
sge_use_nfs --> off
smartmon_3ware --> off
smbd_anon_write --> off
spamassassin_can_network --> off
spamd_enable_home_dirs --> on
squid_connect_any --> on
squid_use_tproxy --> off
ssh_chroot_rw_homedirs --> off
ssh_keysign --> off
ssh_sysadm_login --> off
staff_exec_content --> on
staff_use_svirt --> off
swift_can_network --> off
sysadm_exec_content --> on
telepathy_connect_all_ports --> off
telepathy_tcp_connect_generic_network_ports --> on
tftp_anon_write --> off
tftp_home_dir --> off
tmpreaper_use_nfs --> off
tmpreaper_use_samba --> off
tor_bind_all_unreserved_ports --> off
tor_can_network_relay --> off
unconfined_chrome_sandbox_transition --> on
unconfined_login --> on
unconfined_mozilla_plugin_transition --> on
unprivuser_use_svirt --> off
use_ecryptfs_home_dirs --> off
use_fusefs_home_dirs --> off
use_lpd_server --> off
use_nfs_home_dirs --> off
use_samba_home_dirs --> off
user_exec_content --> on
varnishd_connect_any --> off
virt_read_qemu_ga_data --> off
virt_rw_qemu_ga_data --> off
virt_sandbox_use_all_caps --> on
virt_sandbox_use_audit --> on
virt_sandbox_use_mknod --> off
virt_sandbox_use_netlink --> off
virt_sandbox_use_nfs --> off
virt_sandbox_use_samba --> off
virt_sandbox_use_sys_admin --> off
virt_transition_userdomain --> off
virt_use_comm --> off
virt_use_execmem --> off
virt_use_fusefs --> off
virt_use_nfs --> off
virt_use_rawip --> off
virt_use_samba --> off
virt_use_sanlock --> off
virt_use_usb --> on
virt_use_xserver --> off
webadm_manage_user_files --> off
webadm_read_user_files --> off
wine_mmap_zero_ignore --> off
xdm_bind_vnc_tcp_port --> off
xdm_exec_bootloader --> off
xdm_sysadm_login --> off
xdm_write_home --> off
xen_use_nfs --> off
xend_run_blktap --> on
xend_run_qemu --> on
xguest_connect_network --> on
xguest_exec_content --> on
xguest_mount_media --> on
xguest_use_bluetooth --> on
xserver_clients_write_xshm --> off
xserver_execmem --> off
xserver_object_manager --> off
zabbix_can_network --> off
zarafa_setrlimit --> off
zebra_write_config --> off
zoneminder_anon_write --> off
zoneminder_run_sudo --> offsystemctl dump:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f8c619d340a, pid=19921, tid=140242008348416
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [jna46830484485062808.tmp+0x1240a] ffi_prep_closure_loc+0x1a
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- T H R E A D ---------------
Current thread (0x00007f8c9c00a000): JavaThread "main" [_thread_in_native, id=19939, stack(0x00007f8ca3071000,0x00007f8ca3172000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
Registers:
RAX=0x00007f8c619d38c2, RBX=0x00007f8ca316fb28, RCX=0x00007f8c9c5667f0, RDX=0x00007f8c619c8590
RSP=0x00007f8ca316f8c0, RBP=0x00007f8ca316f8c0, RSI=0x00007f8c9c566810, RDI=0x0000000000000000
R8 =0x00007f8c9c00ab60, R9 =0x0000000000000002, R10=0x00007f8ca316f650, R11=0x00007f8c619d33f0
R12=0x0000000000000000, R13=0x0000000000000008, R14=0x00007f8c9c5667f0, R15=0x0000000000000004
RIP=0x00007f8c619d340a, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f8ca316f8c0)
0x00007f8ca316f8c0: 00007f8ca316f990 00007f8c619ce4dd
0x00007f8ca316f8d0: 00007f8ca316f950 00007f8ca316fb18
0x00007f8ca316f8e0: 00007f8ca316fb30 00007f8c9c00a1f8
0x00007f8ca316f8f0: 0000000100000000 00007f8c9c56b660
0x00007f8ca316f900: 00007f8c9c56b7a0 00007f8c9c566810
0x00007f8ca316f910: 00000001a316fa80 00007f8c9c566910
0x00007f8ca316f920: 00007f8c9c56a5b0 00007f8c9c56a590
0x00007f8ca316f930: 0000000000000001 00000003a316fb28
0x00007f8ca316f940: 00007f8ca316fb20 00007f8ca316fb18
0x00007f8ca316f950: 00007f8c00000000 00007f8c9c00ab60
0x00007f8ca316f960: 00007f8c7dcbcb78 00007f8c7dcbcb78
0x00007f8ca316f970: 0000000000000000 00007f8c7dcbcb78
0x00007f8ca316f980: 00007f8ca316fb30 00007f8c9c00a000
0x00007f8ca316f990: 00007f8ca316fa80 00007f8c8d0156d4
0x00007f8ca316f9a0: 00007f8ca316fb10 00007f8ca316fb08
0x00007f8ca316f9b0: 0000000000000002 00007f8c619d5eb0
0x00007f8ca316f9c0: 00007f8c619d5eb0 00007f8ca316fad8
0x00007f8ca316f9d0: 00007f8ca25ee0e0 0000000000000000
0x00007f8ca316f9e0: 0000000000000000 00007f8ca316fab0
0x00007f8ca316f9f0: 0000000000000000 00007f8ca316faa0
0x00007f8ca316fa00: 00007f8c9c00a000 00007f8c9c00a000
0x00007f8ca316fa10: 00007f8c7dcbcb78 0000000000000000
0x00007f8ca316fa20: 00007f8ca316fa80 00007f8c8d015459
0x00007f8ca316fa30: 00007f8c9c00a000 00007f8c8d015422
0x00007f8ca316fa40: 00007f8ca316fa40 00007f8c7dcbcb78
0x00007f8ca316fa50: 00007f8ca316fb30 00007f8c7dcc02d8
0x00007f8ca316fa60: 0000000000000000 00007f8c7dcbcb78
0x00007f8ca316fa70: 0000000000000000 00007f8ca316faa0
0x00007f8ca316fa80: 00007f8ca316fb88 00007f8c8d0074a0
0x00007f8ca316fa90: 00000006c14d96a8 00007f8c8d00f2b7
0x00007f8ca316faa0: 00000006c0010ad0 0000000000000000
0x00007f8ca316fab0: 00000006c15aa230 0000000000000000
Instructions: (pc=0x00007f8c619d340a)
0x00007f8c619d33ea: 66 90 66 66 66 90 8b 06 55 41 b9 02 00 00 00 48
0x00007f8c619d33fa: 89 e5 ff c8 83 f8 01 77 44 48 8b 05 c6 49 10 00
0x00007f8c619d340a: 66 c7 07 49 bb 4c 89 47 0c 66 c7 47 0a 49 ba 48
0x00007f8c619d341a: 89 47 02 8b 46 1c 48 89 77 18 48 89 57 20 48 89
Register to memory mapping:
RAX=0x00007f8c619d38c2: ffi_closure_unix64+0 in /export/apps/elasticsearch/tmp/jna46830484485062808.tmp at 0x00007f8c619c1000
RBX=0x00007f8ca316fb28 is pointing into the stack for thread: 0x00007f8c9c00a000
RCX=0x00007f8c9c5667f0 is an unknown value
RDX=0x00007f8c619c8590: <offset 0x7590> in /export/apps/elasticsearch/tmp/jna46830484485062808.tmp at 0x00007f8c619c1000
RSP=0x00007f8ca316f8c0 is pointing into the stack for thread: 0x00007f8c9c00a000
RBP=0x00007f8ca316f8c0 is pointing into the stack for thread: 0x00007f8c9c00a000
RSI=0x00007f8c9c566810 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x00007f8c9c00ab60 is an unknown value
R9 =0x0000000000000002 is an unknown value
R10=0x00007f8ca316f650 is pointing into the stack for thread: 0x00007f8c9c00a000
R11=0x00007f8c619d33f0: ffi_prep_closure_loc+0 in /export/apps/elasticsearch/tmp/jna46830484485062808.tmp at 0x00007f8c619c1000
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000008 is an unknown value
R14=0x00007f8c9c5667f0 is an unknown value
R15=0x0000000000000004 is an unknown value
Stack: [0x00007f8ca3071000,0x00007f8ca3172000], sp=0x00007f8ca316f8c0, free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [jna46830484485062808.tmp+0x1240a] ffi_prep_closure_loc+0x1a
C [jna46830484485062808.tmp+0xd4dd] Java_com_sun_jna_Native_registerMethod+0x45d
j com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/Class;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+979
j com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+28
j com.sun.jna.Native.register(Ljava/lang/String;)V+7
j org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+44
v ~StubRoutines::call_stub
V [libjvm.so+0x681a26] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x634fd7] InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0xd7
V [libjvm.so+0x636c2c] InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac
V [libjvm.so+0x636fe1] InstanceKlass::initialize(Thread*)+0x41
V [libjvm.so+0x7e9756] LinkResolver::resolve_static_call(CallInfo&, KlassHandle&, Symbol*, Symbol*, KlassHandle, bool, bool, Thread*)+0x246
V [libjvm.so+0x7e99df] LinkResolver::resolve_invokestatic(CallInfo&, constantPoolHandle, int, Thread*)+0x23f
V [libjvm.so+0x7e9f11] LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle, int, Bytecodes::Code, Thread*)+0x4f1
V [libjvm.so+0x67a972] InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code)+0x1b2
j org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+22
j org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+7
j org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/env/Environment;)V+43
j org.elasticsearch.bootstrap.Bootstrap.init([Ljava/lang/String;)V+235
j org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+1
v ~StubRoutines::call_stub
V [libjvm.so+0x681a26] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x6c3692] jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x362
V [libjvm.so+0x6e009a] jni_CallStaticVoidMethod+0x17a
C [libjli.so+0x7bcc] JavaMain+0x80c
C [libpthread.so.0+0x7dc5] start_thread+0xc5
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/Class;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+979
j com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+28
j com.sun.jna.Native.register(Ljava/lang/String;)V+7
j org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+44
v ~StubRoutines::call_stub
j org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+22
j org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+7
j org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/env/Environment;)V+43
j org.elasticsearch.bootstrap.Bootstrap.init([Ljava/lang/String;)V+235
j org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+1
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f8c9c131000 JavaThread "Service Thread" daemon [_thread_blocked, id=19950, stack(0x00007f8c7cc1e000,0x00007f8c7cd1f000)]
0x00007f8c9c109800 JavaThread "C1 CompilerThread1" daemon [_thread_blocked, id=19949, stack(0x00007f8c7cd1f000,0x00007f8c7ce20000)]
0x00007f8c9c107800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=19948, stack(0x00007f8c7ce20000,0x00007f8c7cf21000)]
0x00007f8c9c105000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=19947, stack(0x00007f8c7cf21000,0x00007f8c7d022000)]
0x00007f8c9c103000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=19946, stack(0x00007f8c7d022000,0x00007f8c7d123000)]
0x00007f8c9c0d8800 JavaThread "Finalizer" daemon [_thread_blocked, id=19945, stack(0x00007f8c7d123000,0x00007f8c7d224000)]
0x00007f8c9c0d6000 JavaThread "Reference Handler" daemon [_thread_blocked, id=19944, stack(0x00007f8c7d224000,0x00007f8c7d325000)]
=>0x00007f8c9c00a000 JavaThread "main" [_thread_in_native, id=19939, stack(0x00007f8ca3071000,0x00007f8ca3172000)]
Other Threads:
0x00007f8c9c0d1000 VMThread [stack: 0x00007f8c7d325000,0x00007f8c7d426000] [id=19943]
0x00007f8c9c134000 WatcherThread [stack: 0x00007f8c7cb1d000,0x00007f8c7cc1e000] [id=19951]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap:
par new generation total 153344K, used 24554K [0x00000006c0000000, 0x00000006ca660000, 0x00000006ca660000)
eden space 136320K, 18% used [0x00000006c0000000, 0x00000006c17fa9b8, 0x00000006c8520000)
from space 17024K, 0% used [0x00000006c8520000, 0x00000006c8520000, 0x00000006c95c0000)
to space 17024K, 0% used [0x00000006c95c0000, 0x00000006c95c0000, 0x00000006ca660000)
concurrent mark-sweep generation total 4023936K, used 0K [0x00000006ca660000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 8237K, capacity 8360K, committed 8576K, reserved 1056768K
class space used 998K, capacity 1040K, committed 1152K, reserved 1048576K
Card table byte_map: [0x00007f8c8c7ff000,0x00007f8c8d000000] byte_map_base: 0x00007f8c891ff000
Marking Bits: (CMSBitMap*) 0x00007f8c9c0499f8
Bits: [0x00007f8c84299000, 0x00007f8c87fff800)
Mod Union Table: (CMSBitMap*) 0x00007f8c9c049ab8
Bits: [0x00007f8c841a3000, 0x00007f8c842989a0)
Polling page: 0x00007f8ca317e000
CodeCache: size=245760Kb used=2282Kb max_used=2282Kb free=243477Kb
bounds [0x00007f8c8d000000, 0x00007f8c8d270000, 0x00007f8c9c000000]
total_blobs=900 nmethods=552 adapters=263
compilation: enabled
Compilation events (10 events):
Event: 0.576 Thread 0x00007f8c9c109800 545 ! 3 java.util.zip.ZipFile$ZipFileInflaterInputStream::close (67 bytes)
Event: 0.577 Thread 0x00007f8c9c109800 nmethod 545 0x00007f8c8d238450 code [0x00007f8c8d238620, 0x00007f8c8d238e58]
Event: 0.577 Thread 0x00007f8c9c109800 546 3 java.util.zip.InflaterInputStream::close (34 bytes)
Event: 0.577 Thread 0x00007f8c9c109800 nmethod 546 0x00007f8c8d239210 code [0x00007f8c8d2393a0, 0x00007f8c8d239698]
Event: 0.577 Thread 0x00007f8c9c109800 547 3 java.util.zip.ZipFile::access$100 (6 bytes)
Event: 0.577 Thread 0x00007f8c9c109800 nmethod 547 0x00007f8c8d239790 code [0x00007f8c8d239900, 0x00007f8c8d239a68]
Event: 0.577 Thread 0x00007f8c9c109800 548 ! 3 java.util.zip.ZipFile::releaseInflater (41 bytes)
Event: 0.578 Thread 0x00007f8c9c109800 nmethod 548 0x00007f8c8d239b10 code [0x00007f8c8d239cc0, 0x00007f8c8d23a3f8]
Event: 0.578 Thread 0x00007f8c9c109800 549 ! 3 java.util.zip.Inflater::reset (69 bytes)
Event: 0.578 Thread 0x00007f8c9c109800 nmethod 549 0x00007f8c8d23a710 code [0x00007f8c8d23a8a0, 0x00007f8c8d23acf8]
GC Heap History (0 events):
No events
Deoptimization events (2 events):
Event: 0.154 Thread 0x00007f8c9c00a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c8d151b40 method=java.lang.String.indexOf(II)I @ 49
Event: 0.501 Thread 0x00007f8c9c00a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c8d1f8274 method=sun.misc.MetaIndex.mayContain(Ljava/lang/String;)Z @ 38
Internal exceptions (10 events):
Event: 0.492 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/LayoutCustomizer> (0x00000006c1385118) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.493 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/PatternLayoutCustomizer> (0x00000006c1390fb8) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/DailyRollingFileAppenderBeanInfo> (0x00000006c13cfdd0) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/FileAppenderBeanInfo> (0x00000006c13d6ff8) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/WriterAppenderBeanInfo> (0x00000006c13de3a0) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.506 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/AppenderSkeletonBeanInfo> (0x00000006c13e6058) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.507 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/AppenderSkeletonCustomizer> (0x00000006c13ee1e8) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.509 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/WriterAppenderCustomizer> (0x00000006c1407818) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.511 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/FileAppenderCustomizer> (0x00000006c141b250) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 0.513 Thread 0x00007f8c9c00a000 Exception <a 'java/lang/ClassNotFoundException': org/apache/log4j/DailyRollingFileAppenderCustomizer> (0x00000006c14308c0) thrown at [/RE-WORK/workspace/8-2-build-linux-amd64/jdk8u45/3457/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Events (10 events):
Event: 0.574 loading class com/sun/jna/Function
Event: 0.574 loading class com/sun/jna/Function done
Event: 0.575 loading class com/sun/jna/MethodParameterContext
Event: 0.575 loading class com/sun/jna/MethodParameterContext done
Event: 0.575 loading class com/sun/jna/FunctionParameterContext
Event: 0.575 loading class com/sun/jna/FunctionParameterContext done
Event: 0.576 loading class com/sun/jna/MethodResultContext
Event: 0.576 loading class com/sun/jna/MethodResultContext done
Event: 0.576 loading class com/sun/jna/FunctionResultContext
Event: 0.576 loading class com/sun/jna/FunctionResultContext done
Dynamic libraries:
00400000-00401000 r-xp 00000000 fd:00 135193 /usr/java/jre1.8.0_45/bin/java
00600000-00601000 rw-p 00000000 fd:00 135193 /usr/java/jre1.8.0_45/bin/java
023b6000-023d7000 rw-p 00000000 00:00 0 [heap]
6c0000000-7c0120000 rw-p 00000000 00:00 0
7c0120000-800000000 ---p 00000000 00:00 0
7f8c4c000000-7f8c4c021000 rw-p 00000000 00:00 0
7f8c4c021000-7f8c50000000 ---p 00000000 00:00 0
7f8c50000000-7f8c50021000 rw-p 00000000 00:00 0
7f8c50021000-7f8c54000000 ---p 00000000 00:00 0
7f8c54000000-7f8c5423f000 rw-p 00000000 00:00 0
7f8c5423f000-7f8c58000000 ---p 00000000 00:00 0
7f8c58000000-7f8c581a4000 rw-p 00000000 00:00 0
7f8c581a4000-7f8c5c000000 ---p 00000000 00:00 0
7f8c5c000000-7f8c5c021000 rw-p 00000000 00:00 0
7f8c5c021000-7f8c60000000 ---p 00000000 00:00 0
7f8c619c1000-7f8c619d8000 r-xp 00000000 fd:05 131152 /export/apps/elasticsearch/tmp/jna46830484485062808.tmp (deleted)
7f8c619d8000-7f8c61ad7000 ---p 00017000 fd:05 131152 /export/apps/elasticsearch/tmp/jna46830484485062808.tmp (deleted)
7f8c61ad7000-7f8c61ad9000 rw-p 00016000 fd:05 131152 /export/apps/elasticsearch/tmp/jna46830484485062808.tmp (deleted)
7f8c61ad9000-7f8c68000000 r--p 00000000 fd:00 4384 /usr/lib/locale/locale-archive
7f8c68000000-7f8c68021000 rw-p 00000000 00:00 0
7f8c68021000-7f8c6c000000 ---p 00000000 00:00 0
7f8c6c000000-7f8c6c021000 rw-p 00000000 00:00 0
7f8c6c021000-7f8c70000000 ---p 00000000 00:00 0
7f8c70000000-7f8c70021000 rw-p 00000000 00:00 0
7f8c70021000-7f8c74000000 ---p 00000000 00:00 0
7f8c74000000-7f8c74021000 rw-p 00000000 00:00 0
7f8c74021000-7f8c78000000 ---p 00000000 00:00 0
7f8c78000000-7f8c78021000 rw-p 00000000 00:00 0
7f8c78021000-7f8c7c000000 ---p 00000000 00:00 0
7f8c7c0bb000-7f8c7c0bd000 rw-p 00000000 00:00 0
7f8c7c0bd000-7f8c7c0c5000 r-xp 00000000 fd:00 135250 /usr/java/jre1.8.0_45/lib/amd64/libmanagement.so
7f8c7c0c5000-7f8c7c2c5000 ---p 00008000 fd:00 135250 /usr/java/jre1.8.0_45/lib/amd64/libmanagement.so
7f8c7c2c5000-7f8c7c2c6000 rw-p 00008000 fd:00 135250 /usr/java/jre1.8.0_45/lib/amd64/libmanagement.so
7f8c7c2c6000-7f8c7c2d1000 r--s 00210000 fd:00 135273 /usr/java/jre1.8.0_45/lib/ext/localedata.jar
7f8c7c2d1000-7f8c7c2f3000 r--s 0038e000 fd:00 135392 /usr/java/jre1.8.0_45/lib/ext/cldrdata.jar
7f8c7c2f3000-7f8c7c304000 r-xp 00000000 fd:00 135253 /usr/java/jre1.8.0_45/lib/amd64/libnio.so
7f8c7c304000-7f8c7c503000 ---p 00011000 fd:00 135253 /usr/java/jre1.8.0_45/lib/amd64/libnio.so
7f8c7c503000-7f8c7c504000 rw-p 00010000 fd:00 135253 /usr/java/jre1.8.0_45/lib/amd64/libnio.so
7f8c7c504000-7f8c7c508000 r--s 00085000 fd:00 135516 /usr/java/jre1.8.0_45/lib/jsse.jar
7f8c7c508000-7f8c7c808000 rw-p 00000000 00:00 0
7f8c7c808000-7f8c7c81e000 r-xp 00000000 fd:00 135252 /usr/java/jre1.8.0_45/lib/amd64/libnet.so
7f8c7c81e000-7f8c7ca1e000 ---p 00016000 fd:00 135252 /usr/java/jre1.8.0_45/lib/amd64/libnet.so
7f8c7ca1e000-7f8c7ca1f000 rw-p 00016000 fd:00 135252 /usr/java/jre1.8.0_45/lib/amd64/libnet.so
7f8c7ca1f000-7f8c7ca23000 r--s 00018000 fd:00 138818 /usr/share/elasticsearch/lib/compiler-0.8.13.jar
7f8c7ca23000-7f8c7ca4a000 r--s 0021c000 fd:00 138832 /usr/share/elasticsearch/lib/lucene-core-5.5.0.jar
7f8c7ca4a000-7f8c7ca52000 r--s 00062000 fd:00 138831 /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar
7f8c7ca52000-7f8c7ca5b000 r--s 00046000 fd:00 138826 /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.2.jar
7f8c7ca5b000-7f8c7ca6c000 r--s 000b2000 fd:00 138830 /usr/share/elasticsearch/lib/jts-1.13.jar
7f8c7ca6c000-7f8c7cb1d000 r--s 0084c000 fd:00 138820 /usr/share/elasticsearch/lib/elasticsearch-2.3.2.jar
7f8c7cb1d000-7f8c7cb1e000 ---p 00000000 00:00 0
7f8c7cb1e000-7f8c7cc1e000 rw-p 00000000 00:00 0 [stack:19951]
7f8c7cc1e000-7f8c7cc21000 ---p 00000000 00:00 0
7f8c7cc21000-7f8c7cd1f000 rw-p 00000000 00:00 0 [stack:19950]
7f8c7cd1f000-7f8c7cd22000 ---p 00000000 00:00 0
7f8c7cd22000-7f8c7ce20000 rw-p 00000000 00:00 0 [stack:19949]
7f8c7ce20000-7f8c7ce23000 ---p 00000000 00:00 0
7f8c7ce23000-7f8c7cf21000 rw-p 00000000 00:00 0 [stack:19948]
7f8c7cf21000-7f8c7cf24000 ---p 00000000 00:00 0
7f8c7cf24000-7f8c7d022000 rw-p 00000000 00:00 0 [stack:19947]
7f8c7d022000-7f8c7d025000 ---p 00000000 00:00 0
7f8c7d025000-7f8c7d123000 rw-p 00000000 00:00 0 [stack:19946]
7f8c7d123000-7f8c7d126000 ---p 00000000 00:00 0
7f8c7d126000-7f8c7d224000 rw-p 00000000 00:00 0 [stack:19945]
7f8c7d224000-7f8c7d227000 ---p 00000000 00:00 0
7f8c7d227000-7f8c7d325000 rw-p 00000000 00:00 0 [stack:19944]
7f8c7d325000-7f8c7d326000 ---p 00000000 00:00 0
7f8c7d326000-7f8c7d426000 rw-p 00000000 00:00 0 [stack:19943]
7f8c7d426000-7f8c7d5fe000 r--s 0341d000 fd:00 135563 /usr/java/jre1.8.0_45/lib/rt.jar
7f8c7d5fe000-7f8c7dd3e000 rw-p 00000000 00:00 0
7f8c7dd3e000-7f8c7ddfe000 ---p 00000000 00:00 0
7f8c7ddfe000-7f8c7ddff000 ---p 00000000 00:00 0
7f8c7ddff000-7f8c80000000 rw-p 00000000 00:00 0 [stack:19942]
7f8c80000000-7f8c80021000 rw-p 00000000 00:00 0
7f8c80021000-7f8c84000000 ---p 00000000 00:00 0
7f8c84000000-7f8c84002000 r--s 00019000 fd:00 135576 /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar
7f8c84002000-7f8c84019000 r--s 00100000 fd:00 138822 /usr/share/elasticsearch/lib/hppc-0.7.1.jar
7f8c84019000-7f8c8402c000 r--s 0016f000 fd:00 135585 /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar
7f8c8402c000-7f8c88000000 rw-p 00000000 00:00 0
7f8c88000000-7f8c88021000 rw-p 00000000 00:00 0
7f8c88021000-7f8c8c000000 ---p 00000000 00:00 0
7f8c8c000000-7f8c8c001000 rw-p 00000000 00:00 0
7f8c8c001000-7f8c8c005000 r--s 00020000 fd:00 138834 /usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar
7f8c8c005000-7f8c8c009000 r--s 00035000 fd:00 138840 /usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar
7f8c8c009000-7f8c8c00b000 r--s 0000e000 fd:00 138829 /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar
7f8c8c00b000-7f8c8c00e000 r--s 00018000 fd:00 138833 /usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar
7f8c8c00e000-7f8c8c027000 r--s 0012c000 fd:00 138844 /usr/share/elasticsearch/lib/netty-3.10.5.Final.jar
7f8c8c027000-7f8c8c052000 r--s 001fc000 fd:00 138821 /usr/share/elasticsearch/lib/guava-18.0.jar
7f8c8c052000-7f8c8cfff000 rw-p 00000000 00:00 0
7f8c8cfff000-7f8c8d000000 rw-p 00000000 00:00 0
7f8c8d000000-7f8c8d270000 rwxp 00000000 00:00 0
7f8c8d270000-7f8c9c000000 ---p 00000000 00:00 0
7f8c9c000000-7f8c9c801000 rw-p 00000000 00:00 0
7f8c9c801000-7f8ca0000000 ---p 00000000 00:00 0
7f8ca0000000-7f8ca0002000 r--s 00008000 fd:00 135583 /usr/share/elasticsearch/lib/joda-convert-1.2.jar
7f8ca0002000-7f8ca0008000 r--s 0003c000 fd:00 138846 /usr/share/elasticsearch/lib/snakeyaml-1.15.jar
7f8ca0008000-7f8ca000c000 r--s 00027000 fd:00 138837 /usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar
7f8ca000c000-7f8ca0010000 r--s 0002d000 fd:00 138842 /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar
7f8ca0010000-7f8ca0015000 r--s 00038000 fd:00 138843 /usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar
7f8ca0015000-7f8ca001a000 r--s 0003b000 fd:00 138841 /usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar
7f8ca001a000-7f8ca026b000 rw-p 00000000 00:00 0
7f8ca026b000-7f8ca026c000 ---p 00000000 00:00 0
7f8ca026c000-7f8ca036c000 rw-p 00000000 00:00 0 [stack:19941]
7f8ca036c000-7f8ca036d000 ---p 00000000 00:00 0
7f8ca036d000-7f8ca0477000 rw-p 00000000 00:00 0 [stack:19940]
7f8ca0477000-7f8ca082d000 ---p 00000000 00:00 0
7f8ca082d000-7f8ca0847000 r-xp 00000000 fd:00 135266 /usr/java/jre1.8.0_45/lib/amd64/libzip.so
7f8ca0847000-7f8ca0a47000 ---p 0001a000 fd:00 135266 /usr/java/jre1.8.0_45/lib/amd64/libzip.so
7f8ca0a47000-7f8ca0a48000 rw-p 0001a000 fd:00 135266 /usr/java/jre1.8.0_45/lib/amd64/libzip.so
7f8ca0a48000-7f8ca0a54000 r-xp 00000000 fd:00 4084 /usr/lib64/libnss_files-2.17.so
7f8ca0a54000-7f8ca0c53000 ---p 0000c000 fd:00 4084 /usr/lib64/libnss_files-2.17.so
7f8ca0c53000-7f8ca0c54000 r--p 0000b000 fd:00 4084 /usr/lib64/libnss_files-2.17.so
7f8ca0c54000-7f8ca0c55000 rw-p 0000c000 fd:00 4084 /usr/lib64/libnss_files-2.17.so
7f8ca0c55000-7f8ca0c5b000 rw-p 00000000 00:00 0
7f8ca0c5b000-7f8ca0c85000 r-xp 00000000 fd:00 135231 /usr/java/jre1.8.0_45/lib/amd64/libjava.so
7f8ca0c85000-7f8ca0e85000 ---p 0002a000 fd:00 135231 /usr/java/jre1.8.0_45/lib/amd64/libjava.so
7f8ca0e85000-7f8ca0e87000 rw-p 0002a000 fd:00 135231 /usr/java/jre1.8.0_45/lib/amd64/libjava.so
7f8ca0e87000-7f8ca0e94000 r-xp 00000000 fd:00 135265 /usr/java/jre1.8.0_45/lib/amd64/libverify.so
7f8ca0e94000-7f8ca1094000 ---p 0000d000 fd:00 135265 /usr/java/jre1.8.0_45/lib/amd64/libverify.so
7f8ca1094000-7f8ca1096000 rw-p 0000d000 fd:00 135265 /usr/java/jre1.8.0_45/lib/amd64/libverify.so
7f8ca1096000-7f8ca109d000 r-xp 00000000 fd:00 4096 /usr/lib64/librt-2.17.so
7f8ca109d000-7f8ca129c000 ---p 00007000 fd:00 4096 /usr/lib64/librt-2.17.so
7f8ca129c000-7f8ca129d000 r--p 00006000 fd:00 4096 /usr/lib64/librt-2.17.so
7f8ca129d000-7f8ca129e000 rw-p 00007000 fd:00 4096 /usr/lib64/librt-2.17.so
7f8ca129e000-7f8ca139f000 r-xp 00000000 fd:00 4074 /usr/lib64/libm-2.17.so
7f8ca139f000-7f8ca159e000 ---p 00101000 fd:00 4074 /usr/lib64/libm-2.17.so
7f8ca159e000-7f8ca159f000 r--p 00100000 fd:00 4074 /usr/lib64/libm-2.17.so
7f8ca159f000-7f8ca15a0000 rw-p 00101000 fd:00 4074 /usr/lib64/libm-2.17.so
7f8ca15a0000-7f8ca224e000 r-xp 00000000 fd:00 135270 /usr/java/jre1.8.0_45/lib/amd64/server/libjvm.so
7f8ca224e000-7f8ca244d000 ---p 00cae000 fd:00 135270 /usr/java/jre1.8.0_45/lib/amd64/server/libjvm.so
7f8ca244d000-7f8ca2524000 rw-p 00cad000 fd:00 135270 /usr/java/jre1.8.0_45/lib/amd64/server/libjvm.so
7f8ca2524000-7f8ca2568000 rw-p 00000000 00:00 0
7f8ca2568000-7f8ca271e000 r-xp 00000000 fd:00 4066 /usr/lib64/libc-2.17.so
7f8ca271e000-7f8ca291e000 ---p 001b6000 fd:00 4066 /usr/lib64/libc-2.17.so
7f8ca291e000-7f8ca2922000 r--p 001b6000 fd:00 4066 /usr/lib64/libc-2.17.so
7f8ca2922000-7f8ca2924000 rw-p 001ba000 fd:00 4066 /usr/lib64/libc-2.17.so
7f8ca2924000-7f8ca2929000 rw-p 00000000 00:00 0
7f8ca2929000-7f8ca292c000 r-xp 00000000 fd:00 4072 /usr/lib64/libdl-2.17.so
7f8ca292c000-7f8ca2b2b000 ---p 00003000 fd:00 4072 /usr/lib64/libdl-2.17.so
7f8ca2b2b000-7f8ca2b2c000 r--p 00002000 fd:00 4072 /usr/lib64/libdl-2.17.so
7f8ca2b2c000-7f8ca2b2d000 rw-p 00003000 fd:00 4072 /usr/lib64/libdl-2.17.so
7f8ca2b2d000-7f8ca2b42000 r-xp 00000000 fd:00 135209 /usr/java/jre1.8.0_45/lib/amd64/jli/libjli.so
7f8ca2b42000-7f8ca2d42000 ---p 00015000 fd:00 135209 /usr/java/jre1.8.0_45/lib/amd64/jli/libjli.so
7f8ca2d42000-7f8ca2d43000 rw-p 00015000 fd:00 135209 /usr/java/jre1.8.0_45/lib/amd64/jli/libjli.so
7f8ca2d43000-7f8ca2d59000 r-xp 00000000 fd:00 4092 /usr/lib64/libpthread-2.17.so
7f8ca2d59000-7f8ca2f59000 ---p 00016000 fd:00 4092 /usr/lib64/libpthread-2.17.so
7f8ca2f59000-7f8ca2f5a000 r--p 00016000 fd:00 4092 /usr/lib64/libpthread-2.17.so
7f8ca2f5a000-7f8ca2f5b000 rw-p 00017000 fd:00 4092 /usr/lib64/libpthread-2.17.so
7f8ca2f5b000-7f8ca2f5f000 rw-p 00000000 00:00 0
7f8ca2f5f000-7f8ca2f80000 r-xp 00000000 fd:00 4059 /usr/lib64/ld-2.17.so
7f8ca2f80000-7f8ca2f82000 r--s 00011000 fd:00 138825 /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.2.jar
7f8ca2f82000-7f8ca2f84000 r--s 00007000 fd:00 138836 /usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar
7f8ca2f84000-7f8ca2f85000 r--s 00001000 fd:00 138845 /usr/share/elasticsearch/lib/securesm-1.0.jar
7f8ca2f85000-7f8ca2f89000 r--s 000dc000 fd:00 138827 /usr/share/elasticsearch/lib/jna-4.1.0.jar
7f8ca2f89000-7f8ca2f93000 r--s 00059000 fd:00 138839 /usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar
7f8ca2f93000-7f8ca2f95000 r--s 00012000 fd:00 138819 /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar
7f8ca2f95000-7f8ca2f96000 r--s 0000c000 fd:00 135578 /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar
7f8ca2f96000-7f8ca2f9d000 r--s 00037000 fd:00 138838 /usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar
7f8ca2f9d000-7f8ca2fa5000 r--s 00066000 fd:00 138817 /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar
7f8ca2fa5000-7f8ca2fa8000 r--s 00022000 fd:00 138847 /usr/share/elasticsearch/lib/spatial4j-0.5.jar
7f8ca2fa8000-7f8ca2fac000 r--s 0003c000 fd:00 138823 /usr/share/elasticsearch/lib/jackson-core-2.6.2.jar
7f8ca2fac000-7f8ca2fbc000 r--s 00088000 fd:00 138828 /usr/share/elasticsearch/lib/joda-time-2.8.2.jar
7f8ca2fbc000-7f8ca2fc5000 r--s 0006f000 fd:00 135584 /usr/share/elasticsearch/lib/log4j-1.2.17.jar
7f8ca2fc5000-7f8ca3069000 rw-p 00000000 00:00 0
7f8ca3069000-7f8ca3071000 rw-s 00000000 fd:01 393218 /tmp/hsperfdata_elasticsearch/19921
7f8ca3071000-7f8ca3074000 ---p 00000000 00:00 0
7f8ca3074000-7f8ca3176000 rw-p 00000000 00:00 0 [stack:19939]
7f8ca3176000-7f8ca3178000 r--s 0000b000 fd:00 138848 /usr/share/elasticsearch/lib/t-digest-3.0.jar
7f8ca3178000-7f8ca317c000 r--s 0001e000 fd:00 138835 /usr/share/elasticsearch/lib/lucene-join-5.5.0.jar
7f8ca317c000-7f8ca317d000 r--s 0000b000 fd:00 138824 /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.2.jar
7f8ca317d000-7f8ca317e000 rw-p 00000000 00:00 0
7f8ca317e000-7f8ca317f000 r--p 00000000 00:00 0
7f8ca317f000-7f8ca3180000 rw-p 00000000 00:00 0
7f8ca3180000-7f8ca3181000 r--p 00021000 fd:00 4059 /usr/lib64/ld-2.17.so
7f8ca3181000-7f8ca3182000 rw-p 00022000 fd:00 4059 /usr/lib64/ld-2.17.so
7f8ca3182000-7f8ca3183000 rw-p 00000000 00:00 0
7ffcb0189000-7ffcb01aa000 rw-p 00000000 00:00 0 [stack]
7ffcb01fc000-7ffcb01fe000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
VM Arguments:
jvm_args: -Xms4g -Xmx4g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.debug_load.jna=true -Djna.debug_load=true -Djava.io.tmpdir=/export/apps/elasticsearch/tmp -Djna.tmpdir=/export/apps/elasticsearch/tmp -Des.path.home=/usr/share/elasticsearch
java_command: org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
java_class_path (initial): /usr/share/elasticsearch/lib/elasticsearch-2.3.2.jar:/usr/share/elasticsearch/lib/elasticsearch-2.3.2.jar:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.2.jar:/usr/share/elasticsearch/lib/log4j-1.2.17.jar:/usr/share/elasticsearch/lib/lucene-join-5.5.0.jar:/usr/share/elasticsearch/lib/joda-time-2.8.2.jar:/usr/share/elasticsearch/lib/jackson-core-2.6.2.jar:/usr/share/elasticsearch/lib/t-digest-3.0.jar:/usr/share/elasticsearch/lib/spatial4j-0.5.jar:/usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar:/usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar:/usr/share/elasticsearch/lib/commons-cli-1.3.1.jar:/usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar:/usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar:/usr/share/elasticsearch/lib/jna-4.1.0.jar:/usr/share/elasticsearch/lib/securesm-1.0.jar:/usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar:/usr/share/elasticsearch/lib/guava-18.0.jar:/usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.2.jar:/usr/share/elasticsearch/lib/netty-3.10.5.Final.jar:/usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar:/usr/share/elasticsearch/lib/snakeyaml-1.15.jar:/usr/share/elasticsearch/lib/joda-convert-1.2.jar:/usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar:/usr/share/elasticsearch/lib/jsr166e-1.1.0.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar:/usr/share/elasticsearch/lib/hppc-0.7.1.jar:/usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar:/usr/share/elasticsearch/lib/jts-1.13.jar:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.2.jar:/usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar:/usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-core-5.5.0.jar:/usr/share/elasticsearch/lib/c
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
SHELL=/sbin/nologin
Signal Handlers:
SIGSEGV: [libjvm.so+0xaad1e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xaad1e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x90b450], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x90ca90], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x90dde0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
--------------- S Y S T E M ---------------
OS:CentOS Linux release 7.2.1511 (Core)
uname:Linux 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 31854, NOFILE 65535, AS infinity
load average:0.25 0.43 0.29
/proc/meminfo:
MemTotal: 7912632 kB
MemFree: 1366836 kB
MemAvailable: 7015480 kB
Buffers: 116304 kB
Cached: 5265648 kB
SwapCached: 4928 kB
Active: 1525396 kB
Inactive: 4327224 kB
Active(anon): 270016 kB
Inactive(anon): 248668 kB
Active(file): 1255380 kB
Inactive(file): 4078556 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 999420 kB
SwapFree: 935612 kB
Dirty: 72 kB
Writeback: 0 kB
AnonPages: 430132 kB
Mapped: 47044 kB
Shmem: 47952 kB
Slab: 607060 kB
SReclaimable: 568100 kB
SUnreclaim: 38960 kB
KernelStack: 3504 kB
PageTables: 8760 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4955736 kB
Committed_AS: 5185924 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 23160 kB
VmallocChunk: 34359710460 kB
HardwareCorrupted: 0 kB
AnonHugePages: 354304 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 43008 kB
DirectMap2M: 8345600 kB
CPU:total 2 (1 cores per cpu, 2 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, ht, tsc, bmi1, bmi2
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping : 2
microcode : 0x2b
cpu MHz : 2400.100
cache size : 30720 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bogomips : 4800.20
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping : 2
microcode : 0x2b
cpu MHz : 2400.100
cache size : 30720 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bogomips : 4800.20
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 7912632k(1366836k free), swap 999420k(935612k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.45-b02) for linux-amd64 JRE (1.8.0_45-b14), built on Apr 10 2015 10:07:45 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Mon May 23 15:16:48 2016
elapsed time: 0 seconds (0d 0h 0m 0s)I've personally tried with setenforce 0 but nothing. what is interesting that data dire and tmp points to the same directory and while data dir created and used without problem tmp cannot be for some reason
fxh
commented
8 years ago [root@storagenode01 ~]# cat /etc/sysconfig/elasticsearch | grep ES_JAVA_OPTS
ES_JAVA_OPTS="-Djna.debug_load.jna=true -Djna.debug_load=true -Djna.tmpdir=/usr/share/elasticsearch/tmp -Djava.io.tmpdir=/usr/share/elasticsearch/tmp"yields
[root@storagenode01 ~]# service elasticsearch restart
Stopping elasticsearch: [2016-05-23 16:07:35,615][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] stopping ...
[2016-05-23 16:07:35,776][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] stopped
[2016-05-23 16:07:35,776][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] closing ...
[2016-05-23 16:07:35,780][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] closed
[ OK ]
Starting elasticsearch: /usr/share/elasticsearch/bin/elasticsearch: line 155: 2037 Aborted (core dumped) exec "$JAVA" $JAVA_OPTS $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH" org.elasticsearch.bootstrap.Elasticsearch start "$@" 0>&-
[FAILED]
[root@storagenode01 ~]#furthermore as requested:
[root@storagenode01 ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: enforcing
Mode from config file: enforcing
Policy version: 24
Policy from config file: targeted[root@storagenode01 ~]# getsebool -a
abrt_anon_write --> off
abrt_handle_event --> off
allow_console_login --> on
allow_cvs_read_shadow --> off
allow_daemons_dump_core --> on
allow_daemons_use_tcp_wrapper --> off
allow_daemons_use_tty --> on
allow_domain_fd_use --> on
allow_execheap --> off
allow_execmem --> on
allow_execmod --> on
allow_execstack --> on
allow_ftpd_anon_write --> off
allow_ftpd_full_access --> off
allow_ftpd_use_cifs --> off
allow_ftpd_use_nfs --> off
allow_gssd_read_tmp --> on
allow_guest_exec_content --> off
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
allow_java_execstack --> off
allow_kerberos --> on
allow_mount_anyfile --> on
allow_mplayer_execstack --> off
allow_nsplugin_execmem --> on
allow_polyinstantiation --> off
allow_postfix_local_write_mail_spool --> on
allow_ptrace --> off
allow_rsync_anon_write --> off
allow_saslauthd_read_shadow --> off
allow_smbd_anon_write --> off
allow_ssh_keysign --> off
allow_staff_exec_content --> on
allow_sysadm_exec_content --> on
allow_unconfined_nsplugin_transition --> off
allow_user_exec_content --> on
allow_user_mysql_connect --> off
allow_user_postgresql_connect --> off
allow_write_xshm --> off
allow_xguest_exec_content --> off
allow_xserver_execmem --> off
allow_ypbind --> off
allow_zebra_write_config --> on
antivirus_can_scan_system --> off
antivirus_use_jit --> off
authlogin_radius --> off
authlogin_shadow --> off
awstats_purge_apache_log_files --> off
bacula_use_nfs --> off
bacula_use_samba --> off
cdrecord_read_content --> off
cluster_can_network_connect --> off
cluster_manage_all_files --> on
cluster_use_execmem --> off
cobbler_anon_write --> off
cobbler_can_network_connect --> off
cobbler_use_cifs --> off
cobbler_use_nfs --> off
collectd_tcp_network_connect --> off
condor_domain_can_network_connect --> off
cron_can_relabel --> off
daemons_enable_cluster_mode --> on
dhcpc_exec_iptables --> off
domain_kernel_load_modules --> off
exim_can_connect_db --> off
exim_manage_user_files --> off
exim_read_user_files --> off
fcron_crond --> off
fenced_can_network_connect --> off
fenced_can_ssh --> off
fips_mode --> on
ftp_home_dir --> off
ftpd_connect_db --> off
ftpd_use_fusefs --> off
ftpd_use_passive_mode --> off
git_cgi_enable_homedirs --> off
git_cgi_use_cifs --> off
git_cgi_use_nfs --> off
git_session_bind_all_unreserved_ports --> off
git_session_users --> off
git_system_enable_homedirs --> off
git_system_use_cifs --> off
git_system_use_nfs --> off
global_ssp --> off
gluster_anon_write --> off
gluster_export_all_ro --> off
gluster_export_all_rw --> on
gpg_agent_env_file --> off
gpg_web_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> on
httpd_dbus_sssd --> off
httpd_enable_cgi --> on
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_manage_ipa --> off
httpd_read_user_content --> off
httpd_run_preupgrade --> off
httpd_run_stickshift --> off
httpd_serve_cobbler_files --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_tty_comm --> on
httpd_unified --> on
httpd_use_cifs --> off
httpd_use_fusefs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
httpd_use_openstack --> off
httpd_verify_dns --> off
icecast_connect_any --> off
init_upstart --> on
irssi_use_full_network --> off
kdumpgui_run_bootloader --> off
logging_syslog_can_read_tmp --> off
logging_syslogd_can_sendmail --> off
logging_syslogd_run_nagios_plugins --> off
logging_syslogd_use_tty --> on
logrotate_use_nfs --> off
lsmd_plugin_connect_any --> off
mcelog_foreground --> off
mmap_low_allowed --> off
mozilla_read_content --> off
mysql_connect_any --> off
nagios_run_sudo --> off
named_bind_http_port --> off
named_write_master_zones --> off
ncftool_read_user_content --> off
nscd_use_shm --> on
nsplugin_can_network --> on
openshift_use_nfs --> off
openvpn_enable_homedirs --> on
openvpn_run_unconfined --> off
pcp_bind_all_unreserved_ports --> off
piranha_lvs_can_network_connect --> off
postgresql_can_rsync --> off
pppd_can_insmod --> off
pppd_for_user --> off
privoxy_connect_any --> on
puppet_manage_all_files --> off
puppetmaster_use_db --> off
qemu_full_network --> on
qemu_use_cifs --> on
qemu_use_comm --> off
qemu_use_nfs --> on
qemu_use_usb --> on
racoon_read_shadow --> off
rsync_client --> off
rsync_export_all_ro --> off
rsync_server --> off
rsync_use_cifs --> off
rsync_use_nfs --> off
samba_create_home_dirs --> off
samba_domain_controller --> off
samba_enable_home_dirs --> off
samba_export_all_ro --> off
samba_export_all_rw --> off
samba_load_libgfapi --> off
samba_portmapper --> off
samba_run_unconfined --> off
samba_share_fusefs --> off
samba_share_nfs --> off
sanlock_use_fusefs --> off
sanlock_use_nfs --> off
sanlock_use_samba --> off
secure_mode --> off
secure_mode_insmod --> off
secure_mode_policyload --> off
sepgsql_enable_users_ddl --> on
sepgsql_unconfined_dbadm --> on
sge_domain_can_network_connect --> off
sge_use_nfs --> off
smartmon_3ware --> off
spamassassin_can_network --> off
spamd_enable_home_dirs --> on
squid_connect_any --> on
squid_use_tproxy --> off
ssh_chroot_full_access --> off
ssh_chroot_manage_apache_content --> off
ssh_chroot_rw_homedirs --> off
ssh_sysadm_login --> off
swift_can_network --> off
telepathy_tcp_connect_generic_network_ports --> off
tftp_anon_write --> off
tftp_use_cifs --> off
tftp_use_nfs --> off
tor_bind_all_unreserved_ports --> off
unconfined_login --> on
unconfined_mmap_zero_ignore --> off
unconfined_mozilla_plugin_transition --> off
use_fusefs_home_dirs --> off
use_lpd_server --> off
use_nfs_home_dirs --> on
use_samba_home_dirs --> off
user_direct_dri --> on
user_direct_mouse --> off
user_ping --> on
user_rw_noexattrfile --> on
user_setrlimit --> on
user_tcp_server --> off
user_ttyfile_stat --> off
varnishd_connect_any --> off
vbetool_mmap_zero_ignore --> off
virt_use_comm --> off
virt_use_execmem --> off
virt_use_fusefs --> off
virt_use_nfs --> off
virt_use_samba --> off
virt_use_sanlock --> off
virt_use_sysfs --> on
virt_use_usb --> on
virt_use_xserver --> off
webadm_manage_user_files --> off
webadm_read_user_files --> off
wine_mmap_zero_ignore --> off
xdm_exec_bootloader --> off
xdm_sysadm_login --> off
xen_use_nfs --> off
xguest_connect_network --> on
xguest_mount_media --> on
xguest_use_bluetooth --> on
xserver_object_manager --> off
zabbix_can_network --> off
[root@storagenode01 ~]#I'll disable SELinux and reboot later tonight and let you know the outcome. Thanks for your investigations.
jasontedor
commented
8 years ago Poking around some more, the segmentation fault is happening in libffi. But JNA includes their own libffi instead of dynamically linking to the system libffi. There are some changes in the offending method (ffi_prep_closure_loc) and related methods in their integrated libffi from 4.1.0 (the version of JNA that we ship with) to 4.2.2 (the latest version of JNA). Can you also try with jna-4.2.2? Just remove the jna-4.1.0.jar in the Elasticsearch lib folder and add the jna-4.2.2.jar.
fxh
commented
8 years ago same result with jna-4.2.2.jar:
[root@storagenode01 lib]# service elasticsearch restart
Stopping elasticsearch: [2016-05-23 19:22:08,554][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] stopping ...
[2016-05-23 19:22:08,626][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] stopped
[2016-05-23 19:22:08,627][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] closing ...
[2016-05-23 19:22:08,630][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] closed
[ OK ]
Starting elasticsearch: /usr/share/elasticsearch/bin/elasticsearch: line 155: 4395 Aborted (core dumped) exec "$JAVA" $JAVA_OPTS $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH" org.elasticsearch.bootstrap.Elasticsearch start "$@" 0>&-
[FAILED]
[root@storagenode01 lib]#regarding SELinux, I disabled it (via /etc/selinux/config and reboot):
[root@storagenode01 ~]# sestatus -v
SELinux status: disabled
[root@storagenode01 ~]#and this seems to work:
[root@storagenode01 ~]# service elasticsearch restart
Stopping elasticsearch: [2016-05-23 19:36:04,135][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] stopping ...
[2016-05-23 19:36:04,233][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] stopped
[2016-05-23 19:36:04,233][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] closing ...
[2016-05-23 19:36:04,238][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] closed
[ OK ]
Starting elasticsearch: [2016-05-23 19:36:05,629][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-05-23 19:36:06,817][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] version[2.2.0], pid[17611], build[3eeb1f1/2016-05-10T14:33:28Z]
[2016-05-23 19:36:06,817][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] initializing ...
[2016-05-23 19:36:07,282][INFO ][plugins ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] modules [lang-groovy, lang-expression], plugins [analysis-smartcn, analysis-kuromoji, delete-by-query, head, analysis-icu], sites [head]
[2016-05-23 19:36:07,297][INFO ][env ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] using [1] data paths, mounts [[/var (/dev/mapper/vg_root-var)]], net usable_space [456.9gb], net total_space [481.9gb], spins? [possibly], types [ext4]
[2016-05-23 19:36:07,297][INFO ][env ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] heap size [247.5mb], compressed ordinary object pointers [true]
[2016-05-23 19:36:08,305][INFO ][script ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] compiling script file [/etc/elasticsearch/scripts/has-multi-value.groovy]
[2016-05-23 19:36:08,923][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] initialized
[2016-05-23 19:36:08,923][INFO ][node ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] starting ...
[2016-05-23 19:36:08,963][INFO ][transport ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {172.16.212.130:9300}, {[fe80::20c:29ff:fe7c:ec79]:9300}
[2016-05-23 19:36:08,970][INFO ][discovery ] [squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7-node01] squirro-ce793b21-a7fd-400f-8b22-42ffbf5d6fd7/TzKoQtmaTKCneN6zKBlSEg
[ OK ]
[root@storagenode01 ~]#Ah, so if I'm reading you correctly, it appears my hunch that it is SELinux is correct! Okay, I'll see if I can take your SELinux configuration somewhere then. Thanks for reporting back.
jasontedor
commented
8 years ago @fxh Can you share the output of cat /proc/mounts and ls -alZ /usr/share/elasticsearch?
fxh
commented
8 years ago [felix@storagenode01 ~]$ cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,seclabel,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,relatime,size=1946820k,nr_inodes=486705,mode=755 0 0
devpts /dev/pts devpts rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/vg_root-root / ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /selinux selinuxfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,seclabel,relatime,size=1946820k,nr_inodes=486705,mode=755 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vg_root-home /home ext4 rw,seclabel,nodev,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vg_root-tmp /tmp ext4 rw,seclabel,nosuid,nodev,noexec,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vg_root-var /var ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vg_root-log /var/log ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vg_root-audit /var/log/audit ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/mapper/vg_root-tmp /var/tmp ext4 rw,seclabel,nosuid,nodev,noexec,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0and
[felix@storagenode01 ~]$ ls -alZ /usr/share/elasticsearch
drwxrwxr-x. root root system_u:object_r:usr_t:s0 .
drwxr-xr-x. root root system_u:object_r:usr_t:s0 ..
-rw-------. elastic elastic system_u:object_r:usr_t:s0 .bash_history
drwxrwxr-x. root root system_u:object_r:bin_t:s0 bin
drwxr-xr-x. root root system_u:object_r:usr_t:s0 config
drwxr-xr-x. root root system_u:object_r:usr_t:s0 data
drwxrwxr-x. root root system_u:object_r:lib_t:s0 lib
-rw-r--r--. root root system_u:object_r:usr_t:s0 LICENSE.txt
drwxr-xr-x. root root system_u:object_r:usr_t:s0 logs
drwxrwxr-x. root root system_u:object_r:usr_t:s0 modules
-rw-r--r--. root root system_u:object_r:usr_t:s0 NOTICE.txt
drwxrwxr-x. root root system_u:object_r:usr_t:s0 plugins
-rw-r--r--. root root system_u:object_r:usr_t:s0 README.textile
drwxr-xr-x. elastic elastic system_u:object_r:usr_t:s0 tmp
[felix@storagenode01 ~]$
olenm
commented
8 years ago I have noticed I am able to install and successfully run ES 2.1.1 on a CoS 7.2 system with SELinux enabled and noexec on /tmp (without using java opts -Djna.tmpdir= nor -Djava.io.tmpdir=
2.1.2 and above (tested on the latest 2.3.3) crash with a similar dump mentioned by bubo77, both with and without the tmpdirs specified.
Also using: jdk1.8.0_45-1.8.0_45-fcs.x86_64
jasontedor
commented
8 years ago @olenm And if you disable SELinux, does Elasticsearch startup successfully?
olenm
commented
8 years ago with SELinux disabled it does not startup successfully (just hit the reboot). forgot to mention in my first post (and more importantly) I have noexec on /tmp
EDIT: and I have verified removing the noexec on /tmp does NOT resolve the issue for 2.3.3 (if needed I can test this again on 2.1.2) - it takes a while longer to fail
jasontedor
commented
8 years ago @olenm It's never going to work if /tmp is noexec, JNA loads native libraries from there. This is the reason that jna.tmpdir exists. Can you try with SELinux disabled and either remove noexec from the mount options for /tmp, or point jna.tmpdir somewhere else?
olenm
commented
8 years ago ran a few tests:
for ES 2.1.1 with noexec on /tmp (selinux still disabled): using no jna.tmpdir I do not see any crashing with my current configs
with ES 2.3.3 with noexec on /tmp (selinux disabled):
jna.tmpdir to a folder with chmod 0777 -Djna.tmpdir=/test within /ect/sysconfig/easticsearch 's ES_JAVA_OPTSI am seeing an entry I have not before through journalctl: plugin cloud-aws is the culprit; yanking the cloud-aws and removing its section from the config for it did yield in a successful start of ES 2.3.3
jasontedor
commented
8 years ago I am seeing an entry I have not before through journalctl: plugin cloud-aws is the culprit
@olenm Can you share the log message?
olenm
commented
8 years ago Exception in thread "main" java.lang.IllegalArgumentException: Plugin [cloud-aws] is incompatible with Elasticsearch [2.3.3]. Was designed for version [2.1.1]
(and before it appears to be overlooked with the combination the tmp noexec)
jasontedor
commented
8 years ago
Exception in thread "main" java.lang.IllegalArgumentException: Plugin [cloud-aws] is incompatible with Elasticsearch [2.3.3]. Was designed for version [2.1.1](and before it appears to be overlooked with the combination the tmp noexec)
Thanks, that's unrelated to the issue here. The plugins are not loaded until node during startup, which happens after bootstrap where JNA is loaded.
jasontedor
commented
8 years ago for ES 2.1.1 with noexec on /tmp (selinux still disabled): using no
jna.tmpdirI do not see any crashing with my current configs
@olenm Right, crashing is not the expected behavior with noexec on /tmp, just that JNA will be useless. You should see log messages saying that unable to load JNA native support library, native methods will be disabled and a non-fatal UnsatisfiedLinkError exception in the logs.
Both of your experiments seem to continue to support the earlier hypothesis that there is a poor interaction between JNA and SELinux occurring here.
olenm
commented
8 years ago Yeah sorry about the user error (with the proper version of that plugin installed and back to selinux + noexec on /tmp I did see the same issue related above and no additional plugin errors).
EDIT: 2.1.1 did not log any issues in journalctl - but did log that the JNA would not be loaded in the flat log file
olenm
commented
8 years ago @jasontedor I've played a bit more with this; only using ES 2.1.1 on CoS7.2 .. I'm editing this comment making it a bit cleaner (or may post again)
jasontedor
commented
8 years ago @olenm I do not think that it's just noexec:
[vagrant@localhost ~]$ cat /etc/fstab | grep tmp
tmpfs /tmp tmpfs defaults,nosuid,noexec,nodev 0 0
[vagrant@localhost ~]$ findmnt --target /tmp
TARGET SOURCE FSTYPE OPTIONS
/tmp tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,seclabel
[vagrant@localhost ~]$ /usr/share/elasticsearch/bin/elasticsearch --path.conf config --path.logs logs --path.data data
[2016-04-27 09:45:45,070][WARN ][bootstrap ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna-226895185/jna7321475784156807333.tmp: /tmp/jna-226895185/jna7321475784156807333.tmp: failed to map segment from shared object
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1938)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1821)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761)
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736)
at com.sun.jna.Native.<clinit>(Native.java:131)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45)
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:89)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
[2016-04-27 09:45:45,079][WARN ][bootstrap ] cannot check if running as root because JNA is not available
[2016-04-27 09:45:45,080][WARN ][bootstrap ] cannot install syscall filters because JNA is not available
[2016-04-27 09:45:45,080][WARN ][bootstrap ] cannot register console handler because JNA is not available
[2016-04-27 09:45:45,371][INFO ][node ] [Hardcore] version[2.1.1], pid[700], build[40e2c53/2015-12-15T13:05:55Z]
[2016-04-27 09:45:45,372][INFO ][node ] [Hardcore] initializing ...
[2016-04-27 09:45:45,454][INFO ][plugins ] [Hardcore] loaded [], sites []
[2016-04-27 09:45:45,502][INFO ][env ] [Hardcore] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.9gb], net total_space [39.2gb], spins? [possibly], types [ext4]
[2016-04-27 09:45:47,923][INFO ][node ] [Hardcore] initialized
[2016-04-27 09:45:47,923][INFO ][node ] [Hardcore] starting ...
[2016-04-27 09:45:48,006][INFO ][transport ] [Hardcore] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2016-04-27 09:45:48,023][INFO ][discovery ] [Hardcore] elasticsearch/D1wQ0Y29QoSoLibbluvM1Q
[2016-04-27 09:45:51,095][INFO ][cluster.service ] [Hardcore] new_master {Hardcore}{D1wQ0Y29QoSoLibbluvM1Q}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-04-27 09:45:51,120][INFO ][http ] [Hardcore] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2016-04-27 09:45:51,121][INFO ][node ] [Hardcore] started
[2016-04-27 09:45:51,164][INFO ][gateway ] [Hardcore] recovered [0] indices into cluster_state
^C[2016-04-27 09:45:53,253][INFO ][node ] [Hardcore] stopping ...
[2016-04-27 09:45:53,272][INFO ][node ] [Hardcore] stopped
[2016-04-27 09:45:53,272][INFO ][node ] [Hardcore] closing ...
[2016-04-27 09:45:53,283][INFO ][node ] [Hardcore] closedfstab tmp line:
none /dev/shm tmpfs defaults,nosuid,noexec,nodev 0 0
Note that /dev/shm is not the same as /tmp.
the ES log warnings/errors and /tmp/hs_dumpfiles were not resultant of the selinux policy; only when enabling noexec on /tmp did the dumpfile occur.
I expect log messages as above when /tmp is mounted with noexec.
with selinux enabled (and I believe disabled/permissive as well), noexec disable, and excluding
jna.tmpdir, the ES logfile would output/tmp/jna--<##>/jna<##>.tmp: failed to map segment from shared object: Operation not permitted
That looks like SELinux. Are you sure that you got a message stating "failed to map segment from shared object: Operation not permitted" with SELinux disabled or in permissive mode? Note that you you'll have to reboot if you switched between SELinux being disabled and enabled.
olenm
commented
8 years ago That is exactly what I get for only grep'ing and sed'ing the fstab for noexec - when I should have nuked the systemd tmp.mount tmpfs service (I never seem to remember this until a bit of pain occurs). Essentially massive user-error on that noexec part (and the plugin part from earlier) (sorry about that!)
I went overkill on cleaning up my scenarios (and manually testing them vs any automation tools) - surprisingly and happily I encountered the crash-dump once (and only with ES 2.1.1))
The same conditions that caused the crash in ES 2.1.1 did not cause a crash in ES 2.3.3 - so I am guessing (since the thread started with 2.2) that all is well when using the tmpdir option and all plugins load properly.
As for the scenarios, here they are (they reiterate what you have said on the expected noexec messages and I have double checked on that Operation not permitted message):
Constants in all scenarios run:
LimitMEMLOCK=infinityrelative fstab tmp lines: 1) /tmp /var/tmp none bind 0 0 2) none /dev/shm tmpfs defaults,nosuid,noexec,nodev 0 0
case 1:
jna.tmpdir not set/tmp/jna--<##>/jna<##>.tmp: failed to map segment from shared object: Operation not permittedcase 2:
jna.tmpdir set to /test (where chmod 0750 test and owned by ES user/group)case 3:
jna.tmpdir not setcase 4:
jna.tmpdir set to /test (where chmod 0750 test and owned by ES user/group)case 5:
jna.tmpdir set to /test (where chmod 0750 test and owned by ES user/group)case 6:
jna.tmpdir not set/tmp/jna--<##>/jna<##>.tmp: failed to map segment from shared object: Operation not permittedcase 7:
jna.tmpdir set to /test (where chmod 0750 test and owned by ES user/group)case 8:
jna.tmpdir not setcase 9:
jna.tmpdir not set/tmp/jna--<##>/jna<##>.tmp: failed to map segment from shared object: Operation not permittedcase 10:
jna.tmpdir set to /test (where chmod 0750 test and owned by ES user/group)case 11:
jna.tmpdir set to /test (where chmod 0750 test and owned by ES user/group)Thanks for bearing with some user-error - and hope anybody else that searches this issue finds this info helpful
olenm
commented
8 years ago @jasontedor after my "success" last night; I could not reproduce the "successful" case for ES 2.3.3 this morning on a fresh system so something contaminated my instance (again!). But now will not be messing around with less than 2.3.3.
Using different CoS7 systems (built from the same template and running side by side), I had success getting ES 2.3.3 to started in the first 2 below and failed on the 3rd:
SYS1:
SYS2:
SYS3:
type=ANOM_ABEND msg=audit(1465327422.704:85): auid=4294967295 uid=993 gid=991 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=2498 comm="java" reason="memory violation" sig=6Couldn't tell you why I did not post the memory violation earlier (saw it last week even) - any other info that I can pull from the system?
jasontedor
commented
8 years ago I feel pretty confident that the issues here are due to poor interaction between SELinux and JNA. The exact circumstances are unclear to me, but I do not think that there is an Elasticsearch issue here to resolve. Please feel free to reopen if you have evidence to the contrary.
FlorianHeigl
commented
8 years ago In https://github.com/elastic/elasticsearch/issues/18406 it was said that so far this wasn't really reproducible. If anyone picks this up within the next months, I can clone make a clone of an empty system/cluster and give access to to that.
vineet01
commented
7 years ago If someone else is facing the same ..
I got the same error as OP with:
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007f9800b8d2ca, pid=21823, tid=0x00007f98f541f700
JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.0_112-b15) Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode linux-amd64 compressed oops) Problematic frame: C [jna1247360738620499207.tmp+0x122ca] ffi_prep_closure_loc+0x1a
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
If you would like to submit a bug report, please visit: http://bugreport.java.com/bugreport/crash.jsp The crash happened outside the Java Virtual Machine in native code. See problematic frame for where to report the bug.
The reason of failure was that the system user with which ES was running did not have an existing home directory.
Once I created the homedir, it started as expected. In the home directory, a directory ".oracle_jre_usage" is created which has a fd53b05c83802a42.timestamp file.
More about this file - https://community.oracle.com/thread/3783686.
Elasticsearch version 2.2
JVM version: 1.7.0_75
OS version: CentOS release 6.7
Description of the problem including expected versus actual behavior: Starting elasticsearch fails to start if the following is added to the /etc/sysconfig/elasticsearch file:
With this line commented out, ES starts but JNA is disabled as the default /tmp/ directory is mounted with noexec. If /tmp/ is mounted without noexec all works as expected. /usr/share/elasticsearch/tmp exists, belongs to the right user/group and has permissions 755.
Steps to reproduce:
Provide logs (if relevant):