[dev.icinga.com #10960] (icinga), uid 183: exited on signal 11 (core dumped) - Githubissues

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10960

Created by leo9641 on 2016-01-13 09:25:14 +00:00

Assignee: (none) Status: New Target Version: (none) Last Update: 2016-01-18 07:48:17 +00:00 (in Redmine)

Icinga Version: 1.13.3
OS Version: FreeBSD 10.2-RELEASE

Hi icinga team!

I upgraded my OS from FreeBSD 9.3 to FreeBSD 10.2 and rebuild icinga-1.13.3 using clang (default FreeBSD version 3.4.1). After that I have a troubles:

pid 48866 (icinga), uid 183: exited on signal 11 pid 68322 (icinga), uid 183: exited on signal 11 pid 12185 (icinga), uid 183: exited on signal 11 (core dumped) pid 63934 (icinga), uid 183: exited on signal 11 (core dumped)

...

I examine a core file:

# gdb /usr/local/bin/icinga /var/coredumps/icinga-12185.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `icinga'. Program terminated with signal 11, Segmentation fault. Error while mapping shared library sections: /var/spool/icinga/ramdisk/tmp/icinganebmodV52rnh: No such file or directory. Error while mapping shared library sections: /var/spool/icinga/ramdisk/tmp/icinganebmod3priuW: No such file or directory. Reading symbols from /lib/libm.so.5...done. Loaded symbols for /lib/libm.so.5 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /usr/local/lib/libltdl.so.7...done. Loaded symbols for /usr/local/lib/libltdl.so.7 Reading symbols from /lib/libz.so.6...done. Loaded symbols for /lib/libz.so.6 Reading symbols from /usr/lib/libssl.so.7...done. Loaded symbols for /usr/lib/libssl.so.7 Reading symbols from /lib/libcrypto.so.7...done. Loaded symbols for /lib/libcrypto.so.7 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Error while reading shared library symbols: /var/spool/icinga/ramdisk/tmp/icinganebmodV52rnh: No such file or directory. Reading symbols from /usr/local/lib/libgearman.so.7...done. Loaded symbols for /usr/local/lib/libgearman.so.7 Reading symbols from /usr/lib/libc.so.1...done. Loaded symbols for /usr/lib/libc.so.1 Reading symbols from /lib/libcxxrt.so.1...done. Loaded symbols for /lib/libcxxrt.so.1 Reading symbols from /usr/local/lib/libuuid.so.1...done. Loaded symbols for /usr/local/lib/libuuid.so.1 Reading symbols from /usr/lib/libexecinfo.so.1...done. Loaded symbols for /usr/lib/libexecinfo.so.1 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /usr/lib/libelf.so.1...done. Loaded symbols for /usr/lib/libelf.so.1 Error while reading shared library symbols: /var/spool/icinga/ramdisk/tmp/icinganebmod3priuW: No such file or directory. Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1

#0  0x0000000800b384b6 in pthread_suspend_all_np () from /lib/libthr.so.3

[New Thread 802006400 (LWP 101403/)] (gdb) t [Current thread is 1 (Thread 802006400 (LWP 101403/))] (gdb) bt

#0  0x0000000800b384b6 in pthread_suspend_all_np () from /lib/libthr.so.3
#1  0x0000000800b3b26a in pthread_getspecific () from /lib/libthr.so.3
#2  0x0000000800b3fc96 in __pthread_cxa_finalize () from /lib/libthr.so.3
#3  0x0000000000427b75 in run_check (processed_command=0x8060cf290 "/usr/local/libexec/nagios/check_http -I st13.REMOVE_DOMAIN.com -t 30", checkresult_dbuf=0x7fffffffde10)

at checks.c:219

#4  0x0000000000426e8a in run_async_service_check (svc=0x8055e4880, check_options=0, latency=0.81699999999999995, scheduled_check=1, reschedule_check=1, time_is_valid=0x7fffffffe57c,

preferred_time=0x7fffffffe588) at checks.c:947

#5  0x0000000000425e51 in run_scheduled_service_check (svc=0x8055e4880, check_options=0, latency=0.81699999999999995) at checks.c:414
#6  0x000000000044aad5 in handle_timed_event (event=0x804e12160) at events.c:1493
#7  0x000000000044a18b in event_execution_loop () at events.c:1390
#8  0x000000000041684e in main (argc=3, argv=0x7fffffffec98, env=0x7fffffffecb8) at icinga.c:891

(gdb)

If you want, I can show variables.

Updated by leo9641 on 2016-01-13 10:41:18 +00:00

# gdb /usr/local/bin/icinga /var/coredumps/icinga-63934.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `icinga'. Program terminated with signal 11, Segmentation fault. Error while mapping shared library sections: /var/spool/icinga/ramdisk/tmp/icinganebmodPMz0Fq: No such file or directory. Error while mapping shared library sections: /var/spool/icinga/ramdisk/tmp/icinganebmodbC2v21: No such file or directory. Reading symbols from /lib/libm.so.5...done. Loaded symbols for /lib/libm.so.5 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /usr/local/lib/libltdl.so.7...done. Loaded symbols for /usr/local/lib/libltdl.so.7 Reading symbols from /lib/libz.so.6...done. Loaded symbols for /lib/libz.so.6 Reading symbols from /usr/lib/libssl.so.7...done. Loaded symbols for /usr/lib/libssl.so.7 Reading symbols from /lib/libcrypto.so.7...done. Loaded symbols for /lib/libcrypto.so.7 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Error while reading shared library symbols: /var/spool/icinga/ramdisk/tmp/icinganebmodPMz0Fq: No such file or directory. Reading symbols from /usr/local/lib/libgearman.so.7...done. Loaded symbols for /usr/local/lib/libgearman.so.7 Reading symbols from /usr/lib/libc.so.1...done. Loaded symbols for /usr/lib/libc.so.1 Reading symbols from /lib/libcxxrt.so.1...done. Loaded symbols for /lib/libcxxrt.so.1 Reading symbols from /usr/local/lib/libuuid.so.1...done. Loaded symbols for /usr/local/lib/libuuid.so.1 Reading symbols from /usr/lib/libexecinfo.so.1...done. Loaded symbols for /usr/lib/libexecinfo.so.1 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /usr/lib/libelf.so.1...done. Loaded symbols for /usr/lib/libelf.so.1 Error while reading shared library symbols: /var/spool/icinga/ramdisk/tmp/icinganebmodbC2v21: No such file or directory. Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1

#0  0x000000080187d4c7 in sbrk () from /lib/libc.so.7

[New Thread 802006400 (LWP 101375/)] (gdb) bt

#0  0x000000080187d4c7 in sbrk () from /lib/libc.so.7
#1  0x000000080187d0cd in sbrk () from /lib/libc.so.7
#2  0x00000008018798ab in sbrk () from /lib/libc.so.7
#3  0x0000000801879a4b in sbrk () from /lib/libc.so.7
#4  0x0000000801884bad in free () from /lib/libc.so.7
#5  0x00000008018fe9de in fclose () from /lib/libc.so.7
#6  0x0000000000469c73 in service_check_sighandler (sig=14) at utils.c:2420
#7  0x0000000800b3b9aa in pthread_sigmask () from /lib/libthr.so.3
#8  0x0000000800b3b1a8 in pthread_getspecific () from /lib/libthr.so.3
#9  
#10 0x000000080187d9b0 in sbrk () from /lib/libc.so.7
#11 0x000000080187d172 in sbrk () from /lib/libc.so.7
#12 0x00000008018798ab in sbrk () from /lib/libc.so.7
#13 0x0000000801879a4b in sbrk () from /lib/libc.so.7
#14 0x0000000801884bad in free () from /lib/libc.so.7
#15 0x00000008018fe9de in fclose () from /lib/libc.so.7
#16 0x0000000000427d7c in run_check (processed_command=0x80461aac0 "/usr/local/libexec/nagios/check_nrpe2 -H h-kpc-trt-ns-8.REMOVE.com -c check_ntp_time -t 62",

checkresult_dbuf=0x7fffffffde10) at checks.c:261

#17 0x0000000000426e8a in run_async_service_check (svc=0x80420ea00, check_options=0, latency=0.21299999999999999, scheduled_check=1, reschedule_check=1, time_is_valid=0x7fffffffe57c,

preferred_time=0x7fffffffe588) at checks.c:947

#18 0x0000000000425e51 in run_scheduled_service_check (svc=0x80420ea00, check_options=0, latency=0.21299999999999999) at checks.c:414
#19 0x000000000044aad5 in handle_timed_event (event=0x804db3b60) at events.c:1493
#20 0x000000000044a18b in event_execution_loop () at events.c:1390
#21 0x000000000041684e in main (argc=3, argv=0x7fffffffec98, env=0x7fffffffecb8) at icinga.c:891

(gdb) up 16

#16 0x0000000000427d7c in run_check (processed_command=0x80461aac0 "/usr/local/libexec/nagios/check_nrpe2 -H h-kpc-trt-ns-8.REMOVE.com -c check_ntp_time -t 62",

checkresult_dbuf=0x7fffffffde10) at checks.c:261 261 fclose(fp); Current language: auto; currently minimal (gdb)

Updated by mfriedrich on 2016-01-13 10:45:08 +00:00

Disable mod_gearman and run it again.

Updated by leo9641 on 2016-01-13 11:23:57 +00:00

dnsmichi wrote:

Disable mod_gearman and run it again.

Ok, now I disable mod_gearman. But I not sure that this help. Because checks for h-kpc-trt-ns-8 and st13 not use gearman queue. Now we need to wait for new coredumps.

Updated by leo9641 on 2016-01-18 07:09:25 +00:00

leo9641 wrote:

dnsmichi wrote: > Disable mod_gearman and run it again.

Ok, now I disable mod_gearman. But I not sure that this help. Because checks for h-kpc-trt-ns-8 and st13 not use gearman queue. Now we need to wait for new coredumps.

New backtrace:

(gdb) # gdb /usr/local/bin/icinga /var/coredumps/icinga-20509.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `icinga'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libm.so.5...done. Loaded symbols for /lib/libm.so.5 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /usr/local/lib/libltdl.so.7...done. Loaded symbols for /usr/local/lib/libltdl.so.7 Reading symbols from /lib/libz.so.6...done. Loaded symbols for /lib/libz.so.6 Reading symbols from /usr/lib/libssl.so.7...done. Loaded symbols for /usr/lib/libssl.so.7 Reading symbols from /lib/libcrypto.so.7...done. Loaded symbols for /lib/libcrypto.so.7 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /var/spool/icinga/ramdisk/tmp/icinganebmodPZVQjd...done. Loaded symbols for /var/spool/icinga/ramdisk/tmp/icinganebmodPZVQjd Reading symbols from /usr/lib/libc.so.1...done. Loaded symbols for /usr/lib/libc.so.1 Reading symbols from /lib/libcxxrt.so.1...done. Loaded symbols for /lib/libcxxrt.so.1 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1

#0  0x00000008018ef81a in localtime () from /lib/libc.so.7

[New Thread 80200b800 (LWP 103229/)] [New Thread 80200b400 (LWP 103152/)] [New Thread 80200b000 (LWP 103147/)] [New Thread 80200ac00 (LWP 103113/)] [New Thread 80200a800 (LWP 103050/)] [New Thread 80200a400 (LWP 103017/)] [New Thread 80200a000 (LWP 102981/)] [New Thread 802009c00 (LWP 102932/)] [New Thread 802009800 (LWP 102920/)] [New Thread 802009400 (LWP 102847/)] [New Thread 802007800 (LWP 102803/)] [New Thread 802007400 (LWP 102792/)] [New Thread 802006400 (LWP 102309/)] (gdb) bt

#0  0x00000008018ef81a in localtime () from /lib/libc.so.7
#1  0x00000008018ef8a0 in localtime_r () from /lib/libc.so.7
#2  0x0000000000466d06 in check_time_against_period (test_time=1452960814, tperiod=0x80316d4c0) at utils.c:958
#3  0x0000000000427a16 in check_service_check_viability (svc=0x803b18f80, check_options=0, time_is_valid=0x7fffffffe3dc, new_time=0x7fffffffe3e8) at checks.c:2053
#4  0x0000000000426174 in run_async_service_check (svc=0x803b18f80, check_options=0, latency=0.14399999999999999, scheduled_check=1, reschedule_check=1, time_is_valid=0x7fffffffe3dc,

preferred_time=0x7fffffffe3e8) at checks.c:527

#5  0x0000000000425e51 in run_scheduled_service_check (svc=0x803b18f80, check_options=0, latency=0.14399999999999999) at checks.c:414
#6  0x000000000044aad5 in handle_timed_event (event=0x8032feca0) at events.c:1493
#7  0x000000000044a18b in event_execution_loop () at events.c:1390
#8  0x000000000041684e in main (argc=3, argv=0x7fffffffeaf0, env=0x7fffffffeb10) at icinga.c:891

(gdb)

Updated by leo9641 on 2016-01-18 07:11:26 +00:00

(gdb) up 3

#3  0x0000000000427a16 in check_service_check_viability (svc=0x803b18f80, check_options=0, time_is_valid=0x7fffffffe3dc, new_time=0x7fffffffe3e8) at checks.c:2053

2053 if (check_time_against_period((unsigned long)current_time, svc->check_period_ptr) == ERROR) { Current language: auto; currently minimal (gdb)

Updated by leo9641 on 2016-01-18 07:12:50 +00:00

(gdb) up 2

#2  0x0000000000466d06 in check_time_against_period (test_time=1452960814, tperiod=0x80316d4c0) at utils.c:958

958 t = localtime_r(&test_time, &tm_s); Current language: auto; currently minimal (gdb)

Updated by leo9641 on 2016-01-18 07:48:17 +00:00

(gdb) info locals temp_timeperiodexclusion = (timeperiodexclusion *) 0x0 first_timeperiodexclusion = (timeperiodexclusion *) 0x0 temp_daterange = (daterange *) 0x0 temp_timerange = (timerange *) 0x0 midnight = 0 start_time = 0 end_time = 0 found_match = 0 t = (struct tm *) 0xd tm_s = {tm_sec = 34, tm_min = 13, tm_hour = 19, tm_mday = 16, tm_mon = 0, tm_year = 116, tm_wday = 6, tm_yday = 15, tm_isdst = 0, tm_gmtoff = 10800, tm_zone = 0x24f6e19747327a2e

<Address 0x24f6e19747327a2e out of bounds> } daterange_type = 0 days = 0 day_range_start = 0 day_range_end = 0 test_time_year = 0 test_time_mon = 0 test_time_wday = 0 year = 0 shift = 1076101120

Icinga / icinga-core

[dev.icinga.com #10960] (icinga), uid 183: exited on signal 11 (core dumped) #1569