search

openhpi2 / Open-HPI

Open HPI is an open source implementation of the SA Forum's Hardware Platform Interface (HPI). HPI provides an abstracted interface to managing computer hardware, typically for chassis and rack based servers
Other
3 stars 1 forks source link

Two openhpid 3.4.0 cores during power set and get operations #2606

Closed openhpi2 closed 8 years ago

openhpi2 commented 8 years ago

We had two cores of openhpid 3.4.0 on one system, one day apart. This was openhpid 3.4.0 + oa_soap plugin on a Red Hat based Linux system managing HP c-Class equipment.

I've included both cores in this ticket. Both were segfaults that happened in the process of dealing with power state. The first one occurred shortly after an upgrade of openhpid and our client application; the second one about a day later.

Core #1, from Sep 8 2015 07:48:49

This first core occurred after a reboot of the server on which the openhpid client application was running. The reboot followed an upgrade of the client application, in which openhpid was upgraded from 2.14.1 to 3.4.0.

Core file created at UTC: 1441712929.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[New Thread 14211]
[New Thread 14246]
[New Thread 14350]
[New Thread 17240]
[New Thread 15117]
[New Thread 15176]
[New Thread 15177]
[New Thread 15178]
[New Thread 15183]
[New Thread 15184]
[New Thread 15185]
[New Thread 16316]
[New Thread 8773]
[New Thread 13304]
[New Thread 13325]
[New Thread 13345]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/sbin/openhpid -c /etc/openhpi/openhpi.conf.4744 -p 4744 -s 0 -f /var/run/o'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000032e9241718 in SSL_shutdown () from /usr/lib64/libssl.so.10

Thread 16 (Thread 0x7fd8357fb700 (LWP 13345)):
#0  0x00000032e320eb5c in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a03c12 in cStreamSock::ReadMsg (this=0x1548890, type=@0x7fd8357fad03, id=@0x7fd8357facfc, payload=0x7fd8357eacf0, payload_len=@0x7fd8357fad04, payload_byte_order=@0x7fd8357facf8) at strmsock.cpp:188
        len = 20
        rawhdr = "\001\021\000\000\001\000\000\000\004\000\000"
        dst = 0x7fd8357eac70 "\001\021"
        need = 12
        hdr = "\001\021\000\000\001\000\000\000\004\000\000"
        got = 0
#2  0x0000000000407e6f in service_thread(void*, void*) ()
No symbol table info available.
#3  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x1548890
        pool = 0x16c9820
#4  0x00000032e466a6c4 in g_thread_create_proxy (data=0x1649c60) at gthread.c:1897
        thread = 0x1649c60
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#5  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7fd8361fc700 (LWP 13325)):
#0  0x00000032e320eb5c in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a03c12 in cStreamSock::ReadMsg (this=0x1530800, type=@0x7fd8361fbd03, id=@0x7fd8361fbcfc, payload=0x7fd8361ebcf0, payload_len=@0x7fd8361fbd04, payload_byte_order=@0x7fd8361fbcf8) at strmsock.cpp:188
        len = 20
        rawhdr = "\001\021\000\000\001\000\000\000\004\000\000"
        dst = 0x7fd8361ebc70 "\001\021"
        need = 12
        hdr = "\001\021\000\000\001\000\000\000\004\000\000"
        got = 0
#2  0x0000000000407e6f in service_thread(void*, void*) ()
No symbol table info available.
#3  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x1530800
        pool = 0x16c9820
#4  0x00000032e466a6c4 in g_thread_create_proxy (data=0x15067d0) at gthread.c:1897
        thread = 0x15067d0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#5  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7fd836bfd700 (LWP 13304)):
#0  0x00000032e320eb5c in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a03c12 in cStreamSock::ReadMsg (this=0x154b060, type=@0x7fd836bfcd03, id=@0x7fd836bfccfc, payload=0x7fd836beccf0, payload_len=@0x7fd836bfcd04, payload_byte_order=@0x7fd836bfccf8) at strmsock.cpp:188
        len = 20
        rawhdr = "\001\021\000\000\001\000\000\000\004\000\000"
        dst = 0x7fd836becc70 "\001\021"
        need = 12
        hdr = "\001\021\000\000\001\000\000\000\004\000\000"
        got = 0
#2  0x0000000000407e6f in service_thread(void*, void*) ()
No symbol table info available.
#3  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x154b060
        pool = 0x16c9820
#4  0x00000032e466a6c4 in g_thread_create_proxy (data=0x16a0d00) at gthread.c:1897
        thread = 0x16a0d00
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#5  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7fd8375fe700 (LWP 8773)):
#0  0x00000032e320eb5c in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a03c12 in cStreamSock::ReadMsg (this=0x1675170, type=@0x7fd8375fdd03, id=@0x7fd8375fdcfc, payload=0x7fd8375edcf0, payload_len=@0x7fd8375fdd04, payload_byte_order=@0x7fd8375fdcf8) at strmsock.cpp:188
        len = 20
        rawhdr = "\001\021\000\000\001\000\000\000\004\000\000"
        dst = 0x7fd8375edc70 "\001\021"
        need = 12
        hdr = "\001\021\000\000\001\000\000\000\004\000\000"
        got = 0
#2  0x0000000000407e6f in service_thread(void*, void*) ()
No symbol table info available.
#3  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x1675170
        pool = 0x16c9820
#4  0x00000032e466a6c4 in g_thread_create_proxy (data=0x168d630) at gthread.c:1897
        thread = 0x168d630
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#5  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7fd837fff700 (LWP 16316)):
#0  0x00000032e320b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e5602352 in g_cond_timed_wait_posix_impl (cond=<value optimized out>, entered_mutex=<value optimized out>, abs_time=<value optimized out>) at gthread-posix.c:242
        result = <value optimized out>
        end_time = {tv_sec = 1441712930, tv_nsec = 594245000}
        timed_out = <value optimized out>
        __PRETTY_FUNCTION__ = "g_cond_timed_wait_posix_impl"
#2  0x00000032e4618751 in g_async_queue_pop_intern_unlocked (queue=0x7fd82c001420, try=0, end_time=0x7fd837fe6e10) at gasyncqueue.c:423
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_pop_intern_unlocked"
#3  0x00000032e4618897 in g_async_queue_timed_pop (queue=0x7fd82c001420, end_time=0x7fd837fe6e10) at gasyncqueue.c:549
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_timed_pop"
#4  0x0000000000428543 in oh_dequeue_session_event ()
No symbol table info available.
#5  0x0000000000419e0e in saHpiEventGet ()
No symbol table info available.
#6  0x000000000040a23c in process_msg(cHpiMarshal*, int, char*, unsigned int&, unsigned int&) ()
No symbol table info available.
#7  0x0000000000407f52 in service_thread(void*, void*) ()
No symbol table info available.
#8  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x1672470
        pool = 0x16c9820
#9  0x00000032e466a6c4 in g_thread_create_proxy (data=0x1654ab0) at gthread.c:1897
        thread = 0x1654ab0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#10 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7fd84d2a3700 (LWP 15185)):
#0  0x00000032e320eb5c in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a03c12 in cStreamSock::ReadMsg (this=0x13b2ef0, type=@0x7fd84d2a2d03, id=@0x7fd84d2a2cfc, payload=0x7fd84d292cf0, payload_len=@0x7fd84d2a2d04, payload_byte_order=@0x7fd84d2a2cf8) at strmsock.cpp:188
        len = 537
        rawhdr = "\001\021\000\000\"\000\000\000\f\000\000"
        dst = 0x7fd84d292c70 "\001\021"
        need = 12
        hdr = "\001\021\000\000\"\000\000\000\f\000\000"
        got = 0
#2  0x0000000000407e6f in service_thread(void*, void*) ()
No symbol table info available.
#3  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x13b2ef0
        pool = 0x16c9820
#4  0x00000032e466a6c4 in g_thread_create_proxy (data=0x1652c80) at gthread.c:1897
        thread = 0x1652c80
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#5  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7fd84dca4700 (LWP 15184)):
#0  0x00000032e2ae12e3 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00000032e8202a75 in oh_ssl_read (bio=0x7fd83c062980, buf=0x7fd84dca2b90 "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:SOAP-ENC=\"http://www.w3.org/2003/05/soap-encoding\" xmlns:xsi=\"http://www.w3.org"..., size=3999, timeout=40) at oh_ssl.c:717
        ssl = 0x7fd83c0bb860
        bytes = 0
        readfds = {fds_bits = {256, 0 <repeats 15 times>}}
        writefds = {fds_bits = {0 <repeats 16 times>}}
        tv = {tv_sec = 30, tv_usec = 479564}
        read_wait = 1
        done = 0
        err = 8
        fd = 8
#2  0x00007fd850857a12 in soap_message (connection=0x1604860, request=0x16049d8 "<?xml version=\"1.0\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xml"..., doc=0x16049d0) at oa_soap_callsupport.c:696
        nbytes = 874
        ret = 117
        header = 0x7fd83c0d2bd0 ""
        response = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:SOAP-ENC=\"http://www.w3.org/2003/05/soap-encoding\" xmlns:xsi=\"http://www.w3.org"...
        parse = 0x0
#3  0x00007fd850858633 in soap_call (connection=0x1604860) at oa_soap_callsupport.c:1085
        session_pos = 0x1604be0 "3db2eea26a6a7d5e</hpoa:oaSessionKey>\n</hpoa:HpOaSessionKeyToken>\n</wsse:Security>\n</SOAP-ENV:Header>\n<SOAP-ENV:Body>\n<hpoa:getAllEventsEx><hpoa:pid>26054</hpoa:pid><hpoa:waitTilEventHappens>1</hpoa:wa"...
        i = 0
        err = 32728
#4  0x00007fd850854a22 in soap_getAllEventsEx (con=0x1604860, request=0x7fd84dca3d30, response=0x7fd84dca3d20) at oa_soap_calls.c:1601
        ret = 0
#5  0x00007fd850831854 in oa_soap_event_thread (oa_pointer=0x13b2d30) at oa_soap_event.c:244
        rv = 0
        request = {pid = 26054, waitTilEventHappens = HPOA_TRUE, lcdEvents = HPOA_FALSE, oaFwVersion = 0x7fd84dca3c10 "3.71"}
        response = {eventInfoArray = 0x0}
        handler = 0x13b4a30
        oa = 0x13b2d30
        ret_code = -1009
        retry_on_switchover = 0
        oa_handler = 0x13b29d0
        is_plugin_initialized = 1 '\001'
        is_discovery_completed = 1 '\001'
        listen_for_events = 1 '\001'
        user_name = 0x13b2080 "pmacadmin"
        password = 0x13b2990 "********"
        url = 0x0
        oa_fw_buf = "3.71", '\000' <repeats 250 times>
#6  0x00000032e466a6c4 in g_thread_create_proxy (data=0x7fd84000ac30) at gthread.c:1897
        thread = 0x7fd84000ac30
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#7  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7fd84e6a5700 (LWP 15183)):
#0  0x00000032e2ae12e3 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00000032e8202a75 in oh_ssl_read (bio=0x7fd838008690, buf=0x7fd84e6a3b90 "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:SOAP-ENC=\"http://www.w3.org/2003/05/soap-encoding\" xmlns:xsi=\"http://www.w3.org"..., size=3999, timeout=40) at oh_ssl.c:717
        ssl = 0x7fd8380bb860
        bytes = 0
        readfds = {fds_bits = {131072, 0 <repeats 15 times>}}
        writefds = {fds_bits = {0 <repeats 16 times>}}
        tv = {tv_sec = 34, tv_usec = 791343}
        read_wait = 1
        done = 0
        err = 17
        fd = 17
#2  0x00007fd850857a12 in soap_message (connection=0x1470200, request=0x1470378 "<?xml version=\"1.0\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xml"..., doc=0x1470370) at oa_soap_callsupport.c:696
        nbytes = 874
        ret = 117
        header = 0x7fd8380d53c0 ""
        response = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:SOAP-ENC=\"http://www.w3.org/2003/05/soap-encoding\" xmlns:xsi=\"http://www.w3.org"...
        parse = 0x0
#3  0x00007fd850858633 in soap_call (connection=0x1470200) at oa_soap_callsupport.c:1085
        session_pos = 0x1470580 "0df66037f5e82729</hpoa:oaSessionKey>\n</hpoa:HpOaSessionKeyToken>\n</wsse:Security>\n</SOAP-ENV:Header>\n<SOAP-ENV:Body>\n<hpoa:getAllEventsEx><hpoa:pid>28807</hpoa:pid><hpoa:waitTilEventHappens>1</hpoa:wa"...
        i = 0
        err = 32728
#4  0x00007fd850854a22 in soap_getAllEventsEx (con=0x1470200, request=0x7fd84e6a4d30, response=0x7fd84e6a4d20) at oa_soap_calls.c:1601
        ret = 0
#5  0x00007fd850831854 in oa_soap_event_thread (oa_pointer=0x13b2be0) at oa_soap_event.c:244
        rv = 0
        request = {pid = 28807, waitTilEventHappens = HPOA_TRUE, lcdEvents = HPOA_FALSE, oaFwVersion = 0x7fd84e6a4c10 "3.71"}
        response = {eventInfoArray = 0x0}
        handler = 0x13b4a30
        oa = 0x13b2be0
        ret_code = -1009
        retry_on_switchover = 0
        oa_handler = 0x13b29d0
        is_plugin_initialized = 1 '\001'
        is_discovery_completed = 1 '\001'
        listen_for_events = 1 '\001'
        user_name = 0x13b2080 "pmacadmin"
        password = 0x13b2990 "********"
        url = 0x0
        oa_fw_buf = "3.71", '\000' <repeats 250 times>
#6  0x00000032e466a6c4 in g_thread_create_proxy (data=0x7fd8400008c0) at gthread.c:1897
        thread = 0x7fd8400008c0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#7  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7fd84f0a6700 (LWP 15178)):
#0  0x00000032e320b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e4618712 in g_async_queue_pop_intern_unlocked (queue=0x13960f0, try=0, end_time=0x0) at gasyncqueue.c:416
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_pop_intern_unlocked"
#2  0x00000032e4618af6 in g_async_queue_pop (queue=0x13960f0) at gasyncqueue.c:456
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_pop"
#3  0x0000000000413620 in oh_process_events ()
No symbol table info available.
#4  0x0000000000428b84 in evtpop_func ()
No symbol table info available.
#5  0x00000032e466a6c4 in g_thread_create_proxy (data=0x16c92c0) at gthread.c:1897
        thread = 0x16c92c0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#6  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7fd84faa7700 (LWP 15177)):
#0  0x00000032e320e264 in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e3209508 in _L_lock_854 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00000032e32093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00000032e466b1ec in g_static_rec_mutex_lock (mutex=0x13b2728) at gthread.c:1424
        self = {data = "\000w\252O\330\177\000", dummy_double = 6.9449338600845915e-310, dummy_pointer = 0x7fd84faa7700, dummy_long = 140567026235136}
        __PRETTY_FUNCTION__ = "g_static_rec_mutex_lock"
#4  0x0000000000415568 in oh_get_handler ()
No symbol table info available.
#5  0x0000000000412870 in oh_harvest_events ()
No symbol table info available.
#6  0x0000000000428a1d in evtget_func ()
No symbol table info available.
#7  0x00000032e466a6c4 in g_thread_create_proxy (data=0x168a2f0) at gthread.c:1897
        thread = 0x168a2f0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#8  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7fd8504a8700 (LWP 15176)):
#0  0x00000032e320b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e5602352 in g_cond_timed_wait_posix_impl (cond=<value optimized out>, entered_mutex=<value optimized out>, abs_time=<value optimized out>) at gthread-posix.c:242
        result = <value optimized out>
        end_time = {tv_sec = 1441713042, tv_nsec = 48752000}
        timed_out = <value optimized out>
        __PRETTY_FUNCTION__ = "g_cond_timed_wait_posix_impl"
#2  0x00000000004288e8 in discovery_func ()
No symbol table info available.
#3  0x00000032e466a6c4 in g_thread_create_proxy (data=0x167a2c0) at gthread.c:1897
        thread = 0x167a2c0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#4  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7fd850ab77e0 (LWP 15117)):
#0  0x00000032e320ea5d in accept () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a04a3c in cServerStreamSock::Accept (this=0x154b040) at strmsock.cpp:567
        sock = 0
#2  0x0000000000407c60 in oh_server_run(int, char const*, unsigned short, unsigned int, int) ()
No symbol table info available.
#3  0x0000000000412420 in main ()
No symbol table info available.

Thread 4 (Thread 0x7fd82abfd700 (LWP 17240)):
#0  0x00000032e320e264 in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e3209508 in _L_lock_854 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00000032e32093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00000032e466b1ec in g_static_rec_mutex_lock (mutex=0x1670290) at gthread.c:1424
        self = {data = "\000\327\277*\330\177\000", dummy_double = 6.9449032598244718e-310, dummy_pointer = 0x7fd82abfd700, dummy_long = 140566406878976}
        __PRETTY_FUNCTION__ = "g_static_rec_mutex_lock"
#4  0x000000000042c55c in __get_domain ()
No symbol table info available.
#5  0x000000000042d0ac in oh_get_domain ()
No symbol table info available.
#6  0x0000000000427d81 in oh_create_session ()
No symbol table info available.
#7  0x0000000000417086 in saHpiSessionOpen ()
No symbol table info available.
#8  0x000000000040821d in process_msg(cHpiMarshal*, int, char*, unsigned int&, unsigned int&) ()
No symbol table info available.
#9  0x0000000000407f52 in service_thread(void*, void*) ()
No symbol table info available.
#10 0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x154a610
        pool = 0x16c9820
#11 0x00000032e466a6c4 in g_thread_create_proxy (data=0x1699ab0) at gthread.c:1897
        thread = 0x1699ab0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#12 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#13 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7fd82b5fe700 (LWP 14350)):
#0  0x00000032e320e264 in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e3209508 in _L_lock_854 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00000032e32093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00000032e466b1ec in g_static_rec_mutex_lock (mutex=0x1670290) at gthread.c:1424
        self = {data = "\000\347_+\330\177\000", dummy_double = 6.9449037780922198e-310, dummy_pointer = 0x7fd82b5fe700, dummy_long = 140566417368832}
        __PRETTY_FUNCTION__ = "g_static_rec_mutex_lock"
#4  0x000000000042c55c in __get_domain ()
No symbol table info available.
#5  0x000000000042d0ac in oh_get_domain ()
No symbol table info available.
#6  0x0000000000427779 in saHpiResourceResetStateSet ()
No symbol table info available.
#7  0x0000000000410acd in process_msg(cHpiMarshal*, int, char*, unsigned int&, unsigned int&) ()
No symbol table info available.
#8  0x0000000000407f52 in service_thread(void*, void*) ()
No symbol table info available.
#9  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x1524940
        pool = 0x16c9820
#10 0x00000032e466a6c4 in g_thread_create_proxy (data=0x16608a0) at gthread.c:1897
        thread = 0x16608a0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#11 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#12 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7fd82bfff700 (LWP 14246)):
#0  0x00000032e320e264 in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e3209508 in _L_lock_854 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00000032e32093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00000032e466b1ec in g_static_rec_mutex_lock (mutex=0x13b2728) at gthread.c:1424
        self = {data = "\000\367\377+\330\177\000", dummy_double = 6.9449042963599677e-310, dummy_pointer = 0x7fd82bfff700, dummy_long = 140566427858688}
        __PRETTY_FUNCTION__ = "g_static_rec_mutex_lock"
#4  0x0000000000415568 in oh_get_handler ()
No symbol table info available.
#5  0x0000000000427861 in saHpiResourceResetStateSet ()
No symbol table info available.
#6  0x0000000000410acd in process_msg(cHpiMarshal*, int, char*, unsigned int&, unsigned int&) ()
No symbol table info available.
#7  0x0000000000407f52 in service_thread(void*, void*) ()
No symbol table info available.
#8  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x154a350
        pool = 0x16c9820
#9  0x00000032e466a6c4 in g_thread_create_proxy (data=0x16a4d30) at gthread.c:1897
        thread = 0x16a4d30
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#10 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7fd834dfa700 (LWP 14211)):
#0  0x00000032e9241718 in SSL_shutdown () from /usr/lib64/libssl.so.10
No symbol table info available.
#1  0x00000032e8202634 in oh_ssl_disconnect (bio=0x7fd830001970, shutdown=OH_SSL_UNI) at oh_ssl.c:594
        ssl = 0x32e466c7f0
        ret = 886968192
        fd = 32728
#2  0x00007fd850857a33 in soap_message (connection=0x13b4ec0, request=0x13b5038 "", doc=0x13b5030) at oa_soap_callsupport.c:700
        nbytes = -2
        ret = 117
        header = 0x7fd830018c00 " \211\001\060\330\177"
        response = '\000' <repeats 657 times>, "\030\000\060\330\177\000\000\000\031\336\064\330\177\000\000\003\000\000\000\000\000\000\000\020\031\336\064\330\177\000\000yo\265\342\062\000\000\000wA\244\342\062\000\000\000zo\265\342\062\000\000\000\001\000\000\000\000\000\000\000\001", '\000' <repeats 47 times>"\360, \027\336\064\330\177\000\000\000\000\000\000\000\000\000\000vo\265\342\062", '\000' <repeats 27 times>, "\\\030\336\064\330\177\000\000x\030\336\064\330\177\000\000\000\000\000\000\000\000\000\000\230\030\336\064\330\177", '\000' <repeats 26 times>, "nF\247\342\062\000\000\000\000\030\000\060\330\177\000\000\340\031\336\064\330\177\000\000?\000\000\000\000\000\000\000\360\032\336\064\330\177\000\000\205\352B\000\000\000\000\000wA\244\342\062\000\000\000\206\352B\000\000\000\000\000\001", '\000' <repeats 15 times>"\310, \027\336\064\330\177\000\000\000\000\000\000\000\000\000\000"...
        parse = 0x0
#3  0x00007fd850858633 in soap_call (connection=0x13b4ec0) at oa_soap_callsupport.c:1085
        session_pos = 0x13b5240 "89f2c5fe797617cf</hpoa:oaSessionKey>\n</hpoa:HpOaSessionKeyToken>\n</wsse:Security>\n</SOAP-ENV:Header>\n<SOAP-ENV:Body>\n<hpoa:getBladeStatus><hpoa:bayNumber>2</hpoa:bayNumber></hpoa:getBladeStatus>\n</SOA"...
        i = 0
        err = 50
#4  0x00007fd850855c8d in soap_getBladeStatus (con=0x13b4ec0, request=0x7fd834de2070, response=0x7fd834de2000) at oa_soap_calls.c:1964
        ret = 50
#5  0x00007fd8508160dc in get_server_power_state (con=0x13b4ec0, bay_number=2, state=0x7fd834de2118) at oa_soap_power.c:242
        rv = 0
        request = {bayNumber = 2}
        response = {bayNumber = 89 'Y', presence = 32728, operationalStatus = 886972496, thermal = 32728, powered = 21307520, powerState = PS_NO_OP, shutdown = 3831941104, uid = 50, powerConsumed = 1, diagnosticChecks = {internalDataError = NOT_RELEVANT, managementProcessorError = 3877717800, thermalWarning = 50, thermalDanger = 3831941104, ioConfigurationError = ERROR, devicePowerRequestError = 20663824, insufficientCooling = NOT_RELEVANT, deviceLocationError = 1073847984, deviceFailure = 32728, deviceDegraded = 1073828640, acFailure = 32728, i2cBuses = 886972544, redundancy = 32728}, diagnosticChecksEx = 0x32e72151cc, extraData = 0x350816a59}
#6  0x00007fd850815dde in oa_soap_get_power_state (oh_handler=0x13b4a30, resource_id=3, state=0x7fd834de2118) at oa_soap_power.c:118
        rv = 0
        rpt = 0x7fd840015320
        oa_handler = 0x13b29d0
        bay_number = 2
        handler = 0x13b4a30
#7  0x00007fd850816c43 in oa_soap_set_reset_state (oh_handler=0x13b4a30, resource_id=3, action=SAHPI_WARM_RESET) at oa_soap_reset.c:188
        rv = 0
        tmp = 3892408096
        rpt = 0x7fd840015320
        bay_number = 0
        server_request = {bayNumber = -463026192, power = 50}
        interconnect_request = {bayNumber = 886972704}
        handler = 0x13b4a30
        oa_handler = 0x13b29d0
#8  0x00000000004278eb in saHpiResourceResetStateSet ()
No symbol table info available.
#9  0x0000000000410acd in process_msg(cHpiMarshal*, int, char*, unsigned int&, unsigned int&) ()
No symbol table info available.
#10 0x0000000000407f52 in service_thread(void*, void*) ()
No symbol table info available.
#11 0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x1452080
        pool = 0x16c9820
#12 0x00000032e466a6c4 in g_thread_create_proxy (data=0x169a9b0) at gthread.c:1897
        thread = 0x169a9b0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#13 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#14 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Core #2, from Sep 9 2015 05:05:33

The second core occurred the next day. As far as I can tell, nothing unusual was happening on the system at the time.

Core file created at UTC: 1441789533.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[New Thread 29199]
[New Thread 11030]
[New Thread 11090]
[New Thread 11091]
[New Thread 11092]
[New Thread 11110]
[New Thread 11153]
[New Thread 14336]
[New Thread 29100]
[New Thread 11093]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/sbin/openhpid -c /etc/openhpi/openhpi.conf.4744 -p 4744 -s 0 -f /var/run/o'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000032e9241718 in SSL_shutdown () from /usr/lib64/libssl.so.10

Thread 10 (Thread 0x7f9093d14700 (LWP 11093)):
#0  0x00000032e320e75d in read () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e4adb3b1 in ?? () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#2  0x00000032e4ad9709 in BIO_read () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#3  0x00000032e9231213 in ssl23_read_bytes () from /usr/lib64/libssl.so.10
No symbol table info available.
#4  0x00000032e92301ce in ssl23_connect () from /usr/lib64/libssl.so.10
No symbol table info available.
#5  0x00000032e82024b6 in oh_ssl_connect (hostname=0x13f0218 "172.21.74.200:443", ctx=0x1396e40, timeout=40) at oh_ssl.c:542
        bio = 0x0
        ssl = 0x7f907c123020
        err = 32656
        len = 17
        retval = 0
        RetVal = 0
        socket_desc = 4
        Server = 0x7f907c1302c0 "172.21.74.200"
        Port = 0x7f907c138e80 "443"
        Hints = {ai_flags = 0, ai_family = 0, ai_socktype = 1, ai_protocol = 0, ai_addrlen = 0, ai_addr = 0x0, ai_canonname = 0x0, ai_next = 0x0}
        AddrInfo = 0x7f907c04ee70
        ai = 0x7f907c04ee70
#6  0x00007f9095ec676a in soap_message (connection=0x13f0200, request=0x13f0378 "<?xml version=\"1.0\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xml"..., doc=0x13f0370) at oa_soap_callsupport.c:643
        nbytes = 0
        ret = 0
        header = 0x0
        response = "d\000\000\000\000\000\000\000\340+\321\223\220\177\000\000\360y\022|\220\177\000\000\070-\321\223\220\177\000\000<\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000@z\022|\220\177\000\000\000\000\000\000\000\000\000\000H\326\355\225\220\177\000\000c\000\000\000\000\000\000\000@z\022|\220\177\000\000]\366\357U\000\000\000\000\360y\022|\220\177\000\000\n\000\000\000\000\000\000\000@-\321\223\220\177\000\000 2\321\223\220\177\000\000\b\000\000\000\000\000\000\000\260-\321\223\220\177\000\000\300d\221\344\062\000\000\000\000P\256\342\062\000\000\000 \000\000\000\060\000\000\000\020-\321\223\220\177\000\000P,\321\223\220\177\000\000\000\000\000\000\000\000\000\000w.w3\004\000\000\000\270,\321\223\220\177\000\000]\324\355\225\220\177\000\000\360y\022|\220\177\000\000\377\377\377\377\000\000\000\000 cap con\300,\321\223\220\177\000\000\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\240"...
        parse = 0x0
#7  0x00007f9095ec7633 in soap_call (connection=0x13f0200) at oa_soap_callsupport.c:1085
        session_pos = 0x13f0580 "8cb7ad1ac245bd1d</hpoa:oaSessionKey>\n</hpoa:HpOaSessionKeyToken>\n</wsse:Security>\n</SOAP-ENV:Header>\n<SOAP-ENV:Body>\n<hpoa:getAllEventsEx><hpoa:pid>1750</hpoa:pid><hpoa:waitTilEventHappens>1</hpoa:wai"...
        i = 0
        err = 32656
#8  0x00007f9095ec3a22 in soap_getAllEventsEx (con=0x13f0200, request=0x7f9093d13d30, response=0x7f9093d13d20) at oa_soap_calls.c:1601
        ret = 0
#9  0x00007f9095ea0854 in oa_soap_event_thread (oa_pointer=0x1332be0) at oa_soap_event.c:244
        rv = 0
        request = {pid = 1750, waitTilEventHappens = HPOA_TRUE, lcdEvents = HPOA_FALSE, oaFwVersion = 0x7f9093d13c10 "3.71"}
        response = {eventInfoArray = 0x0}
        handler = 0x1334a30
        oa = 0x1332be0
        ret_code = -1009
        retry_on_switchover = 0
        oa_handler = 0x13329d0
        is_plugin_initialized = 1 '\001'
        is_discovery_completed = 1 '\001'
        listen_for_events = 1 '\001'
        user_name = 0x1332080 "pmacadmin"
        password = 0x1332990 "********"
        url = 0x0
        oa_fw_buf = "3.71", '\000' <repeats 250 times>
#10 0x00000032e466a6c4 in g_thread_create_proxy (data=0x7f90840008c0) at gthread.c:1897
        thread = 0x7f90840008c0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#11 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#12 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7f9091510700 (LWP 29100)):
#0  0x00000032e320eb5c in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a03c12 in cStreamSock::ReadMsg (this=0x15f5170, type=@0x7f909150fd03, id=@0x7f909150fcfc, payload=0x7f90914ffcf0, payload_len=@0x7f909150fd04, payload_byte_order=@0x7f909150fcf8) at strmsock.cpp:188
        len = 20
        rawhdr = "\001\021\000\000\001\000\000\000\004\000\000"
        dst = 0x7f90914ffc70 "\001\021"
        need = 12
        hdr = "\001\021\000\000\001\000\000\000\004\000\000"
        got = 0
#2  0x0000000000407e6f in service_thread(void*, void*) ()
No symbol table info available.
#3  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x15f5170
        pool = 0x1649820
#4  0x00000032e466a6c4 in g_thread_create_proxy (data=0x162b330) at gthread.c:1897
        thread = 0x162b330
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#5  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7f9091f11700 (LWP 14336)):
#0  0x00000032e320b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e5602352 in g_cond_timed_wait_posix_impl (cond=<value optimized out>, entered_mutex=<value optimized out>, abs_time=<value optimized out>) at gthread-posix.c:242
        result = <value optimized out>
        end_time = {tv_sec = 1441789534, tv_nsec = 638854000}
        timed_out = <value optimized out>
        __PRETTY_FUNCTION__ = "g_cond_timed_wait_posix_impl"
#2  0x00000032e4618751 in g_async_queue_pop_intern_unlocked (queue=0x7f9078001420, try=0, end_time=0x7f9091ef8e10) at gasyncqueue.c:423
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_pop_intern_unlocked"
#3  0x00000032e4618897 in g_async_queue_timed_pop (queue=0x7f9078001420, end_time=0x7f9091ef8e10) at gasyncqueue.c:549
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_timed_pop"
#4  0x0000000000428543 in oh_dequeue_session_event ()
No symbol table info available.
#5  0x0000000000419e0e in saHpiEventGet ()
No symbol table info available.
#6  0x000000000040a23c in process_msg(cHpiMarshal*, int, char*, unsigned int&, unsigned int&) ()
No symbol table info available.
#7  0x0000000000407f52 in service_thread(void*, void*) ()
No symbol table info available.
#8  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x15f2470
        pool = 0x1649820
#9  0x00000032e466a6c4 in g_thread_create_proxy (data=0x15d4ab0) at gthread.c:1897
        thread = 0x15d4ab0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#10 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f9092912700 (LWP 11153)):
#0  0x00000032e320eb5c in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a03c12 in cStreamSock::ReadMsg (this=0x1332ef0, type=@0x7f9092911d03, id=@0x7f9092911cfc, payload=0x7f9092901cf0, payload_len=@0x7f9092911d04, payload_byte_order=@0x7f9092911cf8) at strmsock.cpp:188
        len = 537
        rawhdr = "\001\021\000\000\"\000\000\000\f\000\000"
        dst = 0x7f9092901c70 "\001\021"
        need = 12
        hdr = "\001\021\000\000\"\000\000\000\f\000\000"
        got = 0
#2  0x0000000000407e6f in service_thread(void*, void*) ()
No symbol table info available.
#3  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x1332ef0
        pool = 0x1649820
#4  0x00000032e466a6c4 in g_thread_create_proxy (data=0x15d2c80) at gthread.c:1897
        thread = 0x15d2c80
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#5  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f9093313700 (LWP 11110)):
#0  0x00000032e2ae12e3 in select () from /lib64/libc.so.6
No symbol table info available.
#1  0x00000032e8202a75 in oh_ssl_read (bio=0x7f9080062980, buf=0x7f9093311b90 "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:SOAP-ENC=\"http://www.w3.org/2003/05/soap-encoding\" xmlns:xsi=\"http://www.w3.org"..., size=3999, timeout=40) at oh_ssl.c:717
        ssl = 0x7f90800cf030
        bytes = 0
        readfds = {fds_bits = {256, 0 <repeats 15 times>}}
        writefds = {fds_bits = {0 <repeats 16 times>}}
        tv = {tv_sec = 18, tv_usec = 583691}
        read_wait = 1
        done = 0
        err = 8
        fd = 8
#2  0x00007f9095ec6a12 in soap_message (connection=0x1584860, request=0x15849d8 "<?xml version=\"1.0\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xml"..., doc=0x15849d0) at oa_soap_callsupport.c:696
        nbytes = 873
        ret = 117
        header = 0x7f90800c1060 "\250\001"
        response = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:SOAP-ENC=\"http://www.w3.org/2003/05/soap-encoding\" xmlns:xsi=\"http://www.w3.org"...
        parse = 0x0
#3  0x00007f9095ec7633 in soap_call (connection=0x1584860) at oa_soap_callsupport.c:1085
        session_pos = 0x1584be0 "e334fe0af5e55758</hpoa:oaSessionKey>\n</hpoa:HpOaSessionKeyToken>\n</wsse:Security>\n</SOAP-ENV:Header>\n<SOAP-ENV:Body>\n<hpoa:getAllEventsEx><hpoa:pid>2961</hpoa:pid><hpoa:waitTilEventHappens>1</hpoa:wai"...
        i = 0
        err = 32656
#4  0x00007f9095ec3a22 in soap_getAllEventsEx (con=0x1584860, request=0x7f9093312d30, response=0x7f9093312d20) at oa_soap_calls.c:1601
        ret = 0
#5  0x00007f9095ea0854 in oa_soap_event_thread (oa_pointer=0x1332d30) at oa_soap_event.c:244
        rv = 0
        request = {pid = 2961, waitTilEventHappens = HPOA_TRUE, lcdEvents = HPOA_FALSE, oaFwVersion = 0x7f9093312c10 "3.71"}
        response = {eventInfoArray = 0x0}
        handler = 0x1334a30
        oa = 0x1332d30
        ret_code = -1009
        retry_on_switchover = 0
        oa_handler = 0x13329d0
        is_plugin_initialized = 1 '\001'
        is_discovery_completed = 1 '\001'
        listen_for_events = 1 '\001'
        user_name = 0x1332080 "pmacadmin"
        password = 0x1332990 "********"
        url = 0x0
        oa_fw_buf = "3.71", '\000' <repeats 250 times>
#6  0x00000032e466a6c4 in g_thread_create_proxy (data=0x7f908400ac30) at gthread.c:1897
        thread = 0x7f908400ac30
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#7  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f9094715700 (LWP 11092)):
#0  0x00000032e320b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e4618712 in g_async_queue_pop_intern_unlocked (queue=0x13160f0, try=0, end_time=0x0) at gasyncqueue.c:416
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_pop_intern_unlocked"
#2  0x00000032e4618af6 in g_async_queue_pop (queue=0x13160f0) at gasyncqueue.c:456
        retval = <value optimized out>
        __PRETTY_FUNCTION__ = "g_async_queue_pop"
#3  0x0000000000413620 in oh_process_events ()
No symbol table info available.
#4  0x0000000000428b84 in evtpop_func ()
No symbol table info available.
#5  0x00000032e466a6c4 in g_thread_create_proxy (data=0x16492c0) at gthread.c:1897
        thread = 0x16492c0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#6  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f9095116700 (LWP 11091)):
#0  0x00000032e320e264 in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e3209508 in _L_lock_854 () from /lib64/libpthread.so.0
No symbol table info available.
#2  0x00000032e32093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#3  0x00000032e466b1ec in g_static_rec_mutex_lock (mutex=0x1332728) at gthread.c:1424
        self = {data = "\000g\021\225\220\177\000", dummy_double = 6.9297130181096631e-310, dummy_pointer = 0x7f9095116700, dummy_long = 140258952963840}
        __PRETTY_FUNCTION__ = "g_static_rec_mutex_lock"
#4  0x0000000000415568 in oh_get_handler ()
No symbol table info available.
#5  0x0000000000412870 in oh_harvest_events ()
No symbol table info available.
#6  0x0000000000428a1d in evtget_func ()
No symbol table info available.
#7  0x00000032e466a6c4 in g_thread_create_proxy (data=0x160a2f0) at gthread.c:1897
        thread = 0x160a2f0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#8  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f9095b17700 (LWP 11090)):
#0  0x00000032e320b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e5602352 in g_cond_timed_wait_posix_impl (cond=<value optimized out>, entered_mutex=<value optimized out>, abs_time=<value optimized out>) at gthread-posix.c:242
        result = <value optimized out>
        end_time = {tv_sec = 1441789699, tv_nsec = 702865000}
        timed_out = <value optimized out>
        __PRETTY_FUNCTION__ = "g_cond_timed_wait_posix_impl"
#2  0x00000000004288e8 in discovery_func ()
No symbol table info available.
#3  0x00000032e466a6c4 in g_thread_create_proxy (data=0x15fa2c0) at gthread.c:1897
        thread = 0x15fa2c0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#4  0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f90961267e0 (LWP 11030)):
#0  0x00000032e320ea5d in accept () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000032e6a04a3c in cServerStreamSock::Accept (this=0x14cb040) at strmsock.cpp:567
        sock = 0
#2  0x0000000000407c60 in oh_server_run(int, char const*, unsigned short, unsigned int, int) ()
No symbol table info available.
#3  0x0000000000412420 in main ()
No symbol table info available.

Thread 1 (Thread 0x7f9090b0f700 (LWP 29199)):
#0  0x00000032e9241718 in SSL_shutdown () from /usr/lib64/libssl.so.10
No symbol table info available.
#1  0x00000032e8202634 in oh_ssl_disconnect (bio=0x7f907c0e6480, shutdown=OH_SSL_BI) at oh_ssl.c:594
        ssl = 0x32e466c7f0
        ret = 1
        fd = 0
#2  0x00007f9095ec6d1f in soap_message (connection=0x1334ec0, request=0x1335038 "", doc=0x1335030) at oa_soap_callsupport.c:757
        nbytes = 2476
        ret = 4
        header = 0x14b8120 ""
        response = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:SOAP-ENC=\"http://www.w3.org/2003/05/soap-encoding\" xmlns:xsi=\"http://www.w3.org"...
        parse = 0x14b8860
#3  0x00007f9095ec7633 in soap_call (connection=0x1334ec0) at oa_soap_callsupport.c:1085
        session_pos = 0x1335240 "d1c33f9ac4d9b888</hpoa:oaSessionKey>\n</hpoa:HpOaSessionKeyToken>\n</wsse:Security>\n</SOAP-ENV:Header>\n<SOAP-ENV:Body>\n<hpoa:getPowerCapConfig></hpoa:getPowerCapConfig>\n</SOAP-ENV:Body>\n</SOAP-ENV:Envel"...
        i = 0
        err = 5
#4  0x00007f9095ec4db2 in soap_setBladePower (con=0x1334ec0, request=0x7f9090af7110) at oa_soap_calls.c:1984
        power = "RESET\000\000\000\320)3\001\000\000\000"
#5  0x00007f9095e85d4b in oa_soap_set_reset_state (oh_handler=0x1334a30, resource_id=5, action=SAHPI_WARM_RESET) at oa_soap_reset.c:222
        rv = 0
        tmp = SAHPI_POWER_ON
        rpt = 0x7f90840422b0
        bay_number = 10
        server_request = {bayNumber = 10, power = RESET}
        interconnect_request = {bayNumber = -1867550432}
        handler = 0x1334a30
        oa_handler = 0x13329d0
#6  0x00000000004278eb in saHpiResourceResetStateSet ()
No symbol table info available.
#7  0x0000000000410acd in process_msg(cHpiMarshal*, int, char*, unsigned int&, unsigned int&) ()
No symbol table info available.
#8  0x0000000000407f52 in service_thread(void*, void*) ()
No symbol table info available.
#9  0x00000032e466c60b in g_thread_pool_thread_proxy (data=<value optimized out>) at gthreadpool.c:319
        task = 0x14b0800
        pool = 0x1649820
#10 0x00000032e466a6c4 in g_thread_create_proxy (data=0x1626bf0) at gthread.c:1897
        thread = 0x1626bf0
        __PRETTY_FUNCTION__ = "g_thread_create_proxy"
#11 0x00000032e32079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#12 0x00000032e2ae886d in clone () from /lib64/libc.so.6
No symbol table info available.

Reported by: taclark

openhpi2 commented 8 years ago

Original comment by: taclark

openhpi2 commented 8 years ago

Deleted the description and reposted

Original comment by: dr_mohan

openhpi2 commented 8 years ago

Hi,

As mentioned in the bug description, I upgraded the OpenHPI to 3.4.0 followed by a reboot, but do not see any core dump.

Any steps in order to reproduce the issue would be helpful for me.

Could you please do let us know that exactly what operation was being performed at the time of core observed?

Here what is the meaning of client application upgrade? What exactly do you do over here?

Could you please let us know the below: OA version OS version SSL version Glib version

Thanks! Praveen

Original comment by: openhpi2

openhpi2 commented 8 years ago

If possible could you please try your steps with OpenHPI 3.6.0 and let us know the outcome?

Original comment by: openhpi2

openhpi2 commented 8 years ago

Praveen,

We've never had a reproducer for this crash. It hasn't been seen by us or by our customer since I reported it here.

But to answer your questions, the OS where the problem was originally seen was Oracle Linux 6.5, based on RHEL 6.5, with openssl 1.0.1e and glibc 2.12. The OA hardware and firmware versions are not known. Our application upgrade amounts to moving the system to a runlevel that stops openhpid and the application using it, replaces files with new versions using RPM upgrades, and reboots.

As for OpenHPI 3.6.0, back in late January we updated from 3.4.0 to 3.6.1 in our development builds. Our post-build testing includes various upgrades, and we haven't seen the problem there either.

So it's all good news so far. . . .

Tim Clark

Original comment by: taclark

openhpi2 commented 8 years ago

Thanks Tim for your update.

As we too are unable to re-create this issue with OpenHPI 3.6.1, shall I go ahead and close this issue as of now.

In future, in case this occurs then we can open it.

Thanks! Praveen

Original comment by: openhpi2

openhpi2 commented 8 years ago

Original comment by: openhpi2