Hard to reproduce. I have two applications communicating with zmq. I restart one of them regularly with watch -n5 systemctl restart, and in the other one, the segfault occurs after some time.
Applications communicate over PUB/SUB and REQ/RESP, both use socket monitoring.
What's the actual result? (include assertion message & call stack if applicable)
The attached logs show a gdb run of the segfaulting app, with log statements inserted into zmq.
In the logs, i replaced the address of segfaulting pointer with [SEGFAULT_POINTER] for readability, similar procedure for thread ids.
Thread 3 "[application-name]" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x[ZMQ_IO_THREAD] (LWP 1590)]
0x00000000 in ?? ()
(gdb) thread apply all bt
Thread 3 (Thread 0x[ZMQ_IO_THREAD] (LWP 1590)):
#0 0x00000000 in ?? ()
#1 0xb6702514 in zmq::object_t::process_command (this=0x[SEGFAULT_POINTER], cmd_=...) at libzmq/src/object.cpp:117
Please look at the end of attached file for complete stacktraces.
Assumed Problem
Race condition between CMD pipe_term_ack sent from zmq IO thread to application thread, and send_stats_to_peerexecuting in application thread
(number in braces is line number in attached log fle)
More precisely:
(4202) Something sends pipe_term to application thread
(4205) Application thread in process_term() sends pipe_term_ack to zmq IO thread
(4211) IO thread executes pipe_term_ack, (4226) sending pipe_term_ack back to application thread, and
(4229) destroys pipe object so that later access produces observed segfault.
Issue description
Segfault involving: https://github.com/zeromq/libzmq/blob/bd6fa4bbb3ec775d6ff9df0e1bb3174254daffa4/src/object.cpp#L111 near a socket close.
Potentially related issue: https://github.com/zeromq/libzmq/issues/3446
Environment
Minimal test code / Steps to reproduce the issue
Hard to reproduce. I have two applications communicating with zmq. I restart one of them regularly with watch -n5 systemctl restart, and in the other one, the segfault occurs after some time. Applications communicate over PUB/SUB and REQ/RESP, both use socket monitoring.
What's the actual result? (include assertion message & call stack if applicable)
The attached logs show a gdb run of the segfaulting app, with log statements inserted into zmq.
In the logs, i replaced the address of segfaulting pointer with [SEGFAULT_POINTER] for readability, similar procedure for thread ids.
DOWNLOAD_LOGFILE snippet:
Please look at the end of attached file for complete stacktraces.
Assumed Problem
Race condition between CMD
pipe_term_ack
sent from zmq IO thread to application thread, andsend_stats_to_peer
executing in application thread (number in braces is line number in attached log fle) More precisely:pipe_term
to application threadprocess_term()
sendspipe_term_ack
to zmq IO threadpipe_term_ack
, (4226) sendingpipe_term_ack
back to application thread, andpipe_term_ack
command and so the segfault pointer is still listed as a peer for some pipe in the _pipes list that is looped over here:https://github.com/zeromq/libzmq/blob/3cafc0c26033e1d84ad9887ca4ddd99b934c311f/src/socket_base.cpp#L1602 Producing here https://github.com/zeromq/libzmq/blob/e86237da5845cbe0a7da626e7eb1071a76464ce2/src/pipe.cpp#L586 the request (4235) with the segfaulting pointer asdestination
pipe_peer_stats
, with invalid pointer asdestination
, producing segfault.Note that the line numbers given in the log after the function name will be off because of insertion of log statements.
What's the expected result?
No segfault.
Note
If this is really a race condition, https://github.com/zeromq/libzmq/blob/3cafc0c26033e1d84ad9887ca4ddd99b934c311f/src/socket_base.cpp#L1609 can also potentially be vulnerable to this because it also loops over the currently registered pipes?
I am happy to provide additional info if needed.