two c AMQP_VALUE are cloned while not being destroyed:
cdef c_amqpvalue.AMQP_VALUE cloned_source
cdef c_amqpvalue.AMQP_VALUE cloned_target
destroying them in the finally could correctly decrease the reference count
however random segmentation fault will be triggered.
after keeping investigation into how the properties on the source/targer are used, the address property is returned by source_get_address which is a in_memory opeartion (value not cloned, reference count doesn't + 1), that seems to be place where seg fault took place
it cannot be simply wrapped by the cython AMQPValue (via value_factory), as AMQPValue will call destroy (reference count - 1) when GC kicks in. the reference count will be incorrect at that time.
it should be cloned first, we have other similar codes in amqpvalue.pyx when C *_in_memory methods are called
after fixing this, no seg fault happens
I further searched where value_factory methods are used and found two more places that need fix
the pop method in cdef class CompositeValue(AMQPValue)
the get_body_sequence method in cdef class cMessage
the steps I currently use:
use valgrind to run E2E script first
isolate the memory leak issue at different protocol level (Connection/Session/Link)
write separate scripts to test 1) Connection 2) Connection + Session 3) Connection + Session + Link
use valgrind to run separate scripts
current investigation result:
memory leak DOES NOT happen with the pure AMQP Connection (no Session, no Link)
from valgrind logging, it's the list item within a described value get leaked
the item is instantiate by the C library
the item is passed upward to the cython/python layer via callbacks
which indicates that there could be some properties on session/link get leaked
So far the most suspicious part is within the session.pyx:
The cloned two AMQP value cloned_source and cloned_target in the on_link_attached callback defined in session.pyx looks like they're not destroyed.
Problem:
Root cause:
Client creation would leak memory after connection and session is established because
on_link_attachedcallback insession.pyx,cloned_sourceandcloned_targetare not destroyed.How to Fix:
cloned_sourceandcloned_targetin afinallyblock in the callback.More:
value_factoryare not used properly. fixed them at the same time.Analysis:
finallycould correctly decrease the reference countaddressproperty is returned by source_get_address which is a in_memory opeartion (value not cloned, reference count doesn't + 1), that seems to be place where seg fault took placevalue_factory), as AMQPValue will call destroy (reference count - 1) when GC kicks in. the reference count will be incorrect at that time.popmethod in cdef class CompositeValue(AMQPValue)get_body_sequencemethod in cdef class cMessagethe steps I currently use:
current investigation result:
So far the most suspicious part is within the
session.pyx:The cloned two AMQP value
cloned_sourceandcloned_targetin theon_link_attachedcallback defined insession.pyxlooks like they're not destroyed.