two c AMQP_VALUE are cloned while not being destroyed:
cdef c_amqpvalue.AMQP_VALUE cloned_source
cdef c_amqpvalue.AMQP_VALUE cloned_target
destroying them in the finally could correctly decrease the reference count
however random segmentation fault will be triggered.
after keeping investigation into how the properties on the source/targer are used, the address property is returned by source_get_address which is a in_memory opeartion (value not cloned, reference count doesn't + 1), that seems to be place where seg fault took place
it cannot be simply wrapped by the cython AMQPValue (via value_factory), as AMQPValue will call destroy (reference count - 1) when GC kicks in. the reference count will be incorrect at that time.
it should be cloned first, we have other similar codes in amqpvalue.pyx when C *_in_memory methods are called
after fixing this, no seg fault happens
I further searched where value_factory methods are used and found two more places that need fix
the pop method in cdef class CompositeValue(AMQPValue)
the get_body_sequence method in cdef class cMessage
the steps I currently use:
use valgrind to run E2E script first
isolate the memory leak issue at different protocol level (Connection/Session/Link)
write separate scripts to test 1) Connection 2) Connection + Session 3) Connection + Session + Link
use valgrind to run separate scripts
current investigation result:
memory leak DOES NOT happen with the pure AMQP Connection (no Session, no Link)
from valgrind logging, it's the list item within a described value get leaked
the item is instantiate by the C library
the item is passed upward to the cython/python layer via callbacks
which indicates that there could be some properties on session/link get leaked
So far the most suspicious part is within the session.pyx:
The cloned two AMQP value cloned_source and cloned_target in the on_link_attached callback defined in session.pyx looks like they're not destroyed.
Problem:
Root cause:
Client creation would leak memory after connection and session is established because
on_link_attached
callback insession.pyx
,cloned_source
andcloned_target
are not destroyed.How to Fix:
cloned_source
andcloned_target
in afinally
block in the callback.More:
value_factory
are not used properly. fixed them at the same time.Analysis:
finally
could correctly decrease the reference countaddress
property is returned by source_get_address which is a in_memory opeartion (value not cloned, reference count doesn't + 1), that seems to be place where seg fault took placevalue_factory
), as AMQPValue will call destroy (reference count - 1) when GC kicks in. the reference count will be incorrect at that time.pop
method in cdef class CompositeValue(AMQPValue)get_body_sequence
method in cdef class cMessagethe steps I currently use:
current investigation result:
So far the most suspicious part is within the
session.pyx
:The cloned two AMQP value
cloned_source
andcloned_target
in theon_link_attached
callback defined insession.pyx
looks like they're not destroyed.