We're seeing aiocoap 0.4.7 fail its tests on Python 3.12.2. They work fine on 3.11.8. Specifically the tcp server tests are not being garbage collected, which makes them error out.
Environment
Python version: 3.12.2 (main, Feb 6 2024, 20:19:44) [GCC 13.2.0]
aiocoap version: 0.4.7
Modules missing for subsystems:
dtls: missing DTLSSocket
oscore: missing cbor2, cryptography, filelock, ge25519
linkheader: everything there
prettyprint: missing cbor2, termcolor, pygments
Python platform: linux
Default server transports: tcpserver:tcpclient:tlsserver:tlsclient:udp6
Selected server transports: tcpserver:tcpclient:tlsserver:tlsclient:udp6
Default client transports: tcpclient:tlsclient:udp6
Selected client transports: tcpclient:tlsclient:udp6
SO_REUSEPORT available (default, selected): True, True
List of failing tests
FAILED tests/test_server.py::TestServerTCP::test_big_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_empty_accept - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_error_resources - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_fast_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_js_accept - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_manualbig_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_nonexisting_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_replacing_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_root_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_slow_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_slowbig_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_spurious_resource - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
FAILED tests/test_server.py::TestServerTCP::test_unacceptable_accept - AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.tran...
Example
aiocoap> ____________________ TestServerTCP.test_unacceptable_accept ____________________
aiocoap>
aiocoap> self = <tests.test_server.TestServerTCP testMethod=test_unacceptable_accept>
aiocoap> attribute = {'del': <function WithTestServer.tearDown.<locals>.<lambda> at 0x7ffff50dcae0>, 'get': <function WithTestServer.tearDo...0dcc20>, 'label': 'request_interfaces[<TokenManager for <aiocoap.transports.tcp.TCPServer object at 0x7ffff51c6b10>>]'}
aiocoap>
aiocoap> def _del_to_be_sure(self, attribute):
aiocoap> if isinstance(attribute, str):
aiocoap> getter = lambda self, attribute=attribute: getattr(self, attribute)
aiocoap> deleter = lambda self, attribute=attribute: delattr(self, attribute)
aiocoap> label = attribute
aiocoap> else:
aiocoap> getter = attribute['get']
aiocoap> deleter = attribute['del']
aiocoap> label = attribute['label']
aiocoap> weaksurvivor = weakref.ref(getter(self))
aiocoap> deleter(self)
aiocoap>
aiocoap> if not is_test_successful(self):
aiocoap> # An error was already logged, and that error's backtrace usually
aiocoap> # creates references that make any attempt to detect lingering
aiocoap> # references fuitile. It'll show an error anyway, no use in
aiocoap> # polluting the logs.
aiocoap> return
aiocoap>
aiocoap> # let everything that gets async-triggered by close() happen
aiocoap> self.loop.run_until_complete(asyncio.sleep(CLEANUPTIME))
aiocoap> gc.collect()
aiocoap>
aiocoap> def snapshot():
aiocoap> # This object is created locally and held by the same referrers
aiocoap> # that also hold the now-recreated survivor.
aiocoap> #
aiocoap> # By comparing its referrers to the surviver's referrers, we can
aiocoap> # filter out this tool's entry in the already hard to read list of
aiocoap> # objects that kept the survivor alive.
aiocoap> canary = object()
aiocoap> survivor = weaksurvivor()
aiocoap> if survivor is None:
aiocoap> return None
aiocoap>
aiocoap> all_referrers = gc.get_referrers(survivor)
aiocoap> canary_referrers = gc.get_referrers(canary)
aiocoap> if canary_referrers:
aiocoap> referrers = [r for r in all_referrers if r not in canary_referrers]
aiocoap> assert len(all_referrers) == len(referrers) + 1, "Canary to filter out the debugging tool's reference did not work.\nReferrers:\n%s\ncanary_referrers:\n%s" % (pprint.pformat(all_referrers), pprint.pformat(canary_referrers))
aiocoap> else:
aiocoap> # There is probably an optimization around that makes the
aiocoap> # current locals not show up as referrers. It is hoped (and
aiocoap> # least with the current Python it works) that this also works
aiocoap> # for the survivor, so it's already not in the list.
aiocoap> referrers = all_referrers
aiocoap>
aiocoap> def _format_any(frame, survivor_id):
aiocoap> if str(type(frame)) == "<class 'frame'>":
aiocoap> return _format_frame(frame, survivor_id)
aiocoap>
aiocoap> if isinstance(frame, dict):
aiocoap> # If it's a __dict__, it'd be really handy to know whose dict that is
aiocoap> framerefs = gc.get_referrers(frame)
aiocoap> owners = [o for o in framerefs if getattr(o, "__dict__", None) is frame]
aiocoap> if owners:
aiocoap> return pprint.pformat(frame) + "\n ... which is the __dict__ of %s" % (owners,)
aiocoap>
aiocoap> return pprint.pformat(frame)
aiocoap>
aiocoap> def _format_frame(frame, survivor_id):
aiocoap> return "%s as %s in %s" % (
aiocoap> frame,
aiocoap> " / ".join(k for (k, v) in frame.f_locals.items() if id(v) == survivor_id),
aiocoap> frame.f_code)
aiocoap>
aiocoap> # can't use survivor in list comprehension, or it would be moved
aiocoap> # into a builtins.cell rather than a frame, and that won't spew out
aiocoap> # the details _format_frame can extract
aiocoap> survivor_id = id(survivor)
aiocoap> referrer_strings = [
aiocoap> _format_any(x, survivor_id) for x in
aiocoap> referrers]
aiocoap> formatted_survivor = pprint.pformat(vars(survivor))
aiocoap> return "Survivor found: %r\nReferrers of the survivor:\n*"\
aiocoap> " %s\n\nSurvivor properties: %s" % (
aiocoap> survivor,
aiocoap> "\n* ".join(referrer_strings),
aiocoap> formatted_survivor)
aiocoap>
aiocoap> s = snapshot()
aiocoap>
aiocoap> if s is not None:
aiocoap> original_s = s
aiocoap> if False: # enable this if you think that a longer timeout would help
aiocoap> # this helped finding that timer cancellations don't free the
aiocoap> # callback, but in general, expect to modify this code if you
aiocoap> # have to read it; this will need adjustment to your current
aiocoap> # debugging situation
aiocoap> logging.root.info("Starting extended grace period")
aiocoap> for i in range(10):
aiocoap> self.loop.run_until_complete(asyncio.sleep(1))
aiocoap> gc.collect()
aiocoap> s = snapshot()
aiocoap> if s is None:
aiocoap> logging.root.info("Survivor vanished after %r iterations", i + 1)
aiocoap> break
aiocoap> snapshotsmessage = "Before extended grace period:\n" + original_s + "\n\nAfter extended grace period:\n" + ("the same" if s == original_s else s)
aiocoap> else:
aiocoap> snapshotsmessage = s
aiocoap> errormessage = "Protocol %s was not garbage collected.\n\n" % label + snapshotsmessage
aiocoap> > self.fail(errormessage)
aiocoap> E AssertionError: Protocol request_interfaces[<TokenManager for <aiocoap.transports.tcp.TCPServer object at 0x7ffff51c6b10>>] was not garbage collected.
aiocoap> E
aiocoap> E Survivor found: <TokenManager for <aiocoap.transports.tcp.TCPServer object at 0x7ffff51c6b10>>
aiocoap> E Referrers of the survivor:
aiocoap> E * <coroutine object TokenManager.shutdown at 0x7ffff50b5d40>
aiocoap> E
aiocoap> E Survivor properties: {'_token': 36361,
aiocoap> E 'context': <aiocoap.protocol.Context object at 0x7ffff51c6900>,
aiocoap> E 'incoming_requests': None,
aiocoap> E 'log': <Logger coap-server (NOTSET)>,
aiocoap> E 'loop': <_UnixSelectorEventLoop running=False closed=False debug=False>,
aiocoap> E 'outgoing_requests': None,
aiocoap> E 'token_interface': <aiocoap.transports.tcp.TCPServer object at 0x7ffff51c6b10>}
aiocoap>
aiocoap> tests/fixtures.py:283: AssertionError
We're seeing aiocoap 0.4.7 fail its tests on Python 3.12.2. They work fine on 3.11.8. Specifically the tcp server tests are not being garbage collected, which makes them error out.
Environment
List of failing tests
Example