Some tests in aiosmtpd/tests/test_server.py hang with python 3.12

[kapouer]

This is on debian/trixie, using aiosmtpd 1.4.4.post2-1 or 1.4.3. It hangs even when there is an actual online network (it also hangs in a restricted container).

This one for example:

python3.12 -m pytest -v -k test_unknown_args_direct 
/usr/lib/python3/dist-packages/nose/config.py:142: SyntaxWarning: invalid escape sequence '\.'
  """nose configuration.
Test session starts (platform: linux, Python 3.12.1, pytest 7.4.4, pytest-sugar 0.9.7)
cachedir: .pytest_cache
rootdir: /home/dev/Software/debian/python-aiosmtpd/salsa
configfile: pyproject.toml
plugins: click-1.1.0, sugar-0.9.7, env-0.8.2, pyfakefs-5.3.2, asyncio-0.20.3, order-1.0.1, subtests-0.11.0, xdist-3.4.0, timeout-2.2.0, mock-3.12.0, flaky-3.7.0, case-1.5.3, django-4.5.2
asyncio: mode=Mode.STRICT
collected 565 items / 564 deselected / 1 selected

Works with python3.11 Maybe it's because some plugin is not up-to-date ?

I played with it a little and any clue might unblock the situation.

[cuu508]

I think I figured out what is going wrong here.

test_unknown_args_direct passes an unexpected keyword argument to Controller and checks if an exception is thrown. That part works! But afterwards the test does clean up by calling cont.stop() which then hangs.

While closing the controller, the execution gets stuck waiting on self.server.wait_closed() here.

Looking at wait_closed() source I think we can see why this used to work in 3.11 but does not work in 3.12 any more:

    Historical note: In 3.11 and before, this was broken, returning
    immediately if the server was already closed, even if there
    were still active connections. An attempted fix in 3.12.0 was
    still broken, returning immediately if the server was still
    open and there were no active connections. Hopefully in 3.12.1
    we have it right.

The active connection is the one made by _trigger_server. _trigger_server makes a dummy connection and reads from it, to force the SMTP server to fully initialize.

If an exception is thrown during SMTP server initialization, an instance of _FakeServer is initialized and used instead (see _factory_invoker). And _FakeServer is I think where the actual problem is. It does not seem to have any code to close the client connections. So the connection initiated by _trigger_server stays hanging, wait_closed() stays hanging, and the whole test suite stays hanging.

As an experiment, I added the following to _FakeServer:

    def connection_made(self, transport):
        transport.close()

The idea was to unconditionally close client connections right after they've been opened. I'm new to all of this, so perhaps this is not the right way to do it. But it seems to work, with this in place the test_unknown_args_direct test now passes.

PS. This is not just a test environment problem, the hanging also happens in normal code. Here's an isolated example (run with py 3.12+):

from aiosmtpd.controller import Controller
from aiosmtpd.handlers import Sink

controller = Controller(Sink(), a=1)
try:
    controller.start()
except Exception as e:
    pass

print("This next line will block indefinitely:")
controller.stop()