Connection refused when setting port to zero and inside a docker container

[KGHVerilator]

I am trying to develop a test email server that should run in a docker container. The docker container is run with --network=host -P; though for unit tests I would like to use --network=none.

I am deliberately specifying that port 0 should be used (let the OS pick an available port). I then have a function to get the port that was actually used. The server code is

#! /usr/bin/env python3

import asyncio #pylint: disable=unused-import
from aiosmtpd.controller import Controller

class DemoEmailHandler:
    async def handle_DATA(self, server, session, envelope):
        message = {
            'peer' : session.peer,
            'mailfrom' : envelope.mail_from,
            'to': envelope.rcpt_tos,
            'data' : envelope.content}
        print(message)
        return '250 Message accepted for delivery'

class DemoEmailServer:
    def __init__(self):
        self._handler = DemoEmailHandler()
        self._server = Controller(
            handler=self._handler,
            hostname="0.0.0.0",
            port=0
        )
        self._server.start()
        self._port = self._server.server.sockets[0].getsockname()[1]

    def stop(self):
        """Tell the server to stop"""
        self._server.stop()

    def getPort(self) -> int:
        """Return the port the SMTP (email) server is actually on"""
        return self._port

SERVER = DemoEmailServer()
print(SERVER.getPort())
SERVER.stop()

I keep getting connection refused

Traceback (most recent call last):
  File "./demo_email_server.py", line 36, in <module>
    SERVER = DemoEmailServer()
  File "./demo_email_server.py", line 25, in __init__
    self._server.start()
  File "/usr/local/lib/python3.8/dist-packages/aiosmtpd-1.4.2-py3.8.egg/aiosmtpd/controller.py", line 223, in start
    self._trigger_server()
  File "/usr/local/lib/python3.8/dist-packages/aiosmtpd-1.4.2-py3.8.egg/aiosmtpd/controller.py", line 313, in _trigger_server
    s = stk.enter_context(create_connection((hostname, self.port), 1.0))
  File "/usr/lib/python3.8/socket.py", line 808, in create_connection
    raise err
  File "/usr/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

when running inside a docker container. If I run on the host then the server starts up fine.

(My host and container are both Ubuntu 20.04, python 3.8.5, the container has aiosmtpd 1.4.2 installed from source).

[pepoluan]

Ah, that's a new use case for me (setting port to 0)

Currently the code for Controller tries to connect to the specified port to trigger the server to activate once, in order to ensure that SMTP initializes correctly. It is a simple mechanism and currently does not consider the special port=0 case.

I'll see if I can add more intelligence to _trigger_server()

[KGHVerilator]

I like your package and find it very useful, thank you for your work.

I have some python programming experience, but am only mediocre at it. I can help some with testing changes. There is some documentation that is not clear to me with setting up an SSL and TLS email server, if that is updated I would love to try it and say how it goes; assuming you are interested in that stuff. If this works in my docker containers, I can test stuff like that without worrying about my development environment.

It has been a while since I looked at this. I may repeat a bit of what was in the log item. I will also try to give you my full use case, so you get my idea (good or bad) of how I think your software can be used. Up to you what you do with it.

I am trying to test embedded devices that can send emails. They connect to an email server to store and forward the emails.

I have a shared computer used in the tests. The embedded device is given the IP address and port of the email server on the shared computer.

1. Since lots of developers can be using the shared computer (including jenkins), I need the developer to startup, run, and shutdown the email server using a random port that is not privileged.

2. My test system needs to be contained and not dependent on the host system. Some of the products develop for a while then go dormant, only a few years later to be resurrected, hence containers with the test environment (and build environment).

3. My device needs to do three different kinds of email:    - plain TCP/IP socket with everything in the clear (yuck but I am stuck with it)    - SSL server email     -TLS server for the email. I have been able to get TCP going, but the SSL and TLS servers are not working yet.

4. I have also tried setting up a subsystem test of my email server. The container running the email server also runs an email client with network=none (127.0.0.1 works). This allows us to have tests of the common code (email server) that is not dependent on any device. Again I got unencrypted working but not encrypted.

5. Our emails or both traditional text only emails, and HTML emails. I have not tried extracting the pictures we attach to our HTML emails yet, only working with text only ones.

Kendrick

On 2021-10-19 12:19 a.m., Pandu E POLUAN wrote:

Ah, that's a new use case for me (setting port to 0)

Currently the code for |Controller| tries to connect to the specified port https://urldefense.com/v3/__https://github.com/aio-libs/aiosmtpd/blob/master/aiosmtpd/controller.py*L416-L429__;Iw!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5W_suHBw$ to trigger the server to activate once. It is a simple mechanism and does not consider the special |port=0| case.

I'll see if I can add more intelligence to |_trigger_server()|

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/aio-libs/aiosmtpd/issues/276*issuecomment-946401243__;Iw!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5akcxw-A$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AR3ZTVLW3XVFPVDQACMPVODUHUEX7ANCNFSM47EI37RQ__;!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5yHTvXEw$. Triage notifications on the go with GitHub Mobile for iOS https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5Hv8A5Kg$ or Android https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf7Bp4KCww$.

[waynew]

FWIW you can check out how I create an ssl server https://github.com/waynew/orouboros/blob/89905978a423e130e9ea2c57690b2e4a6cc48f3d/orouboros.py#L313

[diazona]

I've also run into this problem while trying to prepare pytest-dev/pytest-localserver for the removal of smtpd. I wanted to bump this issue to bring it some attention.

I'll probably wind up regretting saying this :stuck_out_tongue: but it seems like a simple fix, just replace create_connection((hostname, self.port), 1.0) with create_connection(self.server_coro.sockets[0].getsockname(), 1.0), or something like that. Would you be open to accepting a pull request for that change @pepoluan? I'd be happy to prepare one, or at least put in a chunk of time to try to figure out a solution.

[diazona]

...and almost immediately I figured out why it's not that simple, with the mixing of synchronous and asynchronous code :laughing: I knew I shouldn't have said anything. But I would still be quite willing to put some time into this, if it'd help.

[waynew]

Okay, so.... I know it's been a very long time since I've touched this, but - yeah, the fix is pretty straightforward, I think.

First off, yeah I was able to reproduce this.

Second off, I think the correct thing for us to do here is in the _create_server function where we do self.loop.create_server we could change it to:

server = self.loop.create_server(...)
if not self.port:
    self.port = server.sockets[0].getsockname()[1]
return server

By making that change locally, it all worked :+1: We'll want some test cases for that obviously, but it shouldn't be too hard to do!

[waynew]

Well, heck! The fix was pretty easy, though I did have to move the change to _trigger_server.

But I'm struggling with the test cases -- I think we actually have some tests that are failing on some more modern Pythons anyway :thinking:

[KGHVerilator]

Hopefully this will work out; thank you for look into it.

[alext]

@waynew I was also hitting this, and I found the fix you describe worked for me as well, so I've raised it as #381.