Traceback on python 3.6 when running together with celery_worker fixture

[trunet]

Hello,

I'm trying to use ftpserver mock to test some parts of my code. It's running together with celery_worker fixture, as I need an ftpserver available to send files from a celery task

The test succeeds, even though the traceback is showing.

Using:

platform darwin -- Python 3.6.8, pytest-4.0.2, py-1.7.0, pluggy-0.8.1 -- .tox/py36/bin/python
cachedir: .tox/py36/.pytest_cache
rootdir: ., inifile: tox.ini
plugins: localftpserver-0.5.1, flake8-1.0.4, celery-4.2.1

If I ran the tests just after another one, I got multiple OSError: [Errno 48] Address already in use. First is a success (with the traceback):

tests/test_basic.py::test_parse_duct_yaml PASSED                                                                                                                                                                    tests/test_basic.py::test_client_structure PASSED
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File ".tox/py36/lib/python3.6/site-packages/pytest_localftpserver/plugin.py", line 986, in run
    self._server.serve_forever()
  File ".tox/py36/lib/python3.6/site-packages/pyftpdlib/servers.py", line 213, in serve_forever
    self.ioloop.loop(timeout, blocking)
  File ".tox/py36/lib/python3.6/site-packages/pyftpdlib/ioloop.py", line 348, in loop
    poll(soonest_timeout)
  File ".tox/py36/lib/python3.6/site-packages/pyftpdlib/ioloop.py", line 709, in poll
    timeout)
OSError: [Errno 9] Bad file descriptor

tests/test_http.py::test_home PASSED
tests/test_http.py::test_organization PASSED
tests/test_http.py::test_non_existant_organization PASSED
tests/test_http.py::test_account PASSED
tests/test_http.py::test_non_existant_account PASSED
tests/test_tasks.py::test_scan_deliverables PASSED
tests/test_tasks.py::test_send_files PASSED

Second, sometimes it's the same as above, but if it's right after the previous run, I got:

ests/test_basic.py::test_parse_duct_yaml ERROR
tests/test_basic.py::test_client_structure ERROR
tests/test_http.py::test_home ERROR
tests/test_http.py::test_organization ERROR
tests/test_http.py::test_non_existant_organization ERROR
tests/test_http.py::test_account ERROR
tests/test_http.py::test_non_existant_account ERROR
tests/test_tasks.py::test_scan_deliverables ERROR
tests/test_tasks.py::test_send_files ERROR
..... [ALL ERRORS THE SAME AS BELOW] .....
self = <pytest_localftpserver.plugin.SimpleFTPServer at 0x10649d1d0>, username = 'ducttestuser', password = 'ducttestpwd', ftp_home = '.tox/py36/tmp', ftp_port = 31175

    def __init__(self, username, password, ftp_home=None, ftp_port=0):
        # Create temp directories for the anonymous and authenticated roots
        self._anon_root = tempfile.mkdtemp(prefix="anon_root_")
        if not ftp_home:
            self.temp_ftp_home = True
            self._ftp_home = tempfile.mkdtemp(prefix="ftp_home_")
        else:
            self.temp_ftp_home = False
            self._ftp_home = ftp_home
        self.username = username
        self.password = password
        authorizer = DummyAuthorizer()
        authorizer.add_user(self.username, self.password, self.ftp_home,
                            perm='elradfmwM')
        authorizer.add_anonymous(self.anon_root)

        handler = FTPHandler
        handler.authorizer = authorizer
        # Create a socket on any free port
        self._ftp_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>       self._ftp_socket.bind(('127.0.0.1', ftp_port))
E       OSError: [Errno 48] Address already in use

authorizer = <pyftpdlib.authorizers.DummyAuthorizer object at 0x10649d0b8>
ftp_home   = '.tox/py36/tmp'
ftp_port   = 31175
handler    = <class 'pyftpdlib.handlers.FTPHandler'>
password   = 'ducttestpwd'
self       = <pytest_localftpserver.plugin.SimpleFTPServer at 0x10649d1d0>
username   = 'ducttestuser'

.tox/py36/lib/python3.6/site-packages/pytest_localftpserver/plugin.py:64: OSError

Can you help me with that?

[s-weigand]

Hi there, from the output I guess that you specifyed the port as in the example in the readme ( FTP_PORT=31175 ). As I understand it celery runs tasks in parallel, so there is already an ftp instance listening on port 31175 and there can't be more than one task listening on the same port, which is why it raises an error.

Possible solution:

1. remove FTP_PORT=31175 from your config, like that the port should get chosen as any free port
2. don't hardcode the port but read it from the fixture ftpserver.get_login_data()["port"]

If that doesn't help, please share your tox.ini / pytest.ini and maybe make a minimal example to reproduce the error.

P.S.: Please keep us in touch if it works, so we can add it to the docs 😄

[oz123]

It's something I have seen too while using a Mac, like you do. It'll be hard to weed out since I don't use a Mac and @s-weigand uses windows. See if using the process based instead of thread based class works better.

[trunet]

Thanks for the prompt reply, I appreciate it.

@s-weigand yes, celery runs in parallel, but I think not the tests, they should be serial. I'm using a function scope fixture (celery_worker). Per documentation, it starts a worker, run the task and stop the worker after the test function ends.

I followed your suggestion 1. 2, were already in place. Now I don't get the error after running one test right after the other. Somehow the ftp port stays listening for around 10 seconds after the test finishes.

@oz123 I understand. I tried to set USE_PROCESS = True for darwin (osx) as well, and the test hangs.

I will do some more troubleshooting tomorrow. A question I have is why did you implemented your own Thread/Process class instead of using pyftpdlib.servers classes?

[s-weigand]

@trunet thanks for the tip with the extended servers i will have a look at them 😃

[s-weigand]

So I did play around with ThreadedFTPServer and MultiprocessFTPServer from pyftpdlib, but they have the same problem as the normal FTPServer, server.serve_forever() blocks all other actions if run in the main thread. Even if you start the server with server.serve_forever(blocking=False) the tests hang, which is why we run the server in a separate thread/process. ThreadedFTPServer and MultiprocessFTPServer run also in the main thread, but start a new task as thread/process for each new connection.

@trunet how did you find out, that the server is listening around 10 seconds after the test finishes? Sadly I can't reproduce your error, so it's hard to debug.

[trunet]

Before taking FTP_PORT out, when I ran the tests right after a completed one 'Address already in use' triggers.

Like 'tox; tox', exception. 'tox; sleep 5; tox', exception. 'tox; sleep 10; tox', success.

I have a jenkins job for this running on a linux slave. When running from linux slave, I don't have any error, so the 'OSError: [Errno 9] Bad file descriptor' traceback is macosx only apparently.

[s-weigand]

Could you maybe provide a minimal example to reproduce the error?

It might help to extend the stopmethod of ThreadFTPServer like this:

    def stop(self):
        self._server.stop()
        self.join()

But since I can't reproduce the error this is just a guess, at least it's not breaking our tests.