2 local FTP servers

[akozyreva]

Hi! I have a question - is it possible to run 2 local ftp servers with different root directories?

[oz123]

I believe so. Just create two instances with different ports, too. You can't have multiple servers listening on the same port.

[akozyreva]

Hi @oz123 ! Thanks for so quick answer! Could you please tell me, which ports I can use safely for that? I know, that I can use 2121. What else?

[oz123]

You can use any port larger than 1024 (see https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html), as long as it is not already used.

[akozyreva]

Hi @oz123 ! Thanks for the answer! Yes, technically it's possible, but if you try to run these 2 ftp servers, system can confuse them - despite 2 different ports, connection can show directory A (from 1 ftp) or B(from 2 ftp), even if you always connect only to the 1 ftp.

[oz123]

Hi, can you post the code you used please?

[akozyreva]

Hi @oz123 ! here is my code

import pytest

@pytest.fixture
def ftp_server(ftpserver):
    class ThingFactory:
        def get(self):
            return ftpserver
    return ThingFactory()

def test_thing(ftp_server):
    thing1 = ftp_server.get()
    print(thing1.get_login_data())
    thing2 = ftp_server.get()
    print(thing2.get_login_data())

I tried to get different ftp servers, but they come back with the same port. When I tried to run ftp servers directly (using pyftpdlib), they started, but system confuse them.

[oz123]

You are using only one ftpserver. You need to create a second fixture. You will have to create a file called fixtures.py in the tests directory. And in there create a fixture with a name of your liking. You can copy the code for the fixture from here:

https://github.com/oz123/pytest-localftpserver/blob/52440e2e87b46cc7dc8167f7a4bf214e27ce55c4/pytest_localftpserver/plugin.py#L17

Take a look in the servers.py file how to create an FTP server instance with different ports and directories.

[akozyreva]

Hi @oz123 ! I created, as you adviced.

from ftplib import FTP

import pytest
from pytest_localftpserver.servers import PytestLocalFTPServer

@pytest.fixture(scope="function")
def ftpserver_from(request):
    server = PytestLocalFTPServer()
    request.addfinalizer(server.stop)
    return server

@pytest.fixture(scope="function")
def ftpserver_to(request):
    server = PytestLocalFTPServer()
    request.addfinalizer(server.stop)
    return server

def test_ftp(ftpserver_from, ftpserver_to):
    usr = 'fakeusername'
    pwd = 'qweqwe'
    ftpserver_from.put_files("test_file", style="rel_path", anon=False)
    ftp1 = FTP()
    ftp1.connect('localhost', ftpserver_from.server_port)
    ftp1.login(usr, pwd)
    print(ftp1.dir())
    print(list(ftpserver_from.get_file_paths(style="rel_path", anon=False)))
    ftp2 = FTP()
    ftp2.connect('localhost', ftpserver_to.server_port)
    ftp2.login(usr, pwd)
    print(ftp2.dir())
    print(list(ftpserver_to.get_file_paths(style="rel_path", anon=False)))
    ftp1.quit()
    ftp2.quit()

Output is

None
['test_file']
None
[]
.

In my case ftp1.dir() and ftp2.dir() both reutrn None, but listing of ftpserver_from is successful. Why?

[oz123]

Unfortunately, the public API does not assume that a person runs 2 FTP servers. So it's not that obvious. Appologies for that. Here is how you can do it:

# cat fixtures.py
import multiprocessing
import pytest

from pytest_localftpserver.servers import PytestLocalFTPServer, ProcessFTPServer, SimpleFTPServer

class ProcessFTPServer:

    def __init__(self, username, password, ftp_home, ftp_port, use_TLS=False):
        self._server = SimpleFTPServer(username, password, ftp_port=ftp_port,
                                       ftp_home=ftp_home, use_TLS=use_TLS)
        self.process = multiprocessing.Process(target=self._server.serve_forever)
        # This is a must in order to clear used sockets
        self.process.daemon = True
        self.process.start()

    def stop(self):
        self.process.terminate()

@pytest.fixture(scope="function")
def ftpserver_from(request):
    # server = PytestLocalFTPServer() # uses environment variables implicitly
    # request.addfinalizer(server.stop)
    server = ProcessFTPServer(username="benz", password="erni1",
                              ftp_home="/home/oznt/Music", ftp_port=34443) # uses explicit parameters
    request.addfinalizer(server.stop)
    return server

@pytest.fixture(scope="function")
def ftpserver_to(request):
    server = ProcessFTPServer(username="fakeusername", password="qweqwe",
                              ftp_home="/home/oznt/Music", ftp_port=34444) # uses explicit parameters
    request.addfinalizer(server.stop)
    return server

and:

# cat tests.py
from ftplib import FTP

from fixtures import ftpserver_from, ftpserver_to

def test_ftp(ftpserver_from, ftpserver_to):
    usr = 'benz'
    pwd = 'erni1'
    ftp1 = FTP()
    ftp1.connect('localhost', ftpserver_from._server._ftp_port)
    ftp1.login(usr, pwd)
    print(ftp1.dir())
    ftp2 = FTP()

    ftp2.connect('localhost', ftpserver_to._server.server_port)
    ftp2.login("fakeusername", "qweqwe")
    print("MUSIC")
    print(ftp2.dir())

I hope this helps.

[akozyreva]

Hi @oz123 ! Please no worries, I'm happy, that somebody tries to help me. Did you run your test_ftp? It fails once in a while(not every time), because of authentication - it tries to connect to ftpserver_from, but current connection is estbalished in computer, using credentials from ftpserver_to. As I described before, system confuse ftpserver_from and ftpserver_to, they are mixed.

ftpserver_from = <test_e.ProcessFTPServer object at 0x7f3ba31b2d60>, ftpserver_to = <test_e.ProcessFTPServer object at 0x7f3ba31b2df0>

    def test_ftp(ftpserver_from, ftpserver_to):
        usr = 'benz'
        pwd = 'erni1'
        ftp1 = FTP()
        ftp1.connect('localhost', ftpserver_from._server._ftp_port)
>       ftp1.login(usr, pwd)

test_e.py:48: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.9/ftplib.py:414: in login
    resp = self.sendcmd('PASS ' + passwd)
/usr/lib/python3.9/ftplib.py:281: in sendcmd
    return self.getresp()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <ftplib.FTP object at 0x7f3ba31b2d90>

    def getresp(self):
        resp = self.getmultiline()
        if self.debugging:
            print('*resp*', self.sanitize(resp))
        self.lastresp = resp[:3]
        c = resp[:1]
        if c in {'1', '2', '3'}:
            return resp
        if c == '4':
            raise error_temp(resp)
        if c == '5':
>           raise error_perm(resp)
E           ftplib.error_perm: 530 Authentication failed.

/usr/lib/python3.9/ftplib.py:254: error_perm
========================================================================================================================== short test summary info ==========================================================================================================================
FAILED test_e.py::test_ftp - ftplib.error_perm: 530 Authentication failed.

[oz123]

I managed to get 200 out of 200 tests pass in the way I described here:

https://stackoverflow.com/questions/68101725/pytest-mutliprocessing-process-instances-are-no-created-in-a-deterministic-way

Unfortunately, I can't explain this behavior. This is probably with how multiprocessing.Process calls are done.

[s-weigand]

I also did some testing+debugging and found the root of the issue. The problem isn't related to pytest or multiprocessing/threading, but pyftpdlib and using multiple instances with different handlers. The FTPHandler saves the authorizer as a class attribute, not an instance attribute.

https://github.com/giampaolo/pyftpdlib/blob/93f6623ca64973cfa34686f5d985044c646f3b8b/pyftpdlib/handlers.py#L1196

Thus when overwriting handler.authorizer its isn't only overwritten for this particular instance of FTPHandler but for all instances (especially since handler isn't an instance but a class).

https://github.com/oz123/pytest-localftpserver/blob/52440e2e87b46cc7dc8167f7a4bf214e27ce55c4/pytest_localftpserver/servers.py#L90

Depending on which instance of SimpleFTPServer is created last, that authorizer will be used for all servers living in the same interpreter. I found that this is a known issue and @TheSil also provided a nice solution for it. @TheSil just wrapped a subclass definition in a function, that way each class is defined in a closure that references to a different point in memory where the function was called and even so the class has the same name, class variables aren't shared.

As for the flaky behavior of pytest (I also experienced it passing on the first run and failing after), I guess that has to do with the .pytest_cache.

[s-weigand]

@akozyreva To validate that the docs I wrote for #138 are working I used the following test script:

import pytest

from pytest_localftpserver.servers import PytestLocalFTPServer

# from mypackage import ftp_copy

from ftplib import FTP

def ensure_files_exist():
    from pathlib import Path

    test_folder = Path(__file__).parent / "test_folder"
    test_folder.mkdir(exist_ok=True)

    foo = test_folder / "test_file1"
    foo.write_text("foo")

    bar = test_folder / "test_file2"
    bar.write_text("bar")

def ftp_copy(source_ftp_login_data, target_ftp_login_data):

    import tempfile

    source_ftp = FTP()
    source_ftp.connect(source_ftp_login_data["host"], source_ftp_login_data["port"])
    source_ftp.login(source_ftp_login_data["user"], source_ftp_login_data["passwd"])

    target_ftp = FTP()
    target_ftp.connect(target_ftp_login_data["host"], target_ftp_login_data["port"])
    target_ftp.login(target_ftp_login_data["user"], target_ftp_login_data["passwd"])

    for file in ["test_file1", "test_file2"]:
        buffer = tempfile.mktemp()
        source_ftp.retrbinary(f"RETR {file}", open(buffer, "wb").write)
        target_ftp.storbinary(f"STOR {file}", open(buffer, "rb"))

@pytest.fixture()
def target_ftpserver():
    """Target FTP server fixture with out TSL."""
    server = PytestLocalFTPServer(username="target_user", password="target_password")
    yield server
    server.stop()

@pytest.fixture()
def target_ftpserver_TLS():
    """Target FTP server fixture with TSL."""
    server = PytestLocalFTPServer(
        username="target_user", password="target_password", use_TLS=True
    )
    yield server
    server.stop()

def test_ftp_copy(
    ftpserver: PytestLocalFTPServer, target_ftpserver: PytestLocalFTPServer
):
    """Copy from one server to another."""
    ensure_files_exist()
    ftpserver.put_files(
        ["test_folder/test_file1", "test_folder/test_file2"], anon=False
    )
    print(list(ftpserver.get_file_contents()))

    ftp_copy(ftpserver.get_login_data(), target_ftpserver.get_login_data())

    assert list(ftpserver.get_file_contents()) == list(
        target_ftpserver.get_file_contents()
    )

which should be pretty close to what you want to do or at least have some useful code snippets. So feel free to check out my PR branch (#138 ) and test it solves your issue.

[oz123]

@s-weigand thanks for the time invested here. It will take me some time to review your large PR. As a side note, I tried running the tests with cache removal, and it was still flaky ('pytest --cache-clear'). Obviously, I was barking at the wrong tree. I didn't expect such odd design from pyftpdlib.

[s-weigand]

@oz123 Well if one thinks of how an FTP server is run in production it kinda makes sense. There you would put each server configuration in its own life and run it as a service, instead of sharing one interpreter. Starting two servers from the same interpreter is a kinda an edge case usage which only makes sense in a testing scenario.