If the local file access very slow, it will hand the ftp server, all other client can not do any operations.

[GoogleCodeExporter]

If the local file access very slow, it will hand the ftp server, all other 
client can not do any operations.

I write a FakeFile class, which it's write, read, close methods will take a 
while.
I attach the source code in the attachment. 

What steps will reproduce the problem?
1. run the fake_ftp_server.py, python fake_ftp_server.py
2. use a ftp client to RETR a file
3. try another ftp client to connect to the server, it will hang.

What is the expected output? What do you see instead?
I think when STOR/RETR a file slow, it can hand the connect which doing the 
operation. But other client need can do the operations.

What version of pyftpdlib are you using? On what operating system? Which
Python version?

pyftpdlib:  version 0.7.0, operating system: CentOS 6.2, python 2.7

Original issue reported on code.google.com by txdyj...@gmail.com on 3 Jul 2012 at 7:37

Attachments:

• fake_ftp_server.py

[GoogleCodeExporter]

Yes, this is a known problem inherited from the fact that pyftpdlib is 
asynchronous and there have been other issues filed for the same reason.
There's no silver bullet for this, only ad-hoc solutions which sometimes are 
applicable, sometimes are not.

The most immediate solution which comes to mind is to override readable() and 
writable() methods of DTPHandler class and use select()/poll() to figure out 
whether the file object is read to be read or written and return True/False in 
accordance.
That way read()/write() calls will be invoked only when the file is ready.

Another solution is to run the DTPHandler IO loop in a separate thread or 
process but that requires to modify how asyncore is used internally and I have 
no immediate solution/code sample for that.

I think it would be nice to have solution #2 officially included in pyftpdlib 
somehow though.
I'll keep this as a reminder.

Original comment by g.rodola on 3 Jul 2012 at 5:08

[GoogleCodeExporter]

Maybe we can replace asyncore with gevent+greenlets?

Original comment by equ...@gmail.com on 5 Jul 2012 at 11:49

[GoogleCodeExporter]

I'm not sure how much that would help. 
Do they provide facilities to solve this specific problem? 
I'm inclined to think it's unlikely, as developing a framework which is able to 
conciliate both sockets and regular files async-IO in a cross platform fashion 
shouldn't be easy to say the least.
Twisted, for example, suffers the same issue:  
http://twistedmatrix.com/trac/ticket/3983

Practically speaking, how much is the slowdown per-single read()/write() you're 
experiencing (in terms of seconds)?

Original comment by g.rodola on 6 Jul 2012 at 12:04

[GoogleCodeExporter]

I'm not familiar with gevent but I am assuming greenlets are related to green 
threads, i.e. It's multi-threaded rather than pure asynchronous code. A quick 
google leads me to believe the idea behind gevent is to make a synchronous 
execution framework that's event based. So it'd be a server that acts like 
asychronous code but executes synchronously. I have no idea how that would work 
out for pyftpdlib though in terms of reimplemntation.

Original comment by jlo...@gmail.com on 6 Jul 2012 at 12:14

[GoogleCodeExporter]

That would surely imply a massive code rewriting and compatibility breakage, 
plus the addition of a 3th party dep.
It probably makes a lot more sense to provide the possibility to run each 
individual "handler" in a separate thread or process as part of the new IO 
poller I implemented in issue 203.
I'm still not sure how exactly though. I'll have to think about it.

Original comment by g.rodola on 6 Jul 2012 at 12:21

[GoogleCodeExporter]

"Maybe we can replace asyncore with gevent+greenlets?"
I guess no higher level async framework can easily beat asyncore with 
Giampaolo's recent epoll and kqueue work in raw performance. They just add 
complexity on top of the same primitives.
-1

Original comment by nagy.att...@gmail.com on 16 Jul 2012 at 6:25

[GoogleCodeExporter]

"It probably makes a lot more sense to provide the possibility to run each 
individual "handler" in a separate thread or process as part of the new IO 
poller I implemented in issue 203.
I'm still not sure how exactly though. I'll have to think about it."
I just start an arbitrary amount of worker threads (or processes) and feed them 
via Queue (Queue.Queue.put).

Original comment by nagy.att...@gmail.com on 16 Jul 2012 at 6:29

[GoogleCodeExporter]

Issue 197 has been merged into this issue.

Original comment by g.rodola on 2 Aug 2012 at 7:56

[GoogleCodeExporter]

To my surprise, it seems I made it after all.
I added two new classes changing the concurrency model from async to multi 
processes/threads based.

From the docstring:

"""
This module contains two FTPServer subclasses changing the asynchronous
concurrency model used natively by pyftpdlib.

You might be interested in these in case your code contains blocking
parts which cannot be adapted to the base async model or if the
underlying filesystem is particularly slow, see:

https://code.google.com/p/pyftpdlib/issues/detail?id=197
https://code.google.com/p/pyftpdlib/issues/detail?id=212

Two classes are provided:

 - ThreadingFTPServer
 - MultiprocessFTPServer

...spawning a new thread or process every time a client connects.
Every handler will live in its own thread/process and will be free to
block without freezing the FTP server.
"""

...So basically what you have to do now is this:

>>> from pyftpdlib import ftpserver
>>> from pyftpdlib.contrib.servers import ThreadedFTPServer   # ...or 
MultiprocessFTPServer
>>>
>>> authorizer = ftpserver.DummyAuthorizer()
>>> #authorizer.add_user(...)
>>> handler = ftpserver.FTPHandler
>>> handler.authorizer = authorizer
>>> server = ThreadedFTPServer(('', 21), handler)
>>> server.serve_forever()

Committed in two big chunks: r1065 and r1072.
I'll blog about this in details hereinafter as it is an important usability 
improvement.

Original comment by g.rodola on 2 Aug 2012 at 8:58

• Changed state: FixedInSVN
• Added labels: Milestone-1.0.0, Priority-High, Type-Enhancement, Usability, Version-0.7.0
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

Releasing 1.0.0 just now. Closing.

Original comment by g.rodola on 19 Feb 2013 at 12:49

• Changed state: Fixed