giampaolo / pyftpdlib

Extremely fast and scalable Python FTP server library
MIT License
1.65k stars 266 forks source link

[MultiprocessFTPServer] TypeError: cannot pickle 'select.kqueue' object #561

Closed june-yub closed 2 years ago

june-yub commented 2 years ago

Hello,

[ TypeError: cannot pickle 'select.kqueue' object ] occured when I tried to connect to the server. I used MultiprocessFTPServer as Multiple process example shows in https://pyftpdlib.readthedocs.io/en/latest/tutorial.html But If I change MultiprocessFTPServer() to FTPServer(), it works well. I can send files easily. Please advise how to use multiprocess correctly. My environment and terminal results from server&client are as follow.

Environment--------------

server: python3.8.9 macOS 11.6.1 (M1) pyftpdlib1.5.6

client: python3.8.2 macOS 10.15.7 pyftpdlib1.5.6

Terminal results (server)--------------

notaryman@Notarymans-Mac-mini sign % python3 sign_server.py
[I 2021-11-18 10:30:12] concurrency model: multi-process
[I 2021-11-18 10:30:12] masquerade (NAT) address: None
[I 2021-11-18 10:30:12] passive ports: 60000->61999
[I 2021-11-18 10:30:26] ::ffff:172.20.5.189:55080-[] FTP session opened (connect)
[E 2021-11-18 10:30:26] Traceback (most recent call last):
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncore.py", line 83, in read
        obj.handle_read_event()
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncore.py", line 414, in handle_read_event
        self.handle_accept()
      File "../pyftpdlib/ioloop.py", line 1050, in handle_accept
        self.handle_accepted(sock, addr)
      File "../pyftpdlib/servers.py", line 458, in handle_accepted
        t.start()
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
        return Popen(process_obj)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
        super().__init__(process_obj)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
        reduction.dump(process_obj, fp)
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
        ForkingPickler(file, protocol).dump(obj)
    TypeError: cannot pickle 'select.kqueue' object 

Terminal results (client)--------------

Junyeops-iMac:Utils junyeopjeon$ python3 /Users/junyeopjeon/Downloads/ftp_test.py 
Traceback (most recent call last):
  File "/Users/junyeopjeon/Downloads/ftp_test.py", line 465, in main
    obj_ftp.connectToServer()
  File "/Users/junyeopjeon/Downloads/ftp_test.py", line 449, in connectToServer
    self.ftp.login( self.SIGN_SERVER_LOGIN_ID, self.SIGN_SERVER_LOGIN_PW )    # user name, passwd
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 401, in login
    resp = self.sendcmd('USER ' + user)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 275, in sendcmd
    return self.getresp()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 238, in getresp
    resp = self.getmultiline()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 224, in getmultiline
    line = self.getline()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 206, in getline
    line = self.file.readline(self.maxline + 1)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/junyeopjeon/Downloads/ftp_test.py", line 475, in <module>
    ret = main()
  File "/Users/junyeopjeon/Downloads/ftp_test.py", line 469, in main
    obj_ftp.disconnectToServer()
  File "/Users/junyeopjeon/Downloads/ftp_test.py", line 452, in disconnectToServer
    self.ftp.quit()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 648, in quit
    resp = self.voidcmd('QUIT')
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 279, in voidcmd
    self.putcmd(cmd)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 201, in putcmd
    self.putline(line)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 196, in putline
    self.sock.sendall(line.encode(self.encoding))
BrokenPipeError: [Errno 32] Broken pipe

P.S. The same "TypeError: cannot pickle 'select.kqueue' object" message occurs during [make test ] with python3.10 as install section instructs in https://pyftpdlib.readthedocs.io/en/latest/install.html...

june-yub commented 2 years ago

It was solved by changing the setting to use fork() instead of spawn() which is the default method on masOS. Thanks.