ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket)

[40b36f1b-5478-46ec-bc77-d5521c86f738]

BPO	4791
Nosy	@vstinner, @giampaolo, @devdanzin
Files	from_filezilla.png: same folder from filezilla

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['extension-modules', 'type-crash'] title = "ftplib.retrlines('LIST') hangs at the end of listing (SocketIO.close() doesn't close the socket)" updated_at = user = 'https://bugs.python.org/chrismahan' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = True closed_date = closer = 'vstinner' components = ['Extension Modules'] creation = creator = 'chris.mahan' dependencies = [] files = ['12508'] hgrepos = [] issue_num = 4791 keywords = ['patch'] message_count = 29.0 messages = ['78603', '78604', '78618', '78630', '78633', '78636', '78720', '78726', '78727', '78760', '78761', '78763', '78799', '78802', '78804', '78879', '78914', '78918', '78919', '78920', '79184', '79202', '79205', '79207', '79210', '79216', '79218', '79219', '79222'] nosy_count = 4.0 nosy_names = ['vstinner', 'giampaolo.rodola', 'ajaksu2', 'chris.mahan'] pr_nums = [] priority = 'normal' resolution = 'duplicate' stage = None status = 'closed' superseder = None type = 'crash' url = 'https://bugs.python.org/issue4791' versions = ['Python 3.0'] ```

[40b36f1b-5478-46ec-bc77-d5521c86f738]

The python program crashes (stops responding) both from the command line and in IDLE (ver 3.0), after listing all the files in the approprate directory, both with ftp.dir() and with ftp.retrlines('LIST') (see prog listing below).

I have to close the window (both window shell and IDLE). No other key combo responds.

Python env (from IDLE): Python 3.0 (r30:67507, Dec 3 2008, 20:14:27) [MSC v.1500 32 bit (Intel)] on win32

Program:

import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword')
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()

The username and password are correct. I am able to use the ftp resource with filezilla (3.1.6) with no problem.

This runs on a Windows Server 2003 Standard Edition, Service Pack 2.

This is also a completely clean install of Python 3.0. I installed it, then wrote this program.

Python 2.5 is installed on this server and runs fine.

Any questions: chris.mahan@gmail.com

[40b36f1b-5478-46ec-bc77-d5521c86f738]

Update: Ran the same code with python 2.6.1 on the same computer, and that worked fine.

[vstinner]

Can you paste the expected result of ftp.retrlines('LIST')? Does a directory contains a non-ASCII character?

[40b36f1b-5478-46ec-bc77-d5521c86f738]

the output: just before the non-responsiveness:

-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat - Why.mp3 -rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi -rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv -rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg -rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv -rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg -rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv -rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg

[40b36f1b-5478-46ec-bc77-d5521c86f738]

Added file screenshot of filezilla view of the folder in question.

[40b36f1b-5478-46ec-bc77-d5521c86f738]

The list does not seem to contain non-ascii characters.

[vstinner]

I'm unable to reproduce the issue. I tried to create files on my FTP server with non-ASCII characters or spaces in the filenames, but everything is fine. Can you reproduce the problem outside IDLE? Or is the issue specific to IDLE? I mean: write a script and execute "python script.py" in a terminal.

[90baf024-6604-450d-8341-d796fe6858f3]

Hi Chris

Since dir calls retrlines and retrlines has a 'while 1:' loop, the bug probably comes from there. Either it hangs in the fp.readline call or the break condition is never met.

Can you put some print diagnostics inside Lib/ftplib.py->FTP.retrlines (around line 405) to help pinpointing the issue?

HTH, Daniel

[vstinner]

Can you put some print diagnostics inside Lib/ftplib.py (...)

\<your ftp object>.set_debuglevel(2) already prints a lot of informations.

[40b36f1b-5478-46ec-bc77-d5521c86f738]

I modified the program by adding line: ftp.set_debuglevel(2) START LISTING----------------

import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='myuserid',
passwd='mypassword')
ftp.set_debuglevel(2)
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()
END LISTING 

---

Running from idle 1.2.4 with python 2.5.4 gives this output:

*cmd 'CWD chrismahan-675' *put 'CWD chrismahan-675\r\n' *get '250 CWD command successful\r\n' *resp '250 CWD command successful' *cmd 'TYPE A' *put 'TYPE A\r\n' *get '200 Type set to A\r\n' *resp '200 Type set to A' *cmd 'PASV' *put 'PASV\r\n' *get '227 Entering Passive Mode (72,21,82,190,234,6).\r\n' *resp '227 Entering Passive Mode (72,21,82,190,234,6).' *cmd 'LIST' *put 'LIST\r\n' *get '150 Opening ASCII mode data connection for file list\r\n' *resp '150 Opening ASCII mode data connection for file list' -rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat - Why.mp3 -rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi -rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv -rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg -rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv -rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg -rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv -rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg *get '226 Transfer complete\r\n' *resp '226 Transfer complete'

Then running with idle 3.0 with python 3.0 gives the following result.

*cmd 'CWD chrismahan-675' *put 'CWD chrismahan-675\r\n' *get '250 CWD command successful\n' *resp '250 CWD command successful' *cmd 'TYPE A' *put 'TYPE A\r\n' *get '200 Type set to A\n' *resp '200 Type set to A' *cmd 'PASV' *put 'PASV\r\n' *get '227 Entering Passive Mode (72,21,82,190,148,178).\n' *resp '227 Entering Passive Mode (72,21,82,190,148,178).' *cmd 'LIST' *put 'LIST\r\n' *get '150 Opening ASCII mode data connection for file list\n' *resp '150 Opening ASCII mode data connection for file list' -rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat - Why.mp3 -rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi -rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv -rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg -rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv -rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg -rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv -rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg

Note that the last two lines are missing.

From a command prompt: C:\>c:\python30\python.exe c:\python_scripts\python3\candee_processor.py *cmd 'CWD chrismahan-675' *put 'CWD chrismahan-675\r\n' *get '250 CWD command successful\n' *resp '250 CWD command successful' *cmd 'TYPE A' *put 'TYPE A\r\n' *get '200 Type set to A\n' *resp '200 Type set to A' *cmd 'PASV' *put 'PASV\r\n' *get '227 Entering Passive Mode (72,21,82,190,219,8).\n' *resp '227 Entering Passive Mode (72,21,82,190,219,8).' *cmd 'LIST' *put 'LIST\r\n' *get '150 Opening ASCII mode data connection for file list\n' *resp '150 Opening ASCII mode data connection for file list' -rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat - Why.mp3 -rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi -rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv -rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg -rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv -rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg -rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv -rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg

same command prompt but with python 2.5:

C:\>c:\python25\python.exe c:\python_scripts\python3\candee_processor.py *cmd 'CWD chrismahan-675' *put 'CWD chrismahan-675\r\n' *get '250 CWD command successful\r\n' *resp '250 CWD command successful' *cmd 'TYPE A' *put 'TYPE A\r\n' *get '200 Type set to A\r\n' *resp '200 Type set to A' *cmd 'PASV' *put 'PASV\r\n' *get '227 Entering Passive Mode (72,21,82,190,147,65).\r\n' *resp '227 Entering Passive Mode (72,21,82,190,147,65).' *cmd 'LIST' *put 'LIST\r\n' *get '150 Opening ASCII mode data connection for file list\r\n' *resp '150 Opening ASCII mode data connection for file list' -rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat - Why.mp3 -rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi -rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv -rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg -rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv -rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg -rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv -rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg *get '226 Transfer complete\r\n' *resp '226 Transfer complete'

C:\>

[40b36f1b-5478-46ec-bc77-d5521c86f738]

I went into the source for Libs/ftplib.py and I noted that line 423 is:

if self.debugging > 2: print('*retr*', repr(line))

so I changed the debuglevel to 3, as such: ftp.set_debuglevel(3) and I got these last 4 lines:

-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg *retr '' *get '226 Transfer complete\r\n' *resp* '226 Transfer complete'

First line, the last line of the listing. second line, the debug, showing the last line read third and fourth lines: the program actually completed and didn't hang!

still digging

[40b36f1b-5478-46ec-bc77-d5521c86f738]

I added the following two lines and I was able to run the code.

Before: line = fp.readline() if self.debugging > 2: print('*retr*', repr(line)) if not line: break if line[-2:] == CRLF: line = line[:-2] elif line[-1:] == '\n': line = line[:-1] callback(line)

After: line = fp.readline() if self.debugging > 2: print('*retr*', repr(line)) if not line: break if line == '': break if line[-2:] == CRLF: line = line[:-2] elif line[-1:] == '\n': line = line[:-1] callback(line)

As a patch:

426,427d425 \< if line == '': \< break

What else do I need to do?

[vstinner]

Can you paste the full output with debuglevel=3?

"if not line:" should be equivalent to "if len(line) == 0:" and to "if line == '':".

ftp.dir() is equivalent to ftp.retrlines('LIST').

By default, the socket created to get the result of LIST has no timeout (is blocking). So fp.readline() only returns an empty string when the socket is closed (by the server).

fp is a io.BufferedReader(SocketIO(socket, "r"), io.DEFAULT_BUFFER_SIZE).

[40b36f1b-5478-46ec-bc77-d5521c86f738]

Full output with debuglevel 3

*cmd 'CWD chrismahan-675' *put 'CWD chrismahan-675\r\n' *get '250 CWD command successful\n' *resp '250 CWD command successful' *cmd 'TYPE A' *put 'TYPE A\r\n' *get '200 Type set to A\n' *resp '200 Type set to A' *cmd 'PASV' *put 'PASV\r\n' *get '227 Entering Passive Mode (72,21,82,190,217,227).\n' *resp '227 Entering Passive Mode (72,21,82,190,217,227).' *cmd 'LIST' *put 'LIST\r\n' *get '150 Opening ASCII mode data connection for file list\n' *resp '150 Opening ASCII mode data connection for file list' *retr '-rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat - Why.mp3\n' -rwxrwxrwx 1 nobody nogroup 3905538 Dec 29 09:51 Bronski Beat - Why.mp3 *retr '-rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi\n' -rwxrwxrwx 1 nobody nogroup 873966 Dec 28 13:53 test9.avi *retr '-rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv\n' -rwxrwxrwx 1 nobody nogroup 2512653 Dec 29 08:28 test9_lg.wmv *retr '-rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg\n' -rwxrwxrwx 1 nobody nogroup 6549 Dec 29 08:28 test9_lg.wmv.jpg *retr '-rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv\n' -rwxrwxrwx 1 nobody nogroup 1788466 Dec 29 03:04 test9_med.flv *retr '-rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg\n' -rwxrwxrwx 1 nobody nogroup 6394 Dec 29 03:04 test9_med.flv.jpg *retr '-rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv\n' -rwxrwxrwx 1 nobody nogroup 1263041 Dec 28 13:53 test9_sm.flv *retr '-rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg\n' -rwxrwxrwx 1 nobody nogroup 6465 Dec 28 13:53 test9_sm.flv.jpg *retr* ''

[40b36f1b-5478-46ec-bc77-d5521c86f738]

haypo writes:

By default, the socket created to get the result of LIST has no timeout (is blocking). So fp.readline() only returns an empty string when the socket is closed (by the server).

Even if the server closed the socket and fp.readline() returned an empty string, the prog should not hang, no?

[giampaolo]

I'm unable to reproduce the issue. Could you please repeat the test by using the "timeout" parameter as such?

\< ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername', passwd='thepassword')

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername', passwd='thepassword', timeout=2)

[40b36f1b-5478-46ec-bc77-d5521c86f738]

Per Giampaolo's suggestion, I added the timeout. Program listing and output below:

-----begin listing----

import ftplib

ftp = ftplib.FTP('ftp.edgecastcdn.net', user='theusername',
passwd='thepassword', timeout=2)
ftp.set_debuglevel(3)
ftp.cwd('chrismahan-675')
ftp.dir()
#ftp.retrlines('LIST')
ftp.close()
----end listing

---

*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,172,22).\n'
*resp* '227 Entering Passive Mode (72,21,82,190,172,22).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\n'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi\n'
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28
test9_lg.wmv\n'
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28
test9_lg.wmv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04
test9_med.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04
test9_med.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53
test9_sm.flv\n'
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53
test9_sm.flv.jpg\n'
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
Traceback (most recent call last):
  File "C:\python_scripts\python3\candee_processor.py", line 6, in <module>
    ftp.dir()
  File "c:\python30\lib\ftplib.py", line 511, in dir
    self.retrlines(cmd, func)
  File "c:\python30\lib\ftplib.py", line 433, in retrlines
    return self.voidresp()
  File "c:\python30\lib\ftplib.py", line 225, in voidresp
    resp = self.getresp()
  File "c:\python30\lib\ftplib.py", line 211, in getresp
    resp = self.getmultiline()
  File "c:\python30\lib\ftplib.py", line 197, in getmultiline
    line = self.getline()
  File "c:\python30\lib\ftplib.py", line 184, in getline
    line = self.file.readline()
  File "C:\Python30\lib\io.py", line 1813, in readline
    while self._read_chunk():
  File "C:\Python30\lib\io.py", line 1560, in _read_chunk
    input_chunk = self.buffer.read1(self._CHUNK_SIZE)
  File "C:\Python30\lib\io.py", line 994, in read1
    self._peek_unlocked(1)
  File "C:\Python30\lib\io.py", line 981, in _peek_unlocked
    current = self.raw.read(to_read)
  File "C:\Python30\lib\io.py", line 575, in read
    n = self.readinto(b)
  File "C:\Python30\lib\socket.py", line 214, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

[vstinner]

The traceback shows that the problem is not related to the socket created to retrieve the content of the directory listing, but the "main" socket (the one used for the whole ftp session) used by dir() command the retrieve the command answer code:

   Traceback (most recent call last):
     ftp.dir()
      => self.retrlines(cmd, func)
      => return self.voidresp()
      => line = self.getline()
      => line = self.file.readline() (ftplib.py:184)
      ...
    socket.timeout: timed out

I don't understand why the readline() blocks. It's maybe related to the newline: voidresp() uses getline() which reads a line using self.file. The file is created by self.sock.makefile('r', encoding=self.encoding) and so the newline is always \n whereas ftplib of Python2 uses \r\n (CRLF) or \n (LF).

[vstinner]

Christopher: Can you paste the full output with debuglevel=3 of Python 2.x (python2.5)?

[40b36f1b-5478-46ec-bc77-d5521c86f738]

When running scritp with python 2.5.4, got this error. I'll remove the "timeout=2" and run again. see below.

Traceback (most recent call last):
  File "C:\python_scripts\python3\candee_processor.py", line 3, in <module>
    ftp = ftplib.FTP('ftp.edgecastcdn.net',
user='client1@arcostream.com', passwd='1210184520', timeout=2)
TypeError: __init__() got an unexpected keyword argument 'timeout'

Same without the timeout argument:

>>> 
*cmd* 'CWD chrismahan-675'
*put* 'CWD chrismahan-675\r\n'
*get* '250 CWD command successful\r\n'
*resp* '250 CWD command successful'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A\r\n'
*resp* '200 Type set to A'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (72,21,82,190,228,195).\r\n'
*resp* '227 Entering Passive Mode (72,21,82,190,228,195).'
*cmd* 'LIST'
*put* 'LIST\r\n'
*get* '150 Opening ASCII mode data connection for file list\r\n'
*resp* '150 Opening ASCII mode data connection for file list'
*retr* '-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski
Beat - Why.mp3\r\n'
-rwxrwxrwx   1 nobody   nogroup   3905538 Dec 29 09:51 Bronski Beat -
Why.mp3
*retr* '-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53
test9.avi\r\n'
-rwxrwxrwx   1 nobody   nogroup    873966 Dec 28 13:53 test9.avi
*retr* '-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28
test9_lg.wmv\r\n'
-rwxrwxrwx   1 nobody   nogroup   2512653 Dec 29 08:28 test9_lg.wmv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28
test9_lg.wmv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6549 Dec 29 08:28 test9_lg.wmv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04
test9_med.flv\r\n'
-rwxrwxrwx   1 nobody   nogroup   1788466 Dec 29 03:04 test9_med.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04
test9_med.flv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6394 Dec 29 03:04 test9_med.flv.jpg
*retr* '-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53
test9_sm.flv\r\n'
-rwxrwxrwx   1 nobody   nogroup   1263041 Dec 28 13:53 test9_sm.flv
*retr* '-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53
test9_sm.flv.jpg\r\n'
-rwxrwxrwx   1 nobody   nogroup      6465 Dec 28 13:53 test9_sm.flv.jpg
*retr* ''
*get* '226 Transfer complete\r\n'
*resp* '226 Transfer complete'
>>>

[40b36f1b-5478-46ec-bc77-d5521c86f738]

Update:

I requested a test ftp account from edgecast. The tech's response was that he would file a request for it. Will update.

[40b36f1b-5478-46ec-bc77-d5521c86f738]

Edgecast.com support created a test ftp account for the purpose of troubleshooting this issue. Please email me (chris.mahan@gmail.com) and I'll forward you the login info.

[vstinner]

Ok, some news about this bug (I tested the ftp test account): the problem is that the data socket is not closed when self.voidresp() is called. I can't see close() syscall before the call to self.voidresp(). SocketIO.close() does nothing: it does not really close the socket, just the file. And socket.close() does also nothing because there is still an io reference... the closed file :-/ The io reference counter is only updated when the file is destroyed. conn = \<create a socket> # conn._io_refs=0 fp = conn.makefile('r', encoding=self.encoding) # conn._io_refs=1 ... fp.close() # only set fp.closed to True conn.close() # only set conn._closed to True # conn._io_refs=1 and the socket (file descriptor) is still open del fp # conn._io_refs=0 -> the socket is finally closed!

[vstinner]

Patch: SocketIO.close() decrements the io reference of its socket, so socket.close() will really close the socket.

About the FTP server ftp2.edgecastcdn.net, it looks like the server doesn't write "226 Transfert completed" until the data socket is really closed. The server says that it's running EdgeFTPD 1.1. nmap says that the ftp server is ProFTPD 1.2.10 running on Linux 2.6.

[vstinner]

Issue bpo-3826 is the same issue and has already patch with 2 unit tests. The patch is different: it uses "del self._sock".

[vstinner]

New patch:

• Change _socket.fileno() behaviour: raise socket.error("I/O operation on closed socket") (instead of returning -1) if the socket is closed
• Change SocketIO.fileno(): raise a socket.error("I/O operation on closed socket") if the file is closed, even if the socket is still open

TODO: Check all operations on closed _socket, socket or SocketIO and make sure that it raises also an error.

exakrun on IRC: What's the fascination with -1? Isn't this Python? How about an exception? Or at least a different type... None, perhaps? Returning -1 is what you would do in a language like C where you *can't* do something nice.

[vstinner]

I created a separated patch to block operation on closed socket: Issue bpo-4853.

So I simplified the patch to change on Modules/socketmodule.c and I reused self._checkClosed() in SocketIO.

[40b36f1b-5478-46ec-bc77-d5521c86f738]

edgecast support said the ftp server is ProFTPD version 1.3.1.

[vstinner]

I'm closing this issue because it's a duplicate of bpo-3826, issue with a longer history and more details about the different problems of closing a socket.