smtplib SMTP_SSL not working.

[6d75dac3-1fdd-4feb-8ba7-4bf3dec198fd]

BPO	4470
Nosy	@pitrou, @giampaolo, @tiran, @lmctv, @bitdancer, @csabella
Dependencies	bpo-4066: smtplib SMTP_SSL._get_socket doesn't return a value
Files	smtplib_01_default_port.diff smtplib_02_fix_ssl.diff smtplib_03_use_makefile.diff smtplib_04_remove_fakefile.diff smtplib_05_shutdown_socket.diff test_smtpnet.py smtplib_05_shutdown_socket_v2.patch: Updated smtplib_05_shutdown_socket.diff v2_01_fix_lmtp_init: Use class attribute for default connection port

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['type-bug', 'library'] title = 'smtplib SMTP_SSL not working.' updated_at = user = 'https://github.com/lmctv' ``` bugs.python.org fields: ```python activity = actor = 'cheryl.sabella' assignee = 'none' closed = True closed_date = closer = 'cheryl.sabella' components = ['Library (Lib)'] creation = creator = 'lcatucci' dependencies = ['4066'] files = ['12197', '12198', '12199', '12200', '12201', '12964', '22398', '22401'] hgrepos = [] issue_num = 4470 keywords = ['patch'] message_count = 21.0 messages = ['76640', '76759', '88235', '138235', '138519', '138520', '138521', '138523', '138552', '138555', '138561', '138581', '138688', '189591', '189593', '189594', '189597', '189617', '189619', '275022', '316120'] nosy_count = 8.0 nosy_names = ['pitrou', 'bronger', 'giampaolo.rodola', 'christian.heimes', 'lcatucci', 'r.david.murray', 'catalin.iacob', 'cheryl.sabella'] pr_nums = [] priority = 'normal' resolution = 'out of date' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue4470' versions = ['Python 3.4'] ```

[6d75dac3-1fdd-4feb-8ba7-4bf3dec198fd]

The enclosed patch does three things:

1. enables SMTP_SSL working: the _get_socket method was setting self.sock instead of returning the socket to the caller, which did reset self.sock to None
2. replace home-grown SSLFakeFile() with calls to ssl.socket's makefile() calls both in the starttls and in the SMTP_SSL cases
3. shutdown sockets before closing them, to avoid server-side piling and connection refused on connection-limited servers The last change is just a cosmetical refactoring, but it really helps the SMTP_SSL case: defaultport should really be a class attribute, instead of being set at \_init__ time.

[6d75dac3-1fdd-4feb-8ba7-4bf3dec198fd]

I've reworked the patch into a series, like haypo requested for poplib and imaplib.

[bitdancer]

With the closure of 4066 all the tests in the test patch pass, so I'm lowering the piority of this ticket.

I haven't reviewed the other patches, but the tests in the test patch appear to be doing tests in the setup method, which doesn't seem like a good idea.

[ddd81299-9e1f-4f0d-8e63-07bdcbc529fe]

I still have to apply Catucci's patch (or a modification of) after every Ubuntu installation or upgrade. Otherwise, I get

... File "/usr/lib/python2.7/smtplib.py", line 752, in __init SMTP.__init(self, host, port, localhostname, timeout) File "/usr/lib/python2.7/smtplib.py", line 239, in \_init__ (code, msg) = self.connect(host, port) File "/usr/lib/python2.7/smtplib.py", line 295, in connect self.sock = self._get_socket(host, port, self.timeout) File "/usr/lib/python2.7/smtplib.py", line 757, in _get_socket new_socket = socket.create_connection((host, port), timeout) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err socket.error: [Errno 111] Connection refused

But isn't it that bpo-4066 should have solved the critical part this issue pair 4066/4470?

[bitdancer]

Torsten, can you provide a clear, failing unittest for this?

[ddd81299-9e1f-4f0d-8e63-07bdcbc529fe]

No, I don't know how to do that. All I can provide is a minimal version of my code that triggers the above mentioned traceback. It is:

import smtplib

s = smtplib.SMTP_SSL("relay-auth.rwth-aachen.de")
s.login("***", "***")
s.sendmail("bronger@physik.rwth-aachen.de", "bronger.randys@googlemail.com"], "Hello")

I hope it helps to understand what I mean.

[ddd81299-9e1f-4f0d-8e63-07bdcbc529fe]

Sorry, it must be:

import smtplib

s = smtplib.SMTP_SSL("relay-auth.rwth-aachen.de")
s.login("***", "***")
s.sendmail("bronger@physik.rwth-aachen.de", ["bronger.randys@googlemail.com"], "Hello")

(A bracket was missing.)

[bitdancer]

According to your traceback you should be seeing the error in the first line (the creation of the SMTP_SSL object). If I run that line at the python prompt of python2.7.1, I get your connection failure. If I run it using 2.7 tip (or 3.3 tip), the connection succeeds. I know there have been other fixes in this area, but I don't know which one makes the difference.

Torsten, can you test with 2.7.2 and/or 2.7 tip?

I'm not sure what is left to do in this issue. Do you have an opinion, Lorenzo?

[6d75dac3-1fdd-4feb-8ba7-4bf3dec198fd]

On Fri, 17 Jun 2011, R. David Murray wrote:

RDM> RDM> R. David Murray \rdmurray@bitdance.com\ added the comment: RDM> RDM> According to your traceback you should be seeing the error in the RDM> first line (the creation of the SMTP_SSL object). If I run that line RDM> at the python prompt of python2.7.1, I get your connection failure.
RDM> If I run it using 2.7 tip (or 3.3 tip), the connection succeeds. I RDM> know there have been other fixes in this area, but I don't know which RDM> one makes the difference. RDM> RDM> Torsten, can you test with 2.7.2 and/or 2.7 tip? RDM> RDM> I'm not sure what is left to do in this issue. Do you have an RDM> opinion, Lorenzo? RDM>

Torsten, would you mind letting us know both the python and the .deb version by running

$ python2.7 --version

and

$ dpkg -l python2.7

At least in debian's python2.7_2.7.1-8, default_port is still an instance attribute instead of a class attribute.

If the ubuntu situation is the same, the missing patch is smtplib_01_default_port.diff , and you could work-around the problem by explicitly calling s = smtplib.SMTP_SSL("relay-auth.rwth-aachen.de", 465).

The patch got in with:

changeset: 69931:bcf04ced5ef1 branch: 2.7 parent: 69915:7c3a20b5943a user: Antoine Pitrou \solipsis@pitrou.net\ date: Sat May 07 19:59:33 2011 +0200 summary: Issue bpo-11927: SMTP_SSL now uses port 465 by default as documented. Patch by Kasun Herath.

The last hunk, which fixes LMTP is still missing.

@@ -776,8 +777,9 @@ authentication, but your mileage might vary."""

     ehlo_msg = "lhlo"
+    default_port = LMTP_PORT

-    def __init__(self, host = '', port = LMTP_PORT, local_hostname = 
None):
+    def __init__(self, host = '', port = 0, local_hostname = None):
         """Initialize a new instance."""
         SMTP.__init__(self, host, port, local_hostname)

Have a nice week-end, yours

lorenzo

+-------------------------+----------------------------------------------+ | Lorenzo M. Catucci | Centro di Calcolo e Documentazione | | catucci@ccd.uniroma2.it | Università degli Studi di Roma "Tor Vergata" | | | Via O. Raimondo 18 I-00173 ROMA ITALY | | Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 | +-------------------------+----------------------------------------------+

[5cd58f33-69a6-4482-a1e1-fa5df5a17f40]

Most of the problems in this issue were solved already so it could almost be closed:

• patch 1 was addressed in bpo-11927
• patch 2 was addressed in bpo-4066
• patches 3 and 4 were addressed in bpo-11893

Torsten's problem was addressed by bcf04ced5ef1.

I'm not sure what is left to do in this issue.

The only patch remaining is patch 5. I attached an updated version against tip of default branch. My patch mimics shutdown in imaplib.py in that it silences ENOTCONN. However I don't have a test that fails without the shutdown and I don't know if checking ENOTCONN is really needed. I tried to get shutdown to produce ENOTCONN by using Postfix as a server with smtpd_timeout=5s, connecting to it and waiting idle for more than 5 seconds before doing close(). In the Postfix logs I see that Postfix disconnects after 5 seconds of inactivity but doing shutdown afterwards doesn't trigger any exception so the ENOTCONN part remains unexercised.

My patch also adds shutdown method and SHUT_RDWR constant to mock_socket.py since otherwise test_smtplib fails.

(Added Antoine to nosy because he reviewed the patches for bpo-11927 and bpo-11893)

[ddd81299-9e1f-4f0d-8e63-07bdcbc529fe]

My Python version is "Python 2.7.1+" and the package is called "python2.7 2.7.1-5ubuntu2" (Ubuntu Natty).

[6d75dac3-1fdd-4feb-8ba7-4bf3dec198fd]

Just finished testing both 2.7 and default branches' socket close behaviour, and it seems 05 is not strictly needed.

I'd still prefer if smtplib_05_shutdown_socket_v2.patch since, this way the REMOTE socket close will be unconditionally correct, instead of being dependent on GC artifacts.

[6d75dac3-1fdd-4feb-8ba7-4bf3dec198fd]

I'd still prefer if smtplib_05_shutdown_socket_v2.patch could get in, \^^^^^^^^^^^^^^ since, this way the REMOTE socket close will be unconditionally correct, instead of being dependent on GC artifacts.

[ddd81299-9e1f-4f0d-8e63-07bdcbc529fe]

For five Ubuntu releases now, I apply this patch. In contrast to Catalin's statement, it is not solved for me unless the upstream changes of two years ago haven't yet found their way into Ubuntu. I'm currently using Python 2.7.4 on Ubuntu 13.04. That said, the patch solves my issue every time.

[pitrou]

When you say "I apply this patch", you mean smtplib_05_shutdown_socket_v2.patch, right?

[pitrou]

In any case, I'm growing wary of bugfix regressions right now, so I would only apply the patch on the default branch.

[ddd81299-9e1f-4f0d-8e63-07bdcbc529fe]

Sorry, after having had another look at it, I realised that I have a different SSMTP issue now, non-Python-related. So for me, Ubuntu 13.04 indeed solves my old issue.

[bitdancer]

Lorenzo, any chance you could supply a unit test that fails without smtplib_05_shutdown_socket.diff applied?

[6d75dac3-1fdd-4feb-8ba7-4bf3dec198fd]

On Sun, 19 May 2013, R. David Murray wrote:

RDM> RDM> R. David Murray added the comment: RDM> RDM> Lorenzo, any chance you could supply a unit test that fails without RDM> smtplib_05_shutdown_socket.diff applied? RDM>

It would really be a functional test, since the problem with half-open connection pile-up stems from smtp server's access control rules.

To test we should setup a fake smtp server, which forbids having multiple connections from the same IP address, and connect twice in a row to the fake server. I'm not sure I'm able to implement both an smtpd.py server serving more than one connection and the client code in a race-free way in the same "unit" test. Will try in the next week.

Thank you very much,

lorenzo

[tiran]

The bug is 8 years old and hasn't seen activity for three years. Is SMTP over SSL still broken for you?

[csabella]

This issue has been in pending status for over 18 months following Christian's question to the OP, so closing as out of date.