Seeing DIGEST-MD5 failure when authenticating SMTP/STARTTLS

[pprindeville]

I'm running Fedora 23 on a VM, with Authen::SASL 2.16 and Net::SMTP 3.08.

I've used the attached test script. test_pl.txt

I get this output when I run it:

$ USER=philipp ./test.pl 
Net::SMTP>>> Net::SMTP(3.08)
Net::SMTP>>>   Net::Cmd(3.08)
Net::SMTP>>>     Exporter(5.72)
Net::SMTP>>>   IO::Socket::IP(0.37)
Net::SMTP>>>     IO::Socket(1.38)
Net::SMTP>>>       IO::Handle(1.35)
Net::SMTP=GLOB(0x1c60500)<<< 220 mail.redfish-solutions.com ESMTP Sendmail 8.15.2/8.15.2; Mon, 13 Jun 2016 11:29:06 -0600
Net::SMTP=GLOB(0x1c60500)>>> EHLO localhost.localdomain
Net::SMTP=GLOB(0x1c60500)<<< 250-mail.redfish-solutions.com Hello [192.168.1.76], pleased to meet you
Net::SMTP=GLOB(0x1c60500)<<< 250-ENHANCEDSTATUSCODES
Net::SMTP=GLOB(0x1c60500)<<< 250-PIPELINING
Net::SMTP=GLOB(0x1c60500)<<< 250-8BITMIME
Net::SMTP=GLOB(0x1c60500)<<< 250-SIZE
Net::SMTP=GLOB(0x1c60500)<<< 250-DSN
Net::SMTP=GLOB(0x1c60500)<<< 250-AUTH DIGEST-MD5 CRAM-MD5
Net::SMTP=GLOB(0x1c60500)<<< 250-STARTTLS
Net::SMTP=GLOB(0x1c60500)<<< 250-DELIVERBY
Net::SMTP=GLOB(0x1c60500)<<< 250 HELP
connected to mail.redfish-solutions.com
banner was 'mail.redfish-solutions.com ESMTP Sendmail 8.15.2/8.15.2; Mon, 13 Jun 2016 11:29:06 -0600'
want: DIGEST-MD5
got: DIGEST-MD5 CRAM-MD5
Net::SMTP=GLOB(0x1c60500)>>> AUTH DIGEST-MD5
Net::SMTP=GLOB(0x1c60500)<<< 334 bm9uY2U9IlRmOCsyY2pwNWx2cnE4RTZCcTZCTExJSXN4ZEtjMUUydlVvVnJmeVZWNFk9IixyZWFsbT0ibWFpbCIscW9wPSJhdXRoLGF1dGgtaW50LGF1dGgtY29uZiIsY2lwaGVyPSJyYzQtNDAscmM0LTU2LHJjNCxkZXMsM2RlcyIsbWF4YnVmPTgxOTIsY2hhcnNldD11dGYtOCxhbGdvcml0aG09bWQ1LXNlc3M=
Net::SMTP=GLOB(0x1c60500)<<< (decoded) nonce="Tf8+2cjp5lvrq8E6Bq6BLLIIsxdKc1E2vUoVrfyVV4Y=",realm="mail",qop="auth,auth-int,auth-conf",cipher="rc4-40,rc4-56,rc4,des,3des",maxbuf=8192,charset=utf-8,algorithm=md5-sess
Net::SMTP=GLOB(0x1c60500)>>> (decoded) authzid="philipp",charset=utf-8,cnonce="fa863102174839f0d56d2386a6b9e71e",digest-uri="smtp/192.168.1.3",nc=00000001,nonce="Tf8+2cjp5lvrq8E6Bq6BLLIIsxdKc1E2vUoVrfyVV4Y=",qop=auth-int,realm="mail",response=d14a2e8dbb152207874fe4d0315ce7fc,username="philipp"
Net::SMTP=GLOB(0x1c60500)>>> YXV0aHppZD0icGhpbGlwcCIsY2hhcnNldD11dGYtOCxjbm9uY2U9ImZhODYzMTAyMTc0ODM5ZjBkNTZkMjM4NmE2YjllNzFlIixkaWdlc3QtdXJpPSJzbXRwLzE5Mi4xNjguMS4zIixuYz0wMDAwMDAwMSxub25jZT0iVGY4KzJjanA1bHZycThFNkJxNkJMTElJc3hkS2MxRTJ2VW9WcmZ5VlY0WT0iLHFvcD1hdXRoLWludCxyZWFsbT0ibWFpbCIscmVzcG9uc2U9ZDE0YTJlOGRiYjE1MjIwNzg3NGZlNGQwMzE1Y2U3ZmMsdXNlcm5hbWU9InBoaWxpcHAi
Net::SMTP=GLOB(0x1c60500)<<< 334 cnNwYXV0aD1hZjI3NTNiMmMyMDYwYTNjNWU4MDdkOGZjYjIxOWUwMg==
Net::SMTP=GLOB(0x1c60500)<<< (decoded) rspauth=af2753b2c2060a3c5e807d8fcb219e02
Net::SMTP=GLOB(0x1c60500)>>> (decoded) 
Net::SMTP=GLOB(0x1c60500)>>> 
Net::SMTP=GLOB(0x1c60500)<<< 235 2.0.0 OK Authenticated
Net::SMTP=GLOB(0x1c60500)>>> MAIL FROM:<philipp>
Net::SMTP: Net::Cmd::getline(): unexpected EOF on command channel:  at ./test.pl line 51.
couldn't do MAIL FROM at ./test.pl line 51.
$ 

I'm using a slightly modified version of Net::SMTP to trace the cleartext SASL messages. That change is described in this PR.

I don't know enough about SASL/DIGEST-MD5 to know if the "rspauth=..." message coming back should be a 3xx or a 2xx message. Can anyone confirm this?

If I look at ::need_step and ::is_success then I get the values 1 and 0, respectively, before sending the blank line; then 1 and 0 after sending it. Sending a blank line over an SMTP connection seems wrong to me, but then so does the server staying in the 3xx state even after indicating a successful negotiation.

I know that DIGEST-MD5 is going to be deprecated soon, but there will be a lot of out-of-date servers out there using it for a while.

Anyone else able to reproduce this?

[zdm]

Did you solved this issue?

[ehuelsmann]

@hrs-allbsd, @zdm I'm interested in solving this issue, but I have no idea what the actual problem is. The issue with Net::SMTP listed above (issue 43), indicates something about auth-conf and auth-int. Can you point me to documentation as to what that is? Before I can even start to fix this, I'll need to understand what you're thinking is the issue.

[ehuelsmann]

@pprindeville does the workaround in steve-m-hay/perl-libnet#43 fix this for you?

[hrs-allbsd]

@hrs-allbsd, @zdm I'm interested in solving this issue, but I have no idea what the actual problem is. The issue with Net::SMTP listed above (issue 43), indicates something about auth-conf and auth-int. Can you point me to documentation as to what that is? Before I can even start to fix this, I'll need to understand what you're thinking is the issue.

The problem is Net::SMTP uses plain text even if the negotiation of Authen::SASL results in a connection with auth-int and/or auth-conf. These protection layers are described in [1]. Net::Cmd::datasend() does not honor them. Therefore an SMTP connection fails just after such an authentication succeeds.

[1] https://datatracker.ietf.org/doc/html/rfc2831#section-2.3

Strictly speaking, this is not a Net::SMTP's fault. It calls Authen::SASL::client_new() for the authentication, and this authentication module always accepts "auth-int" and "auth-conf" when the DIGEST-MD5 mechanism is enabled on the client and the server sends these capabilities. Authen::SASL has no way in client_new() to change the security options, while server_new() supports "no_integrity" and "no_confidentiality" options. My patch effectively disables these security layers on the client side by setting maxssf=0.

However, I admit it is just a workaround and believe that the best solution for Net::SMTP is to support auth-int and auth-conf properly. It should be able to reuse Authen::SASL::Perl::DIGEST_MD5::encode() and decode() methods.

[pprindeville]

However, I admit it is just a workaround and believe that the best solution for Net::SMTP is to support auth-int and auth-conf properly. It should be able to reuse Authen::SASL::Perl::DIGEST_MD5::encode() and decode() methods.

I concur with the above.

[hrs-allbsd]

This is an implementation that uses Authen::SASL::Perl::DIGEST_MD5::encode(). I confirmed that 10-byte HMAC_MD5 + 2-byte type + 4-byte sequence number was appended to each message when auth-int was enabled, but my server (sendmail + cyrus-sasl) rejected them for some reason. I am investigating the cause now...

https://github.com/hrs-allbsd/perl-libnet/commit/683c30315938e29b5299cd4640ed8ece04e5b922

[hrs-allbsd]

I confirmed this patch works: https://github.com/hrs-allbsd/perl-libnet/commit/6212af81288f31a7b0c8bdea087615b41b37a503

@pprindeville Is it possible for you to test it? Your original report was posted a long time ago, but if you can still try the patch, please test it. I will submit another pull request to the perl-libnet repo after someone confirms if it works. Please disregard 683c303 in my previous comment because I made a forced commit.

@ehuelsmann I think no change is required on Authen::SASL side after all, but I have two questions: 1) Are the encode()/decode() methods safe to use from perl-libnet? I am concerned about it because these methods are not listed as public interfaces in the document. 2) Is it a good idea to handle the buffer length field in the encode()/decode()? Cyrus SASL library handles this field as a part of the mechanism-specific encoding/decoding routines. Authen::SASL::Perl::DIGEST_MD5::encode() might also want to return the results with the length field since it simplifies the consumers of these methods.

[ehuelsmann]

@hrs-allbsd Thanks for going these lengths to see where the problem resides and what's the best design to solve this issue. I added a review comment on the commit. Let me know what you think about it.