IOException --> SocketException (An existing connection was forcibly closed by the remote host) when using TLS

[knocte]

I'm trying to connect to my own postfix server configured on Ubuntu 14.04.

I configured it to disallow any old SSL versions, and to disallow mail rely to non-authenticated users.

I've tested it and it works well from Thunderbird, using STARTTLS (instead of SSL/TLS) and "Normal password" authentication settings, port 587.

However, trying to send mail with MailKit results in this exception.

The code:

            var secureSmtpServerUsername = "test";
            var secureSmtpServerDomainForFromAddress = "@staff.mydomain.com";
            var bodyPart = new TextPart("html"){
                Text = body
            };

            var from = new[] { new MailboxAddress(from_name, secureSmtpServerUsername + secureSmtpServerDomainForFromAddress) };

            var mail = new MimeMessage(from, toList, subject, bodyPart);

            using (var smtpClient = new SmtpClient())
            {
                smtpClient.AuthenticationMechanisms.Remove("XOAUTH2");
                smtpClient.Connect(secureSmtpServerHost, (int)secureSmtpServerPort, true);
                smtpClient.AuthenticationMechanisms.Remove("XOAUTH2");

                smtpClient.Authenticate(secureSmtpServerUsername, secureSmtpServerPassword);
                smtpClient.Send(mail);
            }

The exception happens at the Connect() method, before we had a chance to Authenticate().

Any ideas?

Full ex.ToString():

{System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at MailKit.Net.Smtp.SmtpClient.Connect(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at MailKit.MailService.Connect(Uri uri, CancellationToken cancellationToken)
   at BTCEX.Util.SmtpEmailSender.SendEmail(String[] toemail, String subject, String body, Boolean isBodyHtml, String from_name, String bccAddress) in c:\Users\Andres\Documents\Code\api\api\BTCEX\BTCEX.Util\SmtpEmailSender.cs:line 93
   at BTCEX.Util.SmtpEmailSender.<>c__DisplayClass5.<SendEmailAsync>b__4() in c:\Users\Andres\Documents\Code\api\api\BTCEX\BTCEX.Util\SmtpEmailSender.cs:line 139}

[knocte]

Forgot to say, I'm using these versions of {Mail/Mime}Kit, under Windows 8.1 (MS.NET 4.5):

    <Reference Include="MailKit, Version=1.2.0.0, Culture=neutral, PublicKeyToken=4e064fe7c44a8f1b, processorArchitecture=MSIL">
      <HintPath>..\packages\MailKit.1.2.21\lib\net45\MailKit.dll</HintPath>
      <Private>True</Private>
    </Reference>
    </Reference>
    <Reference Include="MimeKit, Version=1.2.0.0, Culture=neutral, PublicKeyToken=bede1c8a46c66814, processorArchitecture=MSIL">
      <HintPath>..\packages\MimeKit.1.2.22\lib\net45\MimeKit.dll</HintPath>
    </Reference>

[knocte]

I've tried attaching a protocolLogger this way:

var smtpClient = new SmtpClient(new ProtocolLogger("C:\\tmp\\MailKit.log"))

And it creates the MailKit.log file, but it ends up with 0 size :-(

[knocte]

Looking at the logs of postfix is not helpful either, I just see:

Mar 17 07:10:39 postfix postfix/submission/smtpd[18401]: connect from 203185050202.static.ctinets.com[203.185.50.202]
Mar 17 07:10:39 postfix postfix/submission/smtpd[18401]: lost connection after UNKNOWN from 203185050202.static.ctinets.com[203.185.50.202]
Mar 17 07:10:39 postfix postfix/submission/smtpd[18401]: disconnect from 203185050202.static.ctinets.com[203.185.50.202]

[knocte]

If I use:

smtpClient.Connect(secureSmtpServerHost, port:(int)secureSmtpServerPort, options:SecureSocketOptions.StartTls);

I then get a different exception:

{System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at MailKit.Net.Smtp.SmtpClient.Connect(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at BTCEX.Util.SmtpEmailSender.SendEmail(String[] toemail, String subject, String body, Boolean isBodyHtml, String from_name, String bccAddress) in c:\Users\Andres\Documents\Code\api\api\BTCEX\BTCEX.Util\SmtpEmailSender.cs:line 100
   at BTCEX.Util.SmtpEmailSender.<>c__DisplayClass5.<SendEmailAsync>b__4() in c:\Users\Andres\Documents\Code\api\api\BTCEX\BTCEX.Util\SmtpEmailSender.cs:line 153}

[knocte]

System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

BTW, the certificate in the email server is fine (otherwise Thunderbird would give a warning about a bad certificate). The email server is even hardened with DKIM&SPF so that emails received from it in Gmail (for example) don't have Phishing or Non-encryption warnings (those annoying, but useful, open red lock icons that Gmail now shows).

[jstedfast]

Hey knocte!

The problem with the Connect (host, 587, true); is that you are trying to connect to a clear-text SMTP port using an SSL-wrapped stream.

As far as the certificate issue, the problem is probably that the Mono cert database does not trust the CA cert that signed your mail server's cert.

You can override the behavior by setting a custom callback on client.ServerCertificateValidationCallback such as:

client.ServerCertificateValidationCallback = () => true;

FWIW, this is the default certificate validation logic: https://github.com/jstedfast/MailKit/blob/master/MailKit/MailService.cs#L253

I wouldn't really recommend that for production code since it accepts all self-signed certs, but it might serve as a starting point for your own custom logic.

[jstedfast]

oh, and make sure to continue using the second Connect() overload that you started using since that gives you more control over what SSL method to use.

The Connect() overload that takes a true/false argument only allows the specification of whether the initial connection should be made using an SslStream or not. If you pass false, it will try to use STARTTLS if the server supports it, but I don't like that API because it's not obvious.

[knocte]

Hey Jeff! thanks for the answer. See inline:

but I don't like that API because it's not obvious.

Agreed.

As far as the certificate issue, the problem is probably that the Mono cert database does not trust the CA cert that signed your mail server's cert.

This cannot be the problem, because, let me remind you, I'm not using Mono currently, but MS.NET ;-) So how do you suggest I could troubleshoot the certificate problem?

[jstedfast]

I'd recommend overriding the callback and seeing what errors it gives.

Remember: MailKit doesn't control the errors or the cert validation process, that is all done by SslStream.

[knocte]

My god, you wouldn't believe what was the problem!

I added the infamous

ServicePointManager.ServerCertificateValidationCallback = delegate(object s, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
    return true;
};

And put a breakpoint on it. Then I found out one of the certificates was signed by AVAST (my fucking antivirus!). Once I disabled my antivirus, sending email started working.

FFS! I'm dying to go back to using Linux where I don't need this kind of shitty software...

Thanks for your help Jeff, always appreciated.

[knocte]

For the record, I guess this is more info about my problem: https://www.avast.com/no-no/faq.php?article=AVKB91

[jstedfast]

Wow, interesting. I had no idea Avast did that...