search

mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.58k stars 1.15k forks source link

Cannot start TLS: handshake failure #2185

Closed tehXor closed 5 years ago

tehXor commented 5 years ago

Recently i'm getting Cannot start TLS: handshake failure for gmx.de and web.de recipients. Other provider like GMail work. However there are a lot of mails to GMX and web.de apperently.

Now I'm not sure if the problem lies on my end, within mailcow or GMX/web.de and the first question is, what's the best way to debug this problem in the containers? All I get from the logs is this:

1/16/2019, 1:31:28 AM | info | SSL_connect error to mx01.emig.gmx.net[212.227.17.5]:25: Connection timed out
1/16/2019, 1:31:28 AM | info | 8BE0C60724: Cannot start TLS: handshake failure
1/16/2019, 1:31:28 AM | info | SSL_connect error to mx00.emig.gmx.net[212.227.15.9]:25: Connection timed out
1/16/2019, 1:31:28 AM | info | B980C60729: Cannot start TLS: handshake failure
1/16/2019, 1:31:27 AM | info | SSL_connect error to mx-ha02.web.de[212.227.17.8]:25: Connection timed out
1/16/2019, 1:31:27 AM | info | 5EDA56072C: Cannot start TLS: handshake failure

...only a lot more.

Since I saw a timeout I already started by installing inetutils in the postfix container to check if it basically can connect to the hosts in question and that works. So it probably really is some specific SSL/TLS problem.

The problem started with a version which was around 4 weeks ago (but the problem itself came up just recently). I now upgraded to the recent master branch and regarding SSL/TLS my postfix configuration is basically the same as in master. This problem still persists though.

Any hint to better debug this or temporally workaround (even if unsecure) would be welcome. Thanks!

andryyy commented 5 years ago

I have seen this happening with a wrong MTU.

Am 16.01.2019 um 01:53 schrieb Till notifications@github.com:

Recently i'm getting Cannot start TLS: handshake failure for gmx.de and web.de recipients. Other provider like GMail work. However there are a lot of mails to GMX and web.de apperently.

Now I'm not sure if the problem lies on my end, within mailcow or GMX/web.de and the first question is, what's the best way to debug this problem in the containers? All I get from the logs is this:

1/16/2019, 1:31:28 AM | info | SSL_connect error to mx01.emig.gmx.net[212.227.17.5]:25: Connection timed out 1/16/2019, 1:31:28 AM | info | 8BE0C60724: Cannot start TLS: handshake failure 1/16/2019, 1:31:28 AM | info | SSL_connect error to mx00.emig.gmx.net[212.227.15.9]:25: Connection timed out 1/16/2019, 1:31:28 AM | info | B980C60729: Cannot start TLS: handshake failure 1/16/2019, 1:31:27 AM | info | SSL_connect error to mx-ha02.web.de[212.227.17.8]:25: Connection timed out 1/16/2019, 1:31:27 AM | info | 5EDA56072C: Cannot start TLS: handshake failure ...only a lot more.

Since I saw a timeout I already started by installing inetutils in the postfix container to check if it basically can connect to the hosts in question and that works. So it probably really is some specific SSL/TLS problem.

The problem started with a version which was around 4 weeks ago (but the problem itself came up just recently). I now upgraded to the recent master branch and regarding SSL/TLS my postfix configuration is basically the same as in master. This problem still persists though.

Any hint to better debug this or temporally workaround (even if unsecure) would be welcome. Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

tehXor commented 5 years ago

Very helpful tip @andryyy thanks a lot - and this made all further troubleshooting invalid since exactly that was the problem. Our VMs got moved to a new virtual network with a MTU of 1400 two days ago...

The fast solution is adding com.docker.network.driver.mtu to the docker-compose file so that the first part of the network looks like this:

networks:
  mailcow-network:
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: 1400

In our case we will now also check if there are better global options, which doesn't involve updatating the docker-compose files of every project. Should we find one, I will add it here.

andryyy commented 5 years ago

Nice! :)

rtulke commented 5 years ago

Same error here and the hint with mtu does not work for me any idea?

part of my docker-compose.yml:

...
...
networks:
  mailcow-network:
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: 1400
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: ${IPV4_NETWORK:-172.22.1}.0/24
        - subnet: ${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}

Undelivered Mail Returned to Sender Message:

This is the mail system at host mx.foobar.com.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                  The mail system

<webmaster@xmail.com>: Cannot start TLS: handshake failure
Reporting-MTA: dns; mx.foobar.com
X-Postcow-Queue-ID: BC4BB880F59
X-Postcow-Sender: rfc822; billing@host.com
Arrival-Date: Thu, 21 Mar 2019 13:23:53 +0100 (CET)

Final-Recipient: rfc822; webmaster@xmail.com
Original-Recipient: rfc822;webmaster@xmail.com
Action: failed
Status: 5.7.5
Diagnostic-Code: X-Postcow; Cannot start TLS: handshake failure
Von: Billing Service <billing@host.com>
Betreff: Ihre Rechnung
Datum: 21. März 2019 um 13:23:46 MEZ
An: Customer Name <webmaster@xmail.com>
andryyy commented 5 years ago

Why do you think this is a MTU issue exactly? Just want to be sure. What is your default MTU on the primary NIC?

rtulke commented 5 years ago 
root@mx ~/dev/mailcow-dockerized # ifconfig eth0|grep mtu
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

I've also tried 1500 but it does also not work :(

andryyy commented 5 years ago

Please don't change the stacks MTU if you don't have to.

It is just a TLS handshake failure then. Check if that happens with other servers, too. Perhaps you enforce TLS for this mailbox and the remote servers protocol/ciphers are just too old.

rtulke commented 5 years ago

It does not affect all remote mailboxes most functioning but there are some that have problems. I had just read the headline "Cannort start TLS" and thought that it might have been there.

Where on my side could I accept that? (old protocol/ciphers) or where can I force TLS for a remote mailbox/domain?

andryyy commented 5 years ago

You should debug if that’s the problem after all.

You could try to connect with OpenSSL on these ports. Also login to that mailbox via mailcow UI and see if it has encryption enforced. It would fall back to a plain session if encryption fails and no encryption was enforced.

If it’s an incoming mail problem, check the certificate! If you use an external acme client, you might have missed to copy the renewed certificate.

rtulke commented 5 years ago

Hi Andre i got this:

root@mx ~/dev/mailcow-dockerized # openssl s_client -showcerts -connect mx-in.planet-school.de:587
139631594094656:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:108:
139631594094656:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:109:
connect:errno=111
root@mx ~/dev/mailcow-dockerized # openssl s_client -showcerts -connect mx-in.planet-school.de:993
140718373584960:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:108:
140718373584960:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:109:
connect:errno=111
root@mx ~/dev/mailcow-dockerized # openssl s_client -showcerts -connect mx-in.planet-school.de:995
140515483316288:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:108:
140515483316288:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:109:
connect:errno=111
root@mx ~/dev/mailcow-dockerized # openssl s_client -showcerts -connect mx-in.planet-school.de:25
CONNECTED(00000003)
139647800877120:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:252:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 176 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1553194492
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

and 

root@mx ~/dev/mailcow-dockerized # openssl s_client -showcerts -connect mx-in.planet-school.de:465
CONNECTED(00000003)
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=1 C = BE, O = GlobalSign nv-sa, CN = AlphaSSL CA - SHA256 - G2
verify return:1
depth=0 OU = Domain Control Validated, CN = *.planet-school.de
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=*.planet-school.de
   i:/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G2
-----BEGIN CERTIFICATE-----
MIIG8jCCBdqgAwIBAgIMNElJwE3VQmDoumMOMA0GCSqGSIb3DQEBCwUAMEwxCzAJ
BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSIwIAYDVQQDExlB
bHBoYVNTTCBDQSAtIFNIQTI1NiAtIEcyMB4XDTE4MDkxOTEwMTYwMFoXDTE5MDky
MDEwMTYwMFowQDEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRhdGVkMRsw
GQYDVQQDDBIqLnBsYW5ldC1zY2hvb2wuZGUwggIiMA0GCSqGSIb3DQEBAQUAA4IC
DwAwggIKAoICAQC3mtJcRqQ+DyTPfDkfdwVzgxgzCCV072XMwMYfHpoId9j6mcUf
kCtqeQXN13s4uD8I5xwbuezFkKEo8XiDK4yU3YsZZifIim5nQ6ytNQZIKd5nCYxM
TnLMcoEOjbGNpf5tw24FQqhmmSCbMIzq2v+8H8FJvDYV3/UdMQpBUz+qMiYcc0Kt
YU7sf3t/STcIoH0mb87Q6/uesTbb0nL6BtHg4Nt3GvcKaXuHP6gYyBIk5wgFL+yx
bnk2K23nm8I1CQqsOR0hbtPjiP1Koo2RcAXNe3vp5MhiI6OZqZ0VXQTAWWgKjjZl
laHlpPf4fiHYEFC0hE0bodsDJPa2MaS7zXpN1hn1h57pyHkO/GfC7MpLtLUsHXmp
JQ6oWF/UUeMZHKm8vxasQOb+csAGFcKkybv+McuHjNfXBECEUdjRWybA7JeQ2Zy2
4e26LIhwCNnrNP8z5R+qn81D4THxkczUwWDj05Y3Tg8HUB17jnvOvRamILyCMcsE
8FCFh2w04DXZ+eIMibQ7FEUQLpt+Vsjm3LHRVbI1ejL/m5OI1Anwda8oElR3dkxX
wdDVtaktZiZLLUpQKHEiIvcijv6/qkB7+PfV44hYC7uWp31wan9dyMW+p7LxLtg3
mzMRibC6rook/VcAJ7MT+F/3y4gr2skGSAPHhRKeV9xu85mBlCgLStYx3wIDAQAB
o4IC3jCCAtowDgYDVR0PAQH/BAQDAgWgMIGJBggrBgEFBQcBAQR9MHswQgYIKwYB
BQUHMAKGNmh0dHA6Ly9zZWN1cmUyLmFscGhhc3NsLmNvbS9jYWNlcnQvZ3NhbHBo
YXNoYTJnMnIxLmNydDA1BggrBgEFBQcwAYYpaHR0cDovL29jc3AyLmdsb2JhbHNp
Z24uY29tL2dzYWxwaGFzaGEyZzIwVwYDVR0gBFAwTjBCBgorBgEEAaAyAQoKMDQw
MgYIKwYBBQUHAgEWJmh0dHBzOi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRv
cnkvMAgGBmeBDAECATAJBgNVHRMEAjAAMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6
Ly9jcmwyLmFscGhhc3NsLmNvbS9ncy9nc2FscGhhc2hhMmcyLmNybDAvBgNVHREE
KDAmghIqLnBsYW5ldC1zY2hvb2wuZGWCEHBsYW5ldC1zY2hvb2wuZGUwHQYDVR0l
BBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBRTbQE8Thnmn87JNSEn
UcK/JnmZhzAfBgNVHSMEGDAWgBT1zdU8CFD5ak86t5faVoPmadJo9zCCAQYGCisG
AQQB1nkCBAIEgfcEgfQA8gB3AKS5CZC0GFgUh7sTosxncAo8NZgE+RvfuON3zQ7I
DdwQAAABZfFUXSEAAAQDAEgwRgIhAPvljgw0QEIlxjEQQCyFTxDyDLW/QM8Vw08Y
8n4Hr3zsAiEAiWSzL9ReVyMVBCqlb4wfUf7DxIFT2V5ZCOcbvaS1kpUAdwBvU3as
MfAxGdiZAKRRFf93FRwR2QLBACkGjbIImjfZEwAAAWXxVFwhAAAEAwBIMEYCIQCq
iIavPWX9WXihYucXw+Vz3+8+gva7UzBbyXRO5EbLCQIhAPDonoTU8gK5yaznEjIQ
arp5Zdk29Y764rGg2RmTk227MA0GCSqGSIb3DQEBCwUAA4IBAQCjrf3INxjJ/F6T
zN3vlc030uxNN7JSCkKkZGVYAt7haoe6s1CiXTeiRDSCRfoTnVj6JriRaocBRzcA
HxcsNCs/7LdRMujFUfN/8BADJunJBWxujyR2pIRkwdFv2knoNpoXG5HqxkTGaACg
MybEFcd7PK2T9RANs1ZCi2vv09OFBbUNJPEBYwmWRV3XjTCqvRIFzRdKw5BsFXPT
omtKv8GDt+VrxXkiIQXL8gx/KdUj/HEMl99UmuZlJ4jPz45kiijT9OBG5k6p6Csv
i9GvgHTS6LysHMw5uhezjuotrr9bgoQgyI7H1HsPUntx9p5yDMf7giC9bte3liBl
42cZpPyn
-----END CERTIFICATE-----
 1 s:/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
-----BEGIN CERTIFICATE-----
MIIETTCCAzWgAwIBAgILBAAAAAABRE7wNjEwDQYJKoZIhvcNAQELBQAwVzELMAkG
A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv
b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw0xNDAyMjAxMDAw
MDBaFw0yNDAyMjAxMDAwMDBaMEwxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i
YWxTaWduIG52LXNhMSIwIAYDVQQDExlBbHBoYVNTTCBDQSAtIFNIQTI1NiAtIEcy
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2gHs5OxzYPt+j2q3xhfj
kmQy1KwA2aIPue3ua4qGypJn2XTXXUcCPI9A1p5tFM3D2ik5pw8FCmiiZhoexLKL
dljlq10dj0CzOYvvHoN9ItDjqQAu7FPPYhmFRChMwCfLew7sEGQAEKQFzKByvkFs
MVtI5LHsuSPrVU3QfWJKpbSlpFmFxSWRpv6mCZ8GEG2PgQxkQF5zAJrgLmWYVBAA
cJjI4e00X9icxw3A1iNZRfz+VXqG7pRgIvGu0eZVRvaZxRsIdF+ssGSEj4k4HKGn
kCFPAm694GFn1PhChw8K98kEbSqpL+9Cpd/do1PbmB6B+Zpye1reTz5/olig4het
ZwIDAQABo4IBIzCCAR8wDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8C
AQAwHQYDVR0OBBYEFPXN1TwIUPlqTzq3l9pWg+Zp0mj3MEUGA1UdIAQ+MDwwOgYE
VR0gADAyMDAGCCsGAQUFBwIBFiRodHRwczovL3d3dy5hbHBoYXNzbC5jb20vcmVw
b3NpdG9yeS8wMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5nbG9iYWxzaWdu
Lm5ldC9yb290LmNybDA9BggrBgEFBQcBAQQxMC8wLQYIKwYBBQUHMAGGIWh0dHA6
Ly9vY3NwLmdsb2JhbHNpZ24uY29tL3Jvb3RyMTAfBgNVHSMEGDAWgBRge2YaRQ2X
yolQL30EzTSo//z9SzANBgkqhkiG9w0BAQsFAAOCAQEAYEBoFkfnFo3bXKFWKsv0
XJuwHqJL9csCP/gLofKnQtS3TOvjZoDzJUN4LhsXVgdSGMvRqOzm+3M+pGKMgLTS
xRJzo9P6Aji+Yz2EuJnB8br3n8NA0VgYU8Fi3a8YQn80TsVD1XGwMADH45CuP1eG
l87qDBKOInDjZqdUfy4oy9RU0LMeYmcI+Sfhy+NmuCQbiWqJRGXy2UzSWByMTsCV
odTvZy84IOgu/5ZR8LrYPZJwR2UcnnNytGAMXOLRc3bgr07i5TelRS+KIz6HxzDm
MTh89N1SyvNTBCVXVmaU6Avu5gMUTu79bZRknl7OedSyps9AsUSoPocZXun4IRZZ
Uw==
-----END CERTIFICATE-----
---
Server certificate
subject=/OU=Domain Control Validated/CN=*.planet-school.de
issuer=/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G2
---
No client certificate CA names sent
Server Temp Key: DH, 1024 bits
---
SSL handshake has read 3992 bytes and written 374 bytes
Verification: OK
---
New, SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 39519FA92ABE4C0C78038A97D0D9798A0F156527CEA5DDE1FC71A13E09691436
    Session-ID-ctx:
    Master-Key: 0196242022A3D232C47641F71403B773D76BE375A4D78DA1E63B832B959E0740CF3CC23A85AE889E0E16483DBCEB46E2
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - 5b 4d 2b 46 55 15 21 99-7c ba 08 76 a0 76 4d 45   [M+FU.!.|..v.vME
    0010 - 8c 81 d5 73 e9 61 33 82-4f a9 91 8a 31 9f c6 57   ...s.a3.O...1..W
    0020 - e0 d1 a9 2b c8 3c bc 88-b9 eb bd c1 11 d1 ce 2c   ...+.<.........,
    0030 - 54 9a f3 37 29 0e e4 25-1d 67 b9 e4 86 a8 bb 0d   T..7)..%.g......
    0040 - 68 24 ac 12 b6 40 d1 6c-e4 f3 1a 23 ca a5 20 33   h$...@.l...#.. 3
    0050 - ba 2c 09 ea f7 74 c1 e2-ab cf d1 9f 96 68 a9 8e   .,...t.......h..
    0060 - 34 79 61 0d 90 e5 55 ef-fd 33 41 8d 8f 66 d0 09   4ya...U..3A..f..
    0070 - 03 21 4f 2f 3e 82 5a d0-8d 86 27 2f 69 a2 51 44   .!O/>.Z...'/i.QD
    0080 - 24 27 c5 af 0d 58 19 99-b6 fe d4 0e 93 d5 eb 27   $'...X.........'
    0090 - a4 e6 33 c1 98 e5 44 03-81 d8 78 d3 80 89 ad 80   ..3...D...x.....

    Start Time: 1553194623
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
220 mx-in.planet-school.de ESMTP PS-MXCluster

is a bit confusing, here it works (same mail-to):

Mar 20 10:36:03 mx postfix/smtp[23597]: Anonymous TLS connection established to mx-in.planet-school.de[194.116.186.73]:25: TLSv1 with cipher ADH-AES256-SHA (256/256 bits)
Mar 20 10:36:03 mx postfix/smtp[23597]: 666A2880F1D: to=<webmaster@xmail.com>, orig_to=<redaktion@mailhost.de>, relay=mx-in.planet-school.de[194.116.186.73]:25, delay=1.3, delays=0.96/0.02/0.14/0.23, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A6EA3804F8)

and same host different messages:

Mar 15 14:20:01 mx enforced-tls-smtp/smtp[23456]: SSL_connect error to mx-in.planet-school.de[194.116.186.73]:25: -1
Mar 15 14:20:01 mx enforced-tls-smtp/smtp[23456]: BA0CB881A81: to=<webmaster@xmail.com>, relay=mx-in.planet-school.de[194.116.186.73]:25, delay=0.28, delays=0.19/0.02/0.07/0, dsn=5.7.5, status=bounced (Cannot start TLS: handshake failure)

i also tried the web based client sogo and i got mailer returns... i don't know what it is but i've only had this problem since the last upgrade 1-2 weeks ago

andryyy commented 5 years ago

enforced-tls-smtp/smtp

That’s exactly what I meant. We tightened the mandatory ciphers. You enforce TLS for that mailbox, so Postfix uses only secure protocols and ciphers and rejects if it cannot negotiate a secure channel.

I don’t plan to lower mandatory encryption levels, but you can change your own setup to accept less secure protocols and ciphers, if you need to. Alternatively disable enforced TLS.

andryyy commented 5 years ago

Since this is outgoing mail, you can also create a TLS policy map in mailcow and override the TLS level requirement for that domain.

mxcd commented 11 months ago

I know this is an older issue, but still stumbled over it.
How would a TLS map configuration look like in this case?
I am having issues with gmx.de at the moment and tried to set up the policy in order to overwrite the otherwise enforced incoming and outgoing TLS requirements of the mailbox.
Below config did not work:

image

Any advise @andryyy ?