search

salesagility / SuiteCRM

SuiteCRM - Open source CRM for the world
https://www.suitecrm.com
GNU Affero General Public License v3.0
4.47k stars 2.09k forks source link

Cannot connect to Dovecot IMAP server with SSL #3709

Open sergio91pt opened 7 years ago

sergio91pt commented 7 years ago

Issue

Cannot save a inbound email account, when using a IMAPS Dovecot server, since folder selection is required but SuiteCRM does not properly connect to the server.

Upon further investigation, it was failling because findOptimumSettings (modules/InboundEmail/InboundEmail.php) was erroneously assuming the credentials were incorrect and stopping the loop early.

The test failed on the service=imap/ssl/validate-cert/secure test, with error [CLOSED] IMAP connection broken (server response), but succeeded on the service=imap/ssl/validate-cert test.

The dovecot server was running with:

Expected Behavior

Able to add dovecot imaps server. Clicking on select folder for Trash and Sent works. Test settings button works.

Actual Behavior

Unable to add inbound email account (personal/group/bounce). Test settings and select folder buttons do not work as expected. In some cases it displayed a generic error, in others (tested in various panels and versions) it showed invalid credentials or similar (credentials were correct).

Possible Fix

Do not use [CLOSED] IMAP connection broken (server response) as invalid credentials, it may lead to false positives.

Steps to Reproduce

  1. Go to profile -> email settings
  2. Add a personal inbound email account using
  3. Try to select the trash folder

Your Environment

Mail Server Environment

pgorod commented 7 years ago

@sergio91pt I was wondering if this could be in any way related to #2807 ? I don't know if this findOptimumSettings function is being called also when connecting to send emails. It seems to be a different part of the email code, but some of the symptoms look similar...

sergio91pt commented 7 years ago

@pgorod I don't think so. When findOptimumSettings runs, the unsuccessful attempts are clearly marked as coming from dovecot and not postfix/smtpd in mail.log.

This loop that failed due to [CLOSED] IMAP connection broken (server response):

---------------STARTING FINDOPTIMUMS LOOP----------------
1: I-E testing string: {redacted.com:993/service=imap/ssl/tls/validate-cert/secure}INBOX
1: I-E failed using [{redacted.com:993/service=imap/ssl/tls/validate-cert/secure}INBOX] - error: Can't open mailbox {redacted.com:993/service=imap/ssl/tls/validate-cert/secure}INBOX: invalid remote specification
1: I-E clearing error and alert stacks.
2: I-E testing string: {redacted.com:993/service=imap/ssl/tls/validate-cert}INBOX
2: I-E failed using [{redacted.com:993/service=imap/ssl/tls/validate-cert}INBOX] - error: Can't open mailbox {redacted.com:993/service=imap/ssl/tls/validate-cert}INBOX: invalid remote specification
2: I-E clearing error and alert stacks.
3: I-E testing string: {redacted.com:993/service=imap/ssl/validate-cert/secure}INBOX
3: I-E failed using [{redacted.com:993/service=imap/ssl/validate-cert/secure}INBOX]
3: I-E ERROR: $ie->findOptimums() failed due to bad user credentials for user login: redacted@redacted.com
Hook called: InboundEmail::after_ui_frame
Creating new instance of hook class hooks without parameters
Creating new instance of hook class AssignGroups without parameters
Hook called: ::after_ui_footer
Creating new instance of hook class AssignGroups without parameters
Hook called: ::server_round_trip
Calling MySQLi::disconnect()

Appears in the mail server as:

dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=redacted_server_ip, lip=redacted_suitecrm_ip, TLS, session=<3uBvZwNR2gA0MKR+>
dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=redacted_server_ip, lip=redacted_suitecrm_ip, TLS, session=<tatwZwNR3AA0MKR+>
dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=redacted_server_ip, lip=redacted_suitecrm_ip, TLS, session=<UndxZwNR3gA0MKR+>
pgorod commented 7 years ago

Ok, thanks. To be frank I don't really understand these configurations and protocols so I was just hoping there might be a connection, so we could make some progress on that other issue, which is creating problems for a lot of people and still doesn't have a full diagnosis. Thanks!

chris001 commented 7 years ago

@sergio91pt

sergio91pt commented 7 years ago

@chris001 This should answer your questions.

# Executed on the crm server.
# Reverse ip lookup points to the same domain presented by the server
$ openssl s_client -CApath /etc/ssl/certs/ -connect redacted.com:465
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = redacted.com
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=redacted.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL/CN=redacted.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4825 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: DBCD15DC4EF4E07060177ECD89FCE5E4006721ADDC7E674C77103E543EDD77FD
    Session-ID-ctx: 
    Master-Key: C755EC455A35797605920CB3B443658CB1EFCC63F0784E82C9189542EDB3D26B18531E892010660B58344587BF3B825E
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1497445091
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
220 redacted.com ESMTP Postfix (Ubuntu)
EHLO crmserver
250-redacted.com
250-PIPELINING
250-SIZE 100000000
250-ETRN
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
QUIT
DONE

findOptimumSettings runs this tests in order:

        $ssl = array(
            'ssl-both-on-secure' => '/ssl/tls/validate-cert/secure',
            'ssl-both-on' => '/ssl/tls/validate-cert',
            'ssl-cert-secure' => '/ssl/validate-cert/secure',
            'ssl-cert' => '/ssl/validate-cert',
            'ssl-tls-secure' => '/ssl/tls/secure',
            'ssl-tls' => '/ssl/tls',
            'ssl-both-off-secure' => '/ssl/notls/novalidate-cert/secure',
            'ssl-both-off' => '/ssl/notls/novalidate-cert',
            'ssl-nocert-secure' => '/ssl/novalidate-cert/secure',
            'ssl-nocert' => '/ssl/novalidate-cert',
            'ssl-notls-secure' => '/ssl/notls/secure',
            'ssl-notls' => '/ssl/notls',
            'ssl-secure' => '/ssl/secure',
            'ssl-none' => '/ssl',
        );

While I don't know the difference between /ssl/validate-cert/secure and /ssl/validate-cert, the fact that it passes /ssl/validate-cert means SuiteCRM/PHPMailer has no problem validating the certificate.

sergio91pt commented 7 years ago

@chris001 Sorry, only realized I sent you the SMTP connection test instead of IMAP.

I don't think dovecot has a server name configured (thats why I without thinking did a SMTP test). The certificate is for a single domain, not wildcard. It also has the www subdomain, in the alt names.

$ openssl s_client -CApath /etc/ssl/certs/ -connect redacted.com:993
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = redacted.com
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=redacted.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL/CN=redacted.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 5004 bytes and written 463 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: E14C60CAB29281BDB075E0A6CBAE7F7920575AE2F1DB14AE1A43C47B70FC8362
    Session-ID-ctx: 
    Master-Key: 1902F00DCF10473C1D22DE048D33E3064A6665032FB6E449BDD8274BDCD8A3F26017DEF1688F027BCFC1DFE65049507A
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - de 43 8b 7f 73 b0 af 27-12 3f e3 05 24 2b e3 2e   .C..s..'.?..$+..
    0010 - fd 38 fe f2 05 c7 50 81-b0 1f b7 00 e6 45 58 36   .8....P......EX6
    0020 - df 1e f5 f1 33 a9 a3 bc-f3 31 08 88 14 6d f1 89   ....3....1...m..
    0030 - 45 02 9c 3f 8d f6 a3 11-a3 2b 87 f2 20 71 8e 84   E..?.....+.. q..
    0040 - de 23 57 fd 19 92 8a 03-c4 52 66 3a 99 f0 61 31   .#W......Rf:..a1
    0050 - b0 7b f4 3e 9a 54 f1 8d-52 86 ec 90 07 af b8 56   .{.>.T..R......V
    0060 - dc ee 49 c5 d2 21 11 b0-44 08 e1 01 d8 34 30 de   ..I..!..D....40.
    0070 - f0 e5 76 68 bd 89 6f 11-fa 4e 10 4b 40 a9 b0 e4   ..vh..o..N.K@...
    0080 - 9b 69 f4 15 7c 5d a8 db-dc 6d ac b3 e6 6d 7e 7a   .i..|]...m...m~z
    0090 - 69 9f 02 92 64 46 40 b0-4c 34 ec c7 04 48 22 ec   i...dF@.L4...H".

    Start Time: 1497445758
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot (Ubuntu) ready.
closed
chris001 commented 7 years ago

@salesagility Another perfect example of poor/nonexistent/obscured error reporting by this app! @sergio91pt Change line 2864 of InboundEmail.php: https://github.com/salesagility/SuiteCRM/blob/master/modules/InboundEmail/InboundEmail.php#L2864

$GLOBALS['log']->debug($l . ': I-E failed using [' . $serviceTest . ']');

to add this one line after it, as shown:

$GLOBALS['log']->debug($l . ': I-E failed using [' . $serviceTest . ']');
$GLOBALS['log']->debug($l . ': I-E exact error message: [' . $errors . ']');

and run your test again.

sergio91pt commented 7 years ago

@chris001 I should have mentioned that I used a debugger to inspect the $errors variable, so that's how I knew which error was the culprit.

I reverted my fix (PR #3710) and added that log (still on hotfix - so not the same line):

Module:InboundEmail using file: ShowInboundFoldersList
---------------STARTING FINDOPTIMUMS LOOP----------------
1: I-E testing string: {redacted.com:993/service=imap/ssl/tls/validate-cert/secure}INBOX
1: I-E failed using [{redacted.com:993/service=imap/ssl/tls/validate-cert/secure}INBOX] - error: Can't open mailbox {redacted.com:993/service=imap/ssl/tls/validate-cert/secure}INBOX: invalid remote specification
1: I-E clearing error and alert stacks.
2: I-E testing string: {redacted.com:993/service=imap/ssl/tls/validate-cert}INBOX
2: I-E failed using [{redacted.com:993/service=imap/ssl/tls/validate-cert}INBOX] - error: Can't open mailbox {redacted.com:993/service=imap/ssl/tls/validate-cert}INBOX: invalid remote specification
2: I-E clearing error and alert stacks.
3: I-E testing string: {redacted.com:993/service=imap/ssl/validate-cert/secure}INBOX
3: I-E failed using [{redacted.com:993/service=imap/ssl/validate-cert/secure}INBOX]
3: I-E exact error message: [[CLOSED] IMAP connection broken (server response)]
3: I-E ERROR: $ie->findOptimums() failed due to bad user credentials for user login: redacted@redacted.com
Hook called: InboundEmail::after_ui_frame
Creating new instance of hook class hooks without parameters
Creating new instance of hook class AssignGroups without parameters
Hook called: ::after_ui_footer
Creating new instance of hook class AssignGroups without parameters
Hook called: ::server_round_trip
Calling MySQLi::disconnect()

On the mail server: (lip is actually the interface internal ip - no point in redacting that)

dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=redacted, lip=172.31.12.174, TLS, session=<DY/FnO1RNwC8+5I3>
dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=redacted, lip=172.31.12.174, TLS, session=<nWbNnO1RUQC8+5I3>
dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=redacted, lip=172.31.12.174, TLS, session=<jOLUnO1RlwC8+5I3>
chris001 commented 7 years ago

@sergio91pt Can you try port 143 instead of 993.

sergio91pt commented 7 years ago

@chris001 Port 143 is STARTTLS and also fails on this method. Looking at the imap-open documentation, it is clear that the $ssl tests will never succeed with a STARTTLS connection upgrade: no server offers STARTTLS if the connection is already protected by SSL/TLS and /ssl/notls is redundant and the same as /ssl, same thing for /notls/novalidate-cert and /notls in the $nonSsl array.

I can see various issues now:

  1. Impossible and redundant params in $nonSsl, $ssl make extra unnecessary connection tests
  2. HCI issue: When you select SSL in the UI, it changes the port to 993, which implies that non SSL is STARTTLS, but it favors plaintext connections and non validated certificates
  3. nonSSL should try opportunistic STARTTLS first, preferring validated certificates
  4. Decide weather or not STARTTLS connection upgrades belong in the $ssl tests
  5. Basically this issue. The use of /secure (disallow LOGIN and PLAIN auth mechanisms) in SSL/TLS protected connections (upgraded or not) results in [CLOSED] IMAP connection broken (server response) which is mistaken as invalid credentials.

Maybe we should split some of them into seperate issues?

Tests for reference

$nonSsl = array(
    'both-secure' => '/notls/novalidate-cert/secure',
    'both' => '/notls/novalidate-cert',
    'nocert-secure' => '/novalidate-cert/secure',
    'nocert' => '/novalidate-cert',
    'notls-secure' => '/notls/secure',
    'secure' => '/secure', // for POP3 servers that force CRAM-MD5
    'notls' => '/notls',
    'none' => '', // try default nothing
);
$ssl = array(
    'ssl-both-on-secure' => '/ssl/tls/validate-cert/secure',
    'ssl-both-on' => '/ssl/tls/validate-cert',
    'ssl-cert-secure' => '/ssl/validate-cert/secure',
    'ssl-cert' => '/ssl/validate-cert',
    'ssl-tls-secure' => '/ssl/tls/secure',
    'ssl-tls' => '/ssl/tls',
    'ssl-both-off-secure' => '/ssl/notls/novalidate-cert/secure',
    'ssl-both-off' => '/ssl/notls/novalidate-cert',
    'ssl-nocert-secure' => '/ssl/novalidate-cert/secure',
    'ssl-nocert' => '/ssl/novalidate-cert',
    'ssl-notls-secure' => '/ssl/notls/secure',
    'ssl-notls' => '/ssl/notls',
    'ssl-secure' => '/ssl/secure',
    'ssl-none' => '/ssl',
);

Port 143 (STARTTLS only), SSL not selected

---------------STARTING FINDOPTIMUMS LOOP----------------
1: I-E testing string: {redacted.com:143/service=imap/notls/novalidate-cert/secure}INBOX
1: I-E failed using [{redacted.com:143/service=imap/notls/novalidate-cert/secure}INBOX] - error: Can't do secure authentication with this server
1: I-E clearing error and alert stacks.
2: I-E testing string: {redacted.com:143/service=imap/notls/novalidate-cert}INBOX
2: I-E failed using [{redacted.com:143/service=imap/notls/novalidate-cert}INBOX] - error: Server disables LOGIN, no recognized SASL authenticator
2: I-E clearing error and alert stacks.
3: I-E testing string: {redacted.com:143/service=imap/novalidate-cert/secure}INBOX
3: I-E failed using [{redacted.com:143/service=imap/novalidate-cert/secure}INBOX]
3: I-E exact error message: [[CLOSED] IMAP connection broken (server response)]
3: I-E ERROR: $ie->findOptimums() failed due to bad user credentials for user login: redacted@redacted.com
Hook called: InboundEmail::after_ui_frame
Creating new instance of hook class hooks without parameters
Creating new instance of hook class AssignGroups without parameters
Hook called: ::after_ui_footer
Creating new instance of hook class AssignGroups without parameters
Hook called: ::server_round_trip
Calling MySQLi::disconnect()

Port 143 (STARTTLS only), SSL selected on UI

---------------STARTING FINDOPTIMUMS LOOP----------------
1: I-E testing string: {redacted.com:143/service=imap/ssl/tls/validate-cert/secure}INBOX
1: I-E failed using [{redacted.com:143/service=imap/ssl/tls/validate-cert/secure}INBOX] - error: Can't open mailbox {redacted.com:143/service=imap/ssl/tls/validate-cert/secure}INBOX: invalid remote specification
1: I-E clearing error and alert stacks.
2: I-E testing string: {redacted.com:143/service=imap/ssl/tls/validate-cert}INBOX
2: I-E failed using [{redacted.com:143/service=imap/ssl/tls/validate-cert}INBOX] - error: Can't open mailbox {redacted.com:143/service=imap/ssl/tls/validate-cert}INBOX: invalid remote specification
2: I-E clearing error and alert stacks.
3: I-E testing string: {redacted.com:143/service=imap/ssl/validate-cert/secure}INBOX
3: I-E failed using [{redacted.com:143/service=imap/ssl/validate-cert/secure}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
3: I-E clearing error and alert stacks.
4: I-E testing string: {redacted.com:143/service=imap/ssl/validate-cert}INBOX
4: I-E failed using [{redacted.com:143/service=imap/ssl/validate-cert}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
4: I-E clearing error and alert stacks.
5: I-E testing string: {redacted.com:143/service=imap/ssl/tls/secure}INBOX
5: I-E failed using [{redacted.com:143/service=imap/ssl/tls/secure}INBOX] - error: Can't open mailbox {redacted.com:143/service=imap/ssl/tls/secure}INBOX: invalid remote specification
5: I-E clearing error and alert stacks.
6: I-E testing string: {redacted.com:143/service=imap/ssl/tls}INBOX
6: I-E failed using [{redacted.com:143/service=imap/ssl/tls}INBOX] - error: Can't open mailbox {redacted.com:143/service=imap/ssl/tls}INBOX: invalid remote specification
6: I-E clearing error and alert stacks.
7: I-E testing string: {redacted.com:143/service=imap/ssl/notls/novalidate-cert/secure}INBOX
7: I-E failed using [{redacted.com:143/service=imap/ssl/notls/novalidate-cert/secure}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
7: I-E clearing error and alert stacks.
8: I-E testing string: {redacted.com:143/service=imap/ssl/notls/novalidate-cert}INBOX
8: I-E failed using [{redacted.com:143/service=imap/ssl/notls/novalidate-cert}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
8: I-E clearing error and alert stacks.
9: I-E testing string: {redacted.com:143/service=imap/ssl/novalidate-cert/secure}INBOX
9: I-E failed using [{redacted.com:143/service=imap/ssl/novalidate-cert/secure}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
9: I-E clearing error and alert stacks.
10: I-E testing string: {redacted.com:143/service=imap/ssl/novalidate-cert}INBOX
10: I-E failed using [{redacted.com:143/service=imap/ssl/novalidate-cert}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
10: I-E clearing error and alert stacks.
11: I-E testing string: {redacted.com:143/service=imap/ssl/notls/secure}INBOX
11: I-E failed using [{redacted.com:143/service=imap/ssl/notls/secure}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
11: I-E clearing error and alert stacks.
12: I-E testing string: {redacted.com:143/service=imap/ssl/notls}INBOX
12: I-E failed using [{redacted.com:143/service=imap/ssl/notls}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
12: I-E clearing error and alert stacks.
13: I-E testing string: {redacted.com:143/service=imap/ssl/secure}INBOX
13: I-E failed using [{redacted.com:143/service=imap/ssl/secure}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
13: I-E clearing error and alert stacks.
14: I-E testing string: {redacted.com:143/service=imap/ssl}INBOX
14: I-E failed using [{redacted.com:143/service=imap/ssl}INBOX] - error: TLS/SSL failure for redacted.com: SSL negotiation failed
14: I-E clearing error and alert stacks.
---------------end FINDOPTIMUMS LOOP----------------
Hook called: InboundEmail::after_ui_frame
Creating new instance of hook class hooks without parameters
Creating new instance of hook class AssignGroups without parameters
Hook called: ::after_ui_footer
Creating new instance of hook class AssignGroups without parameters
Hook called: ::server_round_trip
Calling MySQLi::disconnect()

Port 143 connection tests

Telnet connection

$ telnet redacted.com 143
Trying XXX.XXX.XXX.XXX...
Connected to redacted.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot (Ubuntu) ready.
A01 LOGOUT
* BYE Logging out
A01 OK Logout completed.
Connection closed by foreign host.

STARTTLS connection upgrade

# Note: credentials have also been redacted
# AHJlZGFjdGVkQHJlZGFjdGVkLmNvbQByZWRhY3RlZA== is base64_encode("\0redacted@redacted.com\0redacted")

$ openssl s_client -CApath /etc/ssl/certs/ -starttls imap -connect redacted.com:143
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = redacted.com
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=redacted.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL/CN=redacted.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 5328 bytes and written 489 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: E93B5845B525928D74DB9B9820F84A03A8303201B311AD845577548CD9538478
    Session-ID-ctx: 
    Master-Key: DE48207CCD7B3B5ED6F94156CCA90C21F01AC7B2B5F4D3824C2DC046CCF66C9DA11C953BAFE968311B79E828366ABA03
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - aa d9 56 48 6b 6f 25 74-6b 90 9d ef 3f b9 0e 76   ..VHko%tk...?..v
    0010 - 88 28 ec bc 7c 18 6b 8b-b8 7a 06 e3 db 79 ef 00   .(..|.k..z...y..
    0020 - c4 aa 3c 6c 58 72 6d 59-1e 4b 00 4e 7e 18 c2 12   ..<lXrmY.K.N~...
    0030 - d4 2b e4 24 48 3c b4 01-c3 04 26 c1 0c 6e dc 85   .+.$H<....&..n..
    0040 - c9 0f d1 dd 4a 30 91 6b-de 9a c3 78 06 46 01 21   ....J0.k...x.F.!
    0050 - 2c dc 4e 0d 98 27 54 2e-f7 c2 26 af 70 7d 46 ad   ,.N..'T...&.p}F.
    0060 - 69 57 49 5f 19 81 c2 c8-4b fe bb bb 56 1c a1 0d   iWI_....K...V...
    0070 - 44 49 6d 4f ef b3 8c 68-84 65 eb 9e 65 87 0b 59   DImO...h.e..e..Y
    0080 - 99 cf 2b 3f 0f b3 32 4e-92 7e aa f1 17 4a 9b e3   ..+?..2N.~...J..
    0090 - bb 97 80 1d 64 20 6b af-61 00 09 ce 88 bc ce 1f   ....d k.a.......

    Start Time: 1497527649
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
. OK Pre-login capabilities listed, post-login capabilities have more.
A01 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN
A01 OK Pre-login capabilities listed, post-login capabilities have more.
A02 AUTHENTICATE PLAIN AHJlZGFjdGVkQHJlZGFjdGVkLmNvbQByZWRhY3RlZA==
* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE QUOTA
A02 OK Logged in
A03 LOGOUT
* BYE Logging out
A03 OK Logout completed.
closed
sergio91pt commented 7 years ago

I created a small script to test imap_open "specs". The big difference is that I'm using imap_errors() instead of imap_last_error(), to get all the errors.

Tested it with my PC's php-cli (v5.5.9 - Ubuntu 14.04). Both Ubuntu 14.04 (php 5.5) and 16.04 (php 7.0) use the same IMAP c-Client Version (2007f).

When I intentionally provide the wrong password I get:

ERR: Can not authenticate to IMAP server: [AUTHENTICATIONFAILED] Authentication failed.
ERR: [CLOSED] IMAP connection broken (server response)

Since the server only provides PLAIN and LOGIN authentication, when I use /tls/secure with port 143 or /ssl/secure with port 993, I get:

ERR: Can't do secure authentication with this server
ERR: [CLOSED] IMAP connection broken (server response)

If I do something stupid like use /blah or /ssl/tls as the spec, I get the invalid remote specification and no connection seems to be attempted.

The changelog of the imap c-client library answers some questions:

imap-2006 is a major release. Programs written for imap-2004g should build with this version with minor or no modification. imap-2005 was not released except as development snapshots.

(...)

SSL/TLS certificate validation on UNIX now checks the alternative names in the certificate if the CN does not match.

(...)

The silly mailbox flag combination /ssl/tls is now rejected as an invalid remote specification. Previous versions tried to negotiate TLS over an SSL session; even if the server permitted such a thing it couldn't work.

IMAP connection test script

<?php

if (PHP_SAPI !== 'cli') {
    die('cli only');
}
if ($argc < 5 || $argc > 6) {
    die("USAGE: php $argv[0] HOST PORT USERNAME PASSWORD [SPEC]\n");
}
$host = $argv[1];
$port = intval($argv[2]);
$username = $argv[3];
$password = $argv[4];
$spec = isset($argv[5]) ? $argv[5] : (($port === 993) ? '/ssl' : '/tls');
$params = array('DISABLE_AUTHENTICATOR' => '');
$mailbox = "{{$host}:{$port}{$spec}}INBOX";
$options = 0;

function printArray($array, $prefix)
{
    if (is_array($array)) {
        foreach ($array as $entry) {
            print "$prefix: $entry\n";
        }
    }
    else {
        print "$prefix: None\n";
    }
}

$connection = imap_open($mailbox, $username, $password, $options, 0, $params);

printArray(imap_errors(), "ERR");
printArray(imap_alerts(), "Alert");

if ($connection !== FALSE) {
    imap_close($connection);
}
samus-aran commented 7 years ago

@sergio91pt - Sorry just to confirm where did you guys get to with debugging this?

chris001 commented 7 years ago
sergio91pt commented 7 years ago

@samus-aran We've identified 5 issues related to the findOptimumSettings method:

Issue 1 (this issue)

Unable to connect to IMAP servers that only allow PLAIN and/or LOGIN authentication (and/or unsupported authentication methods like XOAUTH - probably). Erroneously fails with a bad credentials error. On Dovecot, only affects SSL and STARTTLS connections.

Proposed solution:

  1. IMAP connection broken doesn't mean there is an authentication failure
  2. Use imap_errors() instead of imap_last_error()

Issue 2

Security issue: For Non SSL servers, plaintext connections are preferred over opportunistic STARTTLS

Proposed solution:

  1. Change order of tests
  2. Somehow invalidate the connection string of possible affected accounts and force a re-test on the next connection. This is not trivial. The "string" is saved in $_SESSION and in the service column of the inbound_email table (using different syntax).

Issue 3

findOptimumSettings has some impossible and redundant tests.

Proposed solution:

  1. Remove impossible and redundant tests

Issue 4

Each failed test attempts 3 connections in getImapConnection (disables authenticators for compatibility, I guess).

Proposed solution:

  1. Save somewhere if the connection requires an authenticator to be disabled. On the database, we could use the stored options column.

Issue 5

findOptimumSettings and getImapConnection should skip tests/connections depending on the errors of previous attempts.

For example, attempting a SSL connection test to a non-SSL server results in 30 TCP connections, it only needs 1 or 2 to return "invalid" (depending on the error message when connection to a server that presents a invalid certificate).

This issue may trigger the dovecot filter of fail2ban prior to v0.9.0. Doesn't trigger on Ubuntu 14.04, because the sessions identifiers broke the fail regex that shipped with the distro package (fixed before 0.9.0 - it seems).

pgorod commented 7 years ago

@sergio91pt here's a first-place medal for your diagnostic work 🥇

And... you know we're gonna need your help fixing this, don't you? :-)

chris001 commented 7 years ago

@sergio91pt

  1. To fix this bug once and for all. First, improve the unit test for the findOptimumSettings method. https://github.com/salesagility/SuiteCRM/blob/master/tests/tests/modules/InboundEmail/InboundEmailTest.php#L1180 How to improve this unit test method - Have it connect, one by one, to REAL email accounts created for testing purposes: Gmail, Exchange, Yahoo, and the top 10 Linux mail servers.
  2. Then, adjust parameters in findOptimumSettings, until all mail server types connect without error. https://github.com/salesagility/SuiteCRM/blob/master/modules/InboundEmail/InboundEmail.php#L2783 How to know all mail server types connect without error - Run this command: ./vendor/bin/codecept run tests/unit/modules/InboundEmail | grep optimum and it no longer shows F InboundEmailTest: Testfind Optimum Settings.
pgorod commented 7 years ago

I like the idea of getting real email accounts, even though it sounds like a lot of work... but getting even just one email account to be actionable during our tests would greatly decrease risks when touching email code. I'm thinking of all the bugs of the past two weeks here...

And getting a nice array of accounts like @chris001 suggests would be perfect for this particular Issue, and add a lot of extra security to the rest of our email challenges.

For Gmail, Outlook, etc. , it's just a matter of opening the accounts, I guess. But for the "top 10 Linux mail servers" I don't know where we could go. We'd have to know of free public services that use specific types of mail software...

sergio91pt commented 7 years ago

@pgorod Thanks. I do intend to help although time might be a problem. I still don't have my own instance in production and I should already be working on a different project

@chris001 I haven't run the tests yet, only looked around. Does it already use a live account? I see multiple tests using $inboundEmail->id = 1, but I'm not sure if those tests are assuming a functioning mailbox and if said mailbox is setup by the test runner or manually.

chris001 commented 7 years ago

@sergio91pt The test isn't using a live account right now, it never has! Someone should sign up for a free email account on gmail, yahoo, microsoft live. Once you have the mail account credentials, I'll show you how to safely add them to the tests.

dspaan commented 7 years ago

It's amazing to me that the inbound e-mail setup in SuiteCRM has a link to prefill the default setup information for a Gmail account but it's not working? I spent hours on this. I also lost time on setting up an inbound e-mail account with Koozali SME server (which uses Dovecot). And this while i don't even need inbound e-mail but i'm forced to configure one otherwise i can't send outbound e-mails (issue #3741). I finally got my setup working by configuring an IMAP account from a different provider. This should be high priority if you ask me.

dspaan commented 6 years ago

Is this isssue on a roadmap to fixed? I wanted to created an e-mail campaign but it seems i can't because the related domain email is on a server that SuiteCRM does not support.

rajohn96 commented 6 years ago

ALCON,

Any updates on this issue; it is confounding me too. I'll settle for a workaround/one-off, and will commit to working the real fix in exchange.

Thanks!

Alan

rajohn96 commented 6 years ago

BTW, I can connect just fine with a gmail server, its only the Dovecot that's not working (which unfortunately is the one I need to talk to!!)

rajohn96 commented 6 years ago

so I have had a small amount of success, which may be worth something. For my needs I've removed all of the ssl connection options (/ssl/verify-cert, /ssl/noverify-cert, etc) save the one I want, and I've adjusted the imap_open retry count from 0 to 1 and it seems to work for me. What still doesn't work is the selection of folders (which is really not a bit deal to me, but definitely needs to be addressed in the "real" PR). Anyone, does this help at all?

rajohn96 commented 6 years ago

oh, and i updated to the 7.10.5 version; that fixed a different permissions problem I was having elsewhere.

lintech-admin commented 6 years ago

Good day to all!

There is Postfix + Dovecot mail server. Authorization by IMAP for login and password. Email clients connect and work. When I configure the connection in SuiteCRM I get the error "Invalid login or password". The SMTP connection is configured and running. In the server logs such messages:

Jun 26 15:42:29 server dovecot: imap-login: Aborted login (client did not finish SASL auth, waited 0 secs): user = <>, method = GSSAPI, rip = YYYY, lip = XXXX, session = < 7 / 7K1Ipv / ADAqAAz> Jun 26 15:42:29 server dovecot: imap-login: aborted login (no auth attempts in 0 secs): user = <>, rip = Y.Y.Y.Y, lip = X.X.X.X, session = <90bL1Ipv / gDAqAAz> Jun 26 15:42:29 server dovecot: imap-login: aborted login (no auth attempts in 0 secs): user = <>, rip = Y.Y.Y.Y, lip = X.X.X.X, session =

Tell me, please, how to change the authentication method in SuiteCRM !!!

Jorilx commented 6 years ago

It looks like the problem with Dovecot happens when SuiteCRM tries to login using an encrypted password ("/secure") to a server that supports only PLAIN passwords, maybe it would be easier to add a "encrypt password" checkbox and only try with "/secure" if requested by the user?