Outbound Emails SMTP Connect ()

mruch2 commented 7 years ago

Issue

Cannot send outbound emails.

Expected Behavior

The outbound emails should be working.

Actual Behavior

I used to use a web hosting company for my email. I recently switched to gmail and now have SSL/TLS auth. But it is not working.

Steps to Reproduce

Sending any email from anywhere gives an error message. This includes sending out emails from the admin section in setting up outbound emails with trying to test. As well as just saving the settings there assuming they are correct and trying to send an email from the quote section by selecting "Email Quotation"

Version Info

SuiteCRM 7.9.4

Error Message: Wed Aug 23 04:47:31 2017 [5628][5a6f3c04-6cdc-72e2-d455-58ca04892a20][FATAL] SugarPHPMailer encountered an error: The outbound mail server selected for the mail account you are using is invalid. Check the settings or select a different mail server for the mail account. Wed Aug 23 04:47:31 2017 [5628][5a6f3c04-6cdc-72e2-d455-58ca04892a20][FATAL] SugarPHPMailer encountered an error: SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting

stancel commented 7 years ago

Hang tight. I have debugged through the issue and can now reproduce it in my local environment (Mac / MAMP / PHP 7.0.15), but it does not happen on my server (Ubuntu 16.04.3 / Apache / PHP 7.0.22).

I have debugged, see two issues happening and have just overcome them in my local to send a successful email. That involved changing the SMTP password variable in memory though. This was the same issue I mentioned previously, but did not have time to fix before. Just double checking some things, be back with you later today with info to try for a fix, or a pull request.

mruch2 commented 7 years ago

Awesome thanks. Hopefully we can get to the bottom of it. For many peoples sake.

pgorod commented 7 years ago

@stancel there are already a few open issues regarding the bugs with saving email passwords. For that aspect of the problem, make sure you explore those issues too: #4057, #3841, and there may be others.

About this one here, the SMTPConnect fail, if we solve it by adding flags called like this...

        'verify_peer' => false,
        'verify_peer_name' => false,
        'allow_self_signed' => true

... then I think we should investigate exactly these issues, especially the self-signed thing. Does anybody know how the openssl PHP extension handles certificates, where it gets them from? Maybe @mruch2 is running his computer with a self-signed certificate in there?

The phpmailer Troubleshooting Docs have a section on this: https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting#php-56-certificate-verification-failure

and basically their advice is to correct the SSL configuration. That's probably the way forward for @mruch2. But for the SuiteCRM project I'm also trying to make this error more apparent and better reported.

mruch2 commented 7 years ago

@pgorod - wouldnt it use googles certificate as I am sending it through their mailing service? By the way the settings that you have listed there dont exactly work as SuiteCRM does not acknowledge that the email is sent. When sending an email through the quoting module it goes into a forever loading mode even though it sends. It only sends now because I put in that section of code in the phpmailer file.

For whatever reason this does not completely work for SuiteCRM although your script can use it just fine.

Another thing I think you are on to something about the SSL config. Where are the settings for that?

mruch2 commented 7 years ago

@pgorod - Here is something interesting. Changing some settings to isolate the issue and I noticed that only one option in the array actually matters. I modified the code and got the same result. Here is the passed parameter.

public $SMTPOptions = array(
        'ssl' => array(
            'verify_peer' => false,
            //'verify_peer_name' => false//,
            //'allow_self_signed' => false
        )
    );

pgorod commented 7 years ago

There's a certificate on each side, Google identifies itself with one, you identify yourself with another. I really don't know much about certificates, and much less about PHP openssl module certificates. You'll have to Google for it, but at least now you know exactly what to search for: that error message, and that verify_peer option.

I can tell you I never did any certificates configuration to get my emails to send - you might be able to work through this just by playing with your PHP configs.

The other email errors are other email errors. SuiteCRM just underwent a complete rewrite of its Email module, there were many difficulties in this project, it's still a bit wobbly. But my advice is to treat each Issue as a separate Issue: investigate it separately, and report it separately if necessary.

mruch2 commented 7 years ago

Ok I just went to startssl.com and got a free cert. I installed it through the windows installer tool. Now I am going to try and see if I get a different result. Not sure yet.

stancel commented 7 years ago

Hey guys, so here is where things stand at the moment.

When I first reopened my local debugging environment I was able to recreate the issue and saw that the problem was failing in the startTLS() method inside the PHPMailer library. The logic of the method looks fine, but to test further I modified PHPMailer library file class.smtp.php::startTLS() by adding this backstop onto line 363: $crypto_method |= STREAM_CRYPTO_METHOD_ANY_CLIENT The idea behind this was to give the socket the ability to fall back to SSL encryption on the socket - basically still connect using any SSL/TLS method possible (I know this is not the secure long term fix - just getting through the issue at this point). That worked for me but my connection still failed in the PHPMailer library file class.phpmailer.php line 1684: if (!$this->smtp->authenticate That failed because I had taken a backup of my Prod DB from the server and restored it onto my local testing environment. Apparently when the outbound passwords are saved they are hashed or salted using some local variable so the base64 encoded value stored in the DB outbound_email.mail_smtppass did not decode to the correct password thus properly causing it to fail there when the wrong password was sent. Re-saving the correct password into the Outbound Email account UI and clicking save updated the hashed (and probably salted) base64 encoded value in outbound_email.mail_smtppass thus enabling me to connect the next time.

Now on to the nitty gritty. Shortly after the initial debug everything was working for me continuously. I was trying to get it to once again fail there like it has for @mruch2. I was finally able to do so by changing the hostname in my config.php. Mine was still set to our production server's value that has a valid wildcard cert signed by a recognized CA. When changing that value to "localhost/SuiteCRM" the SMTP fails there every time like it does for @mruch2.

@mruch2, what is the value of "hostname" in your config.php file?

mruch2 commented 7 years ago

@stancel - it is localhost.

stancel commented 7 years ago

Do me a favor and temporarily change it to crm.processfast.com and then test again. That is my prod server that has a valid cert setup. Please let me know the results from that test. I believe that Google is trying to test for a valid cert from the calling server and if that fails they send back a failing command to the HELO call being made from SuiteCRM --> PHPMailer --> Gmail/GSuite's server.

mruch2 commented 7 years ago

Just to confirm the setting I am changing is "host_name" correct? If so that does not appear to have worked.

stancel commented 7 years ago

Yes, you need to change "host_name" in config.php to the value I gave you. Now, try one more thing. Please fully shut down your WAMPP instance, clear your browser's cache and close it. Then reopen the browser, start your WAMPP instance back up, login to your local SuiteCRM instance, try the same steps again. However, if the first test fails I ask you to stay on that page and click the "Send Test Email" button again without refreshing the page. Please let me know if it sends then.

stancel commented 7 years ago

FYI - the reason I am asking for you to perform the test this way is to see if the resource ($this->smtp_conn) is being released or not set properly on the first connection with correct parameters, then on the second request that resource is valid and then working. I have seen a few debugging tests in my local that would lead me to believe that this is the case.

@pgorod - Yes, just changing the SMTP options array to pass the parameter of verify_peer = false, would seem to work, but I think there is more going on than that. If you just want to test that then all you or @mruch2 need to do is put the below lines into the first lines of the SugarPHPMailer::smtpConnect().

$options = ['verify_peer' => false]; $connection = parent::smtpConnect($options);

mruch2 commented 7 years ago

@stancel - Ok I tried both mentioned next steps. I traded out my host_name flushed browser cache restarted apache. No change. (I did remove the added code that I previously put in the PHPMailer)

I also attempted to simply add the two lines of code mentioned for the $options, recognizing I actually have to modify a line of code that is in this function as $connection = exists, inside the SugarPHPMailer and removing the code that I put into the PHPMailer file itself. This did not net the same result as having the code inside the PHPMailer file.

Edit: Looking at your code further it did not actually match the parameters that we have been passing so I modified the $options statement a little bit. Now it is the same result.

$options = array(
                'ssl' => array(
                    'verify_peer' => false
                )
            );

Edit 2: Ok I have everything working without any internal errors. I changed my host name to my crm site. Then I un-commented the options code inside the SugarPHPMailer.php that you said to place in there and everything appears to work correctly inside sure now. Still need to figure out why we need to pass that option.

stancel commented 7 years ago

Glad to hear that worked and you are up and running now. I believe you are having to pass that option because client cert validation is being performed by default by either the OpenSSL library or on Google's side from their servers, which is one of the reasons why I was asking you to change around the host name in your config. Passing that would essentially just disable that check. If you want to check this further, I would start by looking at your OpenSSL config file, the chosen options inside of it and any directories it references for self signed certs or CA's (certificate authorities). Per your previous post, you should like into this file:

c:/usr/local/ssl/openssl.cnf

I'll have to bow out for researching this one further as I've got some blocking issues for my own business that I need to solve as well. I'll try to put in a pull request for this issue later today though in case the maintainers decide they would like to merge this in.

If you ( @mruch2 ) or @pgorod would be kind enough to test my pull request #4168 to fix and enhance issue #4111 if you get some free time, I would appreciate it.

pgorod commented 7 years ago

@mruch2 if you try wrapping the smtpConnect in a try block, can you catch that exception?

In include/SugarPHPMailer.php, around line 362

put a try { just at the start, and some block to catch it at the end.

The function would look like this in the end:

    public function smtpConnect($options = array())
    {
    try {
        $connection = parent::smtpConnect();
        if (!$connection) {
            global $app_strings;
            if (isset($this->oe) && $this->oe->type === 'system') {
                $this->setError($app_strings['LBL_EMAIL_INVALID_SYSTEM_OUTBOUND']);
            } else {
                $this->setError($app_strings['LBL_EMAIL_INVALID_PERSONAL_OUTBOUND']);
            } // else
        }
        return $connection;
     } catch (phpmailerException $e) {
          echo '---------------------------------------------------';
          echo $e->errorMessage(); 
          echo '---------------------------------------------------';
     } catch (Exception $e) {
          echo $e->getMessage(); //error messages from anything else
    }
    } // fn

I'd like to know if we can get that detailed message (Exception has occurred. Warning: stream_socket_enable_crypto(): SSL operation failed with code 1. OpenSSL Error messages: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed) from here.

But it seems that for this to work, we'd have to change the statement instantiating PHPMailer to pass true to the constructor. On my script, this looks like this: $mail = new PHPMailer(true);

But in SuiteCRM, there are many new PHPMailer statements, and I'm not sure which one applies for this... :-( I'm afraid I don't have time to investigate right now...

Dillon-Brown commented 7 years ago

@mruch2 @stancel @pgorod Hi guys, brilliant job debugging this. I'll label this as a high priority bug for now and we will discuss the implications of this and stancel's proposed fix as soon as possible. Thanks for all the help!

pgorod commented 7 years ago

Another experience from the forums, with a different workaround...

Hi all I installed version 7.9.6 and had the error: SMTP connect() failed. github.com/PHPMailer/PHPMailer/wiki/Troubleshooting. I tried several workarounds I found in the forum, like changing files in the include-folder, hardcoding my server settings, trying all combinations in the UI (with port 465, 587, 25 and with SSL, TLS, None) etc etc. None of these fixed my error. Now, I finally found a solution, posting it to enable other users to try this fix, if they are fighting with the outgoing email settings: --> Use n email password which is just made up of a-z, A-Z, 0-9. So, DONT use special characters like +=(),. and so on I had a complex password with special chars. Changing this fixed it for me :-)

Hope it helps someone, too

pgorod commented 7 years ago

I made some progress on this today:

I can get the full exception text from smtpConnect and put it in the log: this is really helpful to make the error much more focused and avoids needless mystery. I'll make a PR when I'm finished with all this.
I found yet another very misleading case, which surely is tripping a lot of people: the Send test email from the Outbound Accounts configuration uses the From and FromName from the outbound system account. If that is different from the Account you're actually configuring at the moment, that is enough for many SMTP providers to give you a "not authenticated" reply, which sounds like a bad password problem, but actually isn't. Interestingly, in this case, it's the Test email that doesn't work, but then the actual email or campaign might work, because there you are asked for a From address, and if you put in the correct one, the email will send...

All this I tested on a 7.8.7, I expect the same problems apply to 7.9.x, but I'll have to try it someday.

pgorod commented 7 years ago

I moved this last case I found to a new Issue #4479.

This Issue here has enough entity by itself (with that verify_peer option).

pgorod commented 6 years ago

I suggest closing this one, unless someone has a different opinion.

My logic:

we already have full smtp logging visible from the UI, so this error will be much more apparent to the person configuring email
the real fix is proper SSL configuration
the workaround is insecure (turn off verify_peer), so we shouldn't add it in our code anyway - let people do it manually if they think it's appropriate

Am I right, or am I missing something? Thanks

salesagility / SuiteCRM