SMTP Email Server Configuration Fails with SSL issue

MaximUltimatum commented 7 months ago

Describe your Issue

To be clear, I'm not sure if SealedSecrets are related to the issue, I'm simply using them so I've called it out. My issue: I'm unable to send emails from my nextcloud kubernetes server. I'm using a mailgun domain to send mail. I can send test emails from SMTP on my local developer laptop. I have exec-ed into the nextcloud container itself on k8s via kubectl and verified the necessary secrets are there:

root@nextcloud-fun-name-fun-numberspfp:/var/www/html# env | grep SMTP
SMTP_AUTHTYPE=LOGIN
SMTP_NAME=postmaster@my-fun-domain.com
SMTP_PORT=587
SMTP_SECURE=ssl
SMTP_PASSWORD=a-clever-password
SMTP_HOST=smtp.mailgun.com

I can even reference these secrets and send test emails from the container from the command line.

root@nextcloud-fun-name-fun-numbers-95pfp:/var/www/html/config# ./mailtest.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  149k  100  149k    0     0   742k      0 --:--:-- --:--:-- --:--:--  746k
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
perl is already the newest version (5.36.0-7).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
=== Trying smtp.mailgun.org:587...
=== Connected to smtp.mailgun.org.
<-  220 Mailgun Influx ready
 -> EHLO nextcloud-fun-name-cd995c78d-95pfp
<-  250-1e90be7af010
<-  250-AUTH PLAIN LOGIN
<-  250-SIZE 52428800
<-  250-8BITMIME
<-  250-SMTPUTF8
<-  250-PIPELINING
<-  250 STARTTLS
 -> AUTH LOGIN
<-  334 VXNlcm5hbWU6
 -> cG9zdG1hc3RlckBtZy5tYXhpbXN0cm91ZC5kZQ==
<-  334 UGFzc3dvcmQ6
 -> dW5hd2FyZS1kcmlwcGluZy1jYXB0aXZl
<-  235 Authentication successful
 -> MAIL FROM:<root@nextcloud-fun-name-cd995c78d-95pfp>
<-  250 Sender address accepted
 -> RCPT TO:<a-fun-email-to-receive-things-to@gmail.com>
<-  250 Recipient address accepted
 -> DATA
<-  354 Continue
 -> Date: Fri, 17 Nov 2023 15:00:12 +0000
 -> To: a-fun-email-to-receive-things-to@gmail.com
 -> From: root@nextcloud-fun-name-cd995c78d-95pfp
 -> Subject: You need
 -> X-Mailer: swaks v20130209.0 jetmore.org/john/code/swaks/
 -> 
 -> a lot of coffee this morning
 -> 
 -> .
<-  250 Great success
 -> QUIT
<-  221 See you later. Yours truly, Mailgun
=== Connection closed with remote host.

And yet, when I click the "Test and verify email settings" button in the nextcloud basic admin page, I get

A problem occurred while sending the email. Please revise your settings. (Error: Email could not be sent. Check your mail server log)

Logs and Errors

To my frustration, I can't even find the mailserver log. I've tried running kubectl logs <nextcloud-container-name> after attempting to send a test email several times. When that failed, I exec-ed into the container directly and dug around for it. I installed locate, updated the database (so locate could index the files, not the nextcloud database) and looked for every .log file without luck. I checked every log file in /var/log without fruition.

Describe your Environment

Kubernetes distribution: I'm on k8s.

I'm using SealedSecrets, so you can see my entire setup here

Helm Version (or App that manages helm):

helm version
version.BuildInfo{Version:"v3.13.2", GitCommit:"2a2fb3b98829f1e0be6fb18af2f6599e0f4e8243", GitTreeState:"clean", GoVersion:"go1.20.10"}

Helm Chart Version: 4.5.0
values.yaml: values file is here

If it's important to paste literal values here, let me know and I'll copy them over from the link.

# paste your values.yaml (anonymize any sensitive data)

Additional context, if any

I can't think of any relevant context I haven't already provided. If I'm missing something please let me know!

jessebot commented 7 months ago

Sorry to hear it's been frustrating. I totally get how time consuming and frustrating managing these kinds of environments can be. Thanks for submitting the issue anyway though, because it help others in the community as well :pray:

I keep meaning to put a section about logging from Nextcloud itself into the README (ref: https://github.com/nextcloud/helm/issues/443#issue-1881884427), but can you try adding this to your values:

nextcloud:
  configs:
    # if needed, you can change the log level from 1 for info info to 0 for debug.
    logging.config.php: |-
      <?php
      $CONFIG = array (
        'log_type' => 'file',
        'logfile' => 'nextcloud.log',
        'loglevel' => 1,
        'logdateformat' => 'F d, Y H:i:s'
        );

Then on the pod, you can tail out nextcloud.log (which is generally in /var/www/html/ in my experience, I think). From the command line, it would be would be something like:

kubectl exec -n $nextcloud_namespace $nextcloud_pod -- tail nextcloud.log

Let me know what's in the logs (after anonymizing anything important :pray: ), as your values.yaml looks ok to me :thinking:

Also, I don't think this has anything to do with SealedSecrets, because the secrets would be loaded in as env vars from your existing secret here:

And then used in your smtp.config.php file (probably in something like /var/www/html/config, I think) which is rendered from the helm chart template here: https://github.com/nextcloud/helm/blob/9ba93c739a6ba0a51d1780279057af0d638c76b2/charts/nextcloud/templates/config.yaml#L109-L124

Since you're able to see the env vars on the pod, it's likely not to do with SealedSecrets.

jessebot commented 7 months ago

Another thought: Does your mail server (mailgun in this instance) have any logs you can check?

MaximUltimatum commented 7 months ago

Thank you for getting back to me so quickly!

I keep meaning to put a section about logging from Nextcloud itself into the README (ref: https://github.com/nextcloud/helm/issues/443#issue-1881884427), but can you try adding this to your values:

I added that into my config section, and I see when I exec into the container that the nextcloud.log file is created, at least now (yay!) but I don't see anything being written to it (as in, the file is completely empty)

jessebot commented 7 months ago

Thank you for getting back to me so quickly!

I try my best, but can't always respond quickly. Thank you though :)

I don't see anything being written to it (as in, the file is completely empty)

:thinking: hmmmm, it works for me, but is this perhaps an issue of the config only getting loaded at install time, @provokateurin ?

provokateurin commented 7 months ago

For me nextcloud.log is correctly written to

jessebot commented 7 months ago

hmmmm, weird.

@MaximUltimatum can you try:

setting the logging level to here to 0, for DEBUG?

Kubernetes distribution: I'm on k8s.

Also, which kubernetes distro are you using? K3s, EKS, GKE, RKE?
Are you deploying with just helm, or using something to wrap helm?
Are you able to try this with a fresh installation, just for testing purposes to see if it logs properly when it's a fresh install?

MaximUltimatum commented 7 months ago

1. Setting log level

1. setting the logging level to here
This was the issue - I have logs now (yay!). Needed to set it to 0 and not 2

if needed, you can change the log level from 1 for info info to 2 for debug.

(This was what confused me)

2. K8s Distro

2. Also, which kubernetes distro are you using? K3s, EKS, GKE, RKE?
I'm deploying on bare metal/my own servers. I'm using k8s, which I've deployed via kubespray I think the distro is vanilla kubernetes, bootstrapped with kubeadm Hopefully that answered your question.

3. Helm

3. Are you deploying with just helm, or using something to wrap helm?
I'm just using helm at the moment.

4 Yay Logs!

Yay..... SSL errors. My favorite!

{
  "reqId": "FtoKtoW7iGQymAd1YIMQ",
  "level": 0,
  "time": "November 19, 2023 17:16:37",
  "remoteAddr": "10.233.64.0",
  "user": "a-fun-username",
  "app": "core",
  "method": "POST",
  "url": "/index.phpadmin/mailtest",
  "message": "Sending mail to \"Array\n(\n    [a-fun-email@gmail.com] => AFunUsername\n)\n\" with subject \"Email setting test\" failed",
  "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0",
  "version": "27.1.3.2",
  "exception": {
    "Exception": "Symfony\\Component\\Mailer\\Exception\\TransportException",
    "Message": "Connection could not be established with host \"ssl://smtp.mailgun.com:587\": stream_socket_client(): SSL operation failed with code 1. OpenSSL Error messages:\nerror:0A00010B:SSL routines::wrong version number",
    "Code": 0,
    "Trace": [
      {
        "function": "Symfony\\Component\\Mailer\\Transport\\Smtp\\Stream\\{closure}",
        "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\Stream\\SocketStream",
        "type": "->",
        "args": [
          "*** sensitive parameters replaced ***"
        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/Stream/SocketStream.php",
        "line": 157,
        "function": "stream_socket_client",
        "args": [
          "ssl://smtp.mailgun.com:587",
          0,
          "",
          10,
          4,
          null
        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
        "line": 251,
        "function": "initialize",
        "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\Stream\\SocketStream",
        "type": "->",
        "args": [

        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
        "line": 194,
        "function": "start",
        "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
        "type": "->",
        "args": [

        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/mailer/Transport/AbstractTransport.php",
        "line": 72,
        "function": "doSend",
        "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Mailer\\SentMessage"
          ]
        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
        "line": 136,
        "function": "send",
        "class": "Symfony\\Component\\Mailer\\Transport\\AbstractTransport",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Mailer\\SentMessage"
          ],
          [
            "Symfony\\Component\\Mailer\\DelayedEnvelope"
          ]
        ]
      },
      {
        "file": "/var/www/html/3rdparty/symfony/mailer/Mailer.php",
        "line": 45,
        "function": "send",
        "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Mime\\Email"
          ],
          null
        ]
      },
      {
        "file": "/var/www/html/lib/private/Mail/Mailer.php",
        "line": 216,
        "function": "send",
        "class": "Symfony\\Component\\Mailer\\Mailer",
        "type": "->",
        "args": [
          [
            "Symfony\\Component\\Mime\\Email"
          ]
        ]
      },
      {
        "file": "/var/www/html/apps/settings/lib/Controller/MailSettingsController.php",
        "line": 168,
        "function": "send",
        "class": "OC\\Mail\\Mailer",
        "type": "->",
        "args": [
          [
            "OC\\Mail\\Message"
          ]
        ]
      },
      {
        "file": "/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",
        "line": 230,
        "function": "sendTestMail",
        "class": "OCA\\Settings\\Controller\\MailSettingsController",
        "type": "->",
        "args": [

        ]
      },
      {
        "file": "/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",
        "line": 137,
        "function": "executeController",
        "class": "OC\\AppFramework\\Http\\Dispatcher",
        "type": "->",
        "args": [
          [
            "OCA\\Settings\\Controller\\MailSettingsController"
          ],
          "sendTestMail"
        ]
      },
      {
        "file": "/var/www/html/lib/private/AppFramework/App.php",
        "line": 183,
        "function": "dispatch",
        "class": "OC\\AppFramework\\Http\\Dispatcher",
        "type": "->",
        "args": [
          [
            "OCA\\Settings\\Controller\\MailSettingsController"
          ],
          "sendTestMail"
        ]
      },
      {
        "file": "/var/www/html/lib/private/Route/Router.php",
        "line": 315,
        "function": "main",
        "class": "OC\\AppFramework\\App",
        "type": "::",
        "args": [
          "OCA\\Settings\\Controller\\MailSettingsController",
          "sendTestMail",
          [
            "OC\\AppFramework\\DependencyInjection\\DIContainer"
          ],
          [
            "settings.MailSettings.sendTestMail"
          ]
        ]
      },
      {
        "file": "/var/www/html/lib/base.php",
        "line": 1068,
        "function": "match",
        "class": "OC\\Route\\Router",
        "type": "->",
        "args": [
          "/settings/admin/mailtest"
        ]
      },
      {
        "file": "/var/www/html/index.php",
        "line": 36,
        "function": "handleRequest",
        "class": "OC",
        "type": "::",
        "args": [

        ]
      }
    ],
    "File": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/Stream/SocketStream.php",
    "Line": 154,
    "message": "Sending mail to \"Array\n(\n    [a-fun-email@gmail.com] => AFunUsername\n)\n\" with subject \"Email setting test\" failed",
    "exception": {

    },
    "CustomMessage": "Sending mail to \"Array\n(\n    [a-fun-email@gmail.com] => AFunUsername\n)\n\" with subject \"Email setting test\" failed"
}

I'm thinking this may be an SSL negotiation error? Not sure

jessebot commented 7 months ago

This was the issue - I have logs now (yay!). Needed to set it to 0 and not 2 if needed, you can change the log level from 1 for info info to 2 for debug. (This was what confused me)

Doh! :facepalm: Sorry, that is my typoing fault!

I'm deploying on bare metal/my own servers. I'm using k8s, which I've deployed via kubespray I think the distro is vanilla kubernetes, bootstrapped with kubeadm Hopefully that answered your question.

Yeah that helps, thank you :)

Yay..... SSL errors. My favorite!

Yay logs! Boo ssl errors, haha. I hate troubleshooting mail servers, but especially secure mail stuff, but we can probably get through it! _{(also made a tiny edit to your post just to add syntax highlighting on the code block)}

I'm thinking this may be an SSL negotiation error? Not sure

So I took a look at email settings from the docs here and I still feel like all your settings are correct, unless Nextcloud really does require you use port 465 (which is weird, because 587 should still work as it's like "encryption when available" :thinking: ): https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/email_configuration.html#email

Looking at this error:

Connection could not be established with host \"ssl://smtp.mailgun.com:587\": stream_socket_client(): SSL operation failed with code 1. OpenSSL Error messages:\nerror:0A00010B:SSL routines::wrong version number"

I don't think this is a helm chart issue anymore, so I checked the nextcloud/server repo and searched the issues and found this promising closed issue: https://github.com/nextcloud/server/issues/37329

There's a lot of potential fixes there, but let's start with perhaps disabling the nextcloud.mail section of your values.yaml and instead passing in a config file for smtp.config.php the same way you did for logging, but this time, following the format described in the docs above, and then trying a couple of the fixes in the above linked issue:

If the above fixes don't work for you, it may be time to open an issue with nextcloud/server directly, but if you find anything we could do to spruce up the helm chart and add additional options, we're always happy to consider them or review a PR. Good luck and let us know how it goes! (also if any other community members are reading this and have run into this email issue, please feel free to chime in :pray: )

MaximUltimatum commented 7 months ago

Aha! Good call on switching to port 465. For reasons that escape me given the documentation, this worked. Should I make a PR updating any of documentation on this repo for that? Or is that just something that needs to be better outlined on Nextclouds docs?

Note: Embarrassingly enough, I also missed that smtp.mailgun.com as host should, in fact, have been smtp.mailgun.org. However, I don't believe this was the root issue, as I updated this and continued to get the same error when hitting smtp.mailgun.org:587. Given how smoothly mailgun setup has gone thus far, and how thorough their documentation and help pages are, I'm willing to believe they anticipated people making that error and forward traffic on those ports at mailgun.com to mailgun.org

jessebot commented 7 months ago

Aha! Good call on switching to port 465. For reasons that escape me given the documentation, this worked.

Woo!

Should I make a PR updating any of documentation on this repo for that? Or is that just something that needs to be better outlined on Nextclouds docs?

I think this makes sense to update in the Nextcloud docs, but it wouldn't hurt to submit a comment to the values.yaml mentioning that if 587 doesn't work and you're using TLS/SSL, to be please try 465 before submitting an issue. (not that submitting issues is a bad thing btw, just to try and empower users more)

Closing this now, as you're good to go, but feel free to keep responding :)

Also, changed the title, to help others find this issue better via search if they need it.

Given how smoothly mailgun setup has gone thus far, and how thorough their documentation and help pages are, I'm willing to believe they anticipated people making that error and forward traffic on those ports at mailgun.com to mailgun.org

we love a smart hosting provider :)

nextcloud / helm