bug can't send emails when specifying SMTP config from compose ENV vars

[xsolinsx]

I have the configured my docker-compose.yml to have the SMTP_* variables specified, in order to be able to send emails.

services:
  nextcloud:
    container_name: nextcloud
    image: nextcloud:29.0.2
    restart: unless-stopped
    environment:
      NEXTCLOUD_ADMIN_PASSWORD_FILE: /run/secrets/NEXTCLOUD_ADMIN_PASSWORD
      NEXTCLOUD_ADMIN_USER_FILE: /run/secrets/NEXTCLOUD_ADMIN_USER
      NEXTCLOUD_TRUSTED_DOMAINS: nextcloud.${PROJECTS_DOMAIN}
      OVERWRITECLIURL: https://nextcloud.${PROJECTS_DOMAIN}
      OVERWRITEPROTOCOL: https
      PHP_UPLOAD_LIMIT: 128G
      PHP_MEMORY_LIMIT: 2G
      POSTGRES_DB_FILE: /run/secrets/POSTGRES_DB
      POSTGRES_HOST: postgres
      POSTGRES_PASSWORD_FILE: /run/secrets/POSTGRES_PASSWORD
      POSTGRES_USER_FILE: /run/secrets/POSTGRES_USER
      REDIS_HOST: redis
      REDIS_HOST_PASSWORD_FILE: /run/secrets/REDIS_HOST_PASSWORD
      MAIL_FROM_ADDRESS: nextcloud
      MAIL_DOMAIN: ${PROJECTS_DOMAIN}
      SMTP_HOST: smtp.gmail.com
      SMTP_SECURE: tls
      SMTP_PORT: 587
      SMTP_AUTHTYPE: LOGIN
      SMTP_NAME: CENSORED
      SMTP_PASSWORD_FILE: /run/secrets/SMTP_PASSWORD
      TRUSTED_PROXIES: 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 fd00::/8
    volumes:
      - nextcloud_conf:/var/www/html/config:rw
      - nextcloud_secrets:/run/secrets:ro

Credentials work fine, I already have other services using the same credentials and there is no problem with them.

When I start the container and explore the config.php file I cannot find any mail_ settings, but when I go to the /index.php/settings/admin I can see that the configuration is being populated, even though it is not complete. You can notice that indeed the authentication required flag is disabled and that the fields for SMTP_NAME and SMTP_PASSWORD are not visible

When clicking on the authentication required flag to enable it, I get the following config.php lines which are reflected on the UI

  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => 'smtp.gmail.com',
  'mail_smtpport' => '587',
  'mail_smtpauth' => 1,
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpname' => 'CENSORED',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'mydomain.com',
  'mail_smtppassword' => '',
  'mail_sendmailmode' => 'smtp',

Notice that the password field is not displaying anything, even though the password is present in the SMTP_PASSWORD_FILE. Now, when I update the password field and click the Save button something weird happens, the mail_smtpauth flag goes to false

  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => 'smtp.gmail.com',
  'mail_smtpport' => '587',
  'mail_smtpauth' => false,
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpname' => 'CENSORED',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'mydomain.com',
  'mail_smtppassword' => 'PWD',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpsecure' => 'tls',

Of course the Send Test Email button returns an error A problem occurred while sending the email. Please revise your settings. (Error: Email could not be sent. Check your mail server log) With the following relevant error message only visible at DEBUG level from the log viewer

{
    "reqId": "Y0NwRvAnhAoLVynjWvYC",
    "level": 0,
    "time": "2024-06-19T19:58:56+00:00",
    "remoteAddr": "192.168.1.1",
    "user": "MYUSER",
    "app": "core",
    "method": "POST",
    "url": "/index.php/settings/admin/mailtest",
    "message": "Sending mail to \"Array\n(\n    [myemail] => myname\n)\n\" with subject \"Email setting test\" failed",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0",
    "version": "29.0.2.2",
    "exception": {
        "Exception": "Symfony\\Component\\Mailer\\Exception\\TransportException",
        "Message": "Expected response code \"250\" but got code \"530\", with message \"530-5.7.0 Authentication Required. For more information, go to\r\n530 5.7.0  https://support.google.com/mail/?p=WantAuthError 5b1f17b1804b1-4247d0beb08sm961475e9.17 - gsmtp\".",
        "Code": 530,
        "Trace": [
            {
                "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
                "line": 182,
                "function": "assertResponseCode",
                "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
                "type": "->",
                "args": [
                    "530-5.7.0 Authentication Required. For more information, go to\r\n530 5.7.0  https://support.google.com/mail/?p=WantAuthError 5b1f17b1804b1-4247d0beb08sm961475e9.17 - gsmtp\r\n",
                    [
                        250
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
                "line": 235,
                "function": "executeCommand",
                "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
                "type": "->",
                "args": [
                    "MAIL FROM:<nextcloud@mydomain.com>\r\n",
                    [
                        250
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
                "line": 199,
                "function": "doMailFromCommand",
                "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
                "type": "->",
                "args": [
                    "nextcloud@mydomain.com"
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/mailer/Transport/AbstractTransport.php",
                "line": 72,
                "function": "doSend",
                "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Mailer\\SentMessage"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
                "line": 136,
                "function": "send",
                "class": "Symfony\\Component\\Mailer\\Transport\\AbstractTransport",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Mailer\\SentMessage"
                    ],
                    [
                        "Symfony\\Component\\Mailer\\DelayedEnvelope"
                    ]
                ]
            },
            {
                "file": "/var/www/html/3rdparty/symfony/mailer/Mailer.php",
                "line": 45,
                "function": "send",
                "class": "Symfony\\Component\\Mailer\\Transport\\Smtp\\SmtpTransport",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Mime\\Email"
                    ],
                    null
                ]
            },
            {
                "file": "/var/www/html/lib/private/Mail/Mailer.php",
                "line": 196,
                "function": "send",
                "class": "Symfony\\Component\\Mailer\\Mailer",
                "type": "->",
                "args": [
                    [
                        "Symfony\\Component\\Mime\\Email"
                    ]
                ]
            },
            {
                "file": "/var/www/html/apps/settings/lib/Controller/MailSettingsController.php",
                "line": 168,
                "function": "send",
                "class": "OC\\Mail\\Mailer",
                "type": "->",
                "args": [
                    [
                        "OC\\Mail\\Message"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",
                "line": 232,
                "function": "sendTestMail",
                "class": "OCA\\Settings\\Controller\\MailSettingsController",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",
                "line": 138,
                "function": "executeController",
                "class": "OC\\AppFramework\\Http\\Dispatcher",
                "type": "->",
                "args": [
                    [
                        "OCA\\Settings\\Controller\\MailSettingsController"
                    ],
                    "sendTestMail"
                ]
            },
            {
                "file": "/var/www/html/lib/private/AppFramework/App.php",
                "line": 184,
                "function": "dispatch",
                "class": "OC\\AppFramework\\Http\\Dispatcher",
                "type": "->",
                "args": [
                    [
                        "OCA\\Settings\\Controller\\MailSettingsController"
                    ],
                    "sendTestMail"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Route/Router.php",
                "line": 338,
                "function": "main",
                "class": "OC\\AppFramework\\App",
                "type": "::",
                "args": [
                    "OCA\\Settings\\Controller\\MailSettingsController",
                    "sendTestMail",
                    [
                        "OC\\AppFramework\\DependencyInjection\\DIContainer"
                    ],
                    [
                        "settings.mailsettings.sendtestmail"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/base.php",
                "line": 1050,
                "function": "match",
                "class": "OC\\Route\\Router",
                "type": "->",
                "args": [
                    "/settings/admin/mailtest"
                ]
            },
            {
                "file": "/var/www/html/index.php",
                "line": 49,
                "function": "handleRequest",
                "class": "OC",
                "type": "::",
                "args": []
            }
        ],
        "File": "/var/www/html/3rdparty/symfony/mailer/Transport/Smtp/SmtpTransport.php",
        "Line": 307,
        "message": "Sending mail to \"Array\n(\n    [myemail] => myname\n)\n\" with subject \"Email setting test\" failed",
        "exception": {},
        "CustomMessage": "Sending mail to \"Array\n(\n    [myemail] => myname\n)\n\" with subject \"Email setting test\" failed"
    }
}

Ignoring the fact that from 1 this field moved to the boolean false, one would expect that setting it as boolean true or 1 and restarting the container would fix the problem, however, this is not the case. Setting it as boolean true or 1 and restarting the container makes the GUI appear as the first picture (with the flag disabled). I tried also with '1' and 'true' but to no avail.

I have also checked that the smtp.config.php is aligned to the latest version

<?php
if (getenv('SMTP_HOST') && getenv('MAIL_FROM_ADDRESS') && getenv('MAIL_DOMAIN')) {
  $CONFIG = array (
    'mail_smtpmode' => 'smtp',
    'mail_smtphost' => getenv('SMTP_HOST'),
    'mail_smtpport' => getenv('SMTP_PORT') ?: (getenv('SMTP_SECURE') ? 465 : 25),
    'mail_smtpsecure' => getenv('SMTP_SECURE') ?: '',
    'mail_smtpauth' => getenv('SMTP_NAME') && (getenv('SMTP_PASSWORD') || (getenv('SMTP_PASSWORD_FILE') && file_exists(getenv('SMTP_PASSWORD_FILE')))),
    'mail_smtpauthtype' => getenv('SMTP_AUTHTYPE') ?: 'LOGIN',
    'mail_smtpname' => getenv('SMTP_NAME') ?: '',
    'mail_from_address' => getenv('MAIL_FROM_ADDRESS'),
    'mail_domain' => getenv('MAIL_DOMAIN'),
  );

  if (getenv('SMTP_PASSWORD_FILE') && file_exists(getenv('SMTP_PASSWORD_FILE'))) {
      $CONFIG['mail_smtppassword'] = trim(file_get_contents(getenv('SMTP_PASSWORD_FILE')));
  } elseif (getenv('SMTP_PASSWORD')) {
      $CONFIG['mail_smtppassword'] = getenv('SMTP_PASSWORD');
  } else {
      $CONFIG['mail_smtppassword'] = '';
  }
}

[joshtrichards]

When I start the container and explore the config.php file I cannot find any mail_ settings, but when I go to the /index.php/settings/admin I can see that the configuration is being populated, even though it is not complete.

The image uses Nextcloud's multi-config.php support. You must use occ config:list system to see your real runtime config.

Even if there is a bug of some sort, you'll need to check that output before touching anything in the Web UI, in order to see what is/isn't making it through for some reason.

Note, you should not use the Web UI to modify anything SMTP/MAIL settings if you're using Compose (I understand you're mostly using it at the moment to look at the values, but I'm mentioning it since you did attempt to change some options). If you change any SMTP/MAIL options in the Web UI, they get written to your main config.php, but the smtp.config.php will override them when Nextcloud merges the configs. That's why occ config:system list is the best way to check your real config.

There may also be some clues in your nextcloud container's Docker logs, such as if it can't load the secrets from some of those files or something.

[xsolinsx]

Thanks for pointing that out! I was able to find what was wrong on my configuration.

The command docker exec -u www-data -it nextcloud php occ config:list system returned

Warning: file_get_contents(/run/secrets/REDIS_HOST_PASSWORD): Failed to open stream: Permission denied in /var/www/html/config/redis.config.php on line 8
An unhandled exception has been thrown:
Exception: Config file has leading content, please remove everything before "<?php" in redis.config.php in /var/www/html/lib/private/Config.php:257
Stack trace:
#0 /var/www/html/lib/private/Config.php(71): OC\Config->readData()
#1 /var/www/html/lib/base.php(150): OC\Config->__construct('/var/www/html/c...')
#2 /var/www/html/lib/base.php(622): OC::initPaths()
#3 /var/www/html/lib/base.php(1181): OC::init()
#4 /var/www/html/console.php(48): require_once('/var/www/html/l...')
#5 /var/www/html/occ(11): require_once('/var/www/html/c...')
#6 {main}

I checked what was said in the error but the redis.config.php was looking fine. Then I checked the permissions on the files and they were screwed, so I updated them from outside the container with chown -R www-data:root path/to/nextcloud/secrets/ (I use a docker plugin for volumes that allows me to persist files wherever I want on disk even after deleting the volume) which allowed me to execute the command you mentioned and to see the actual configuration, which was now looking fine, also the GUI was reflecting the values inside the docker compose.

The Send Test Email button did actually work and I also received an email which I believe was stuck for quite some time 😆

So it was my bad for not actually executing the command written in the initial comment of the bug report in the first place 👀
Sorry for the bother and thanks again :)