Cannot send mail with msmtp

[GrexAut]

Hi,

I configured msmtp, because ssmtp was removed in the container. I tried so send mail with echo -e "To: example@mail.com\r\nSubject: Blah\r\n\r\nThis is a test" | msmtp example@mail.com

This is working

But with command mail (and all notification scrpts are working with command mail), I'm trying with: echo "Test" | mail -s "Testsubject" example@mail.com I get: mail: cannot send message: Process exited with a non-zero status

echo $? returns me erorr code 36, any ideas?

Thanks

[GrexAut]

Found out, that /etc/mail.rc is not existing, I created it with following content: "set sendmail="/usr/bin/msmtp -t"

Now it is working

[jjethwa]

Hi @GrexAut

Nice catch! Do you want to submit a PR to have that added? Thanks 😄

[GrexAut]

I created the pull request, hope its correct, because I'm n00b at building docker images :)

BTW: I noticed, that the files msmtprc and aliases have to be at /etc/ , not at /etc/msmtp Maybe you check this, and correct the wiki as required.

see here: https://marlam.de/msmtp/msmtp.html

3.5 Files

‘SYSCONFDIR/msmtprc’

The system configuration file. Use the --version option to find out what SYSCONFDIR is on your platform. 

Result from msmtp --version msmtp --version msmtp version 1.8.3 Platform: x86_64-pc-linux-gnu TLS/SSL library: GnuTLS Authentication library: GNU SASL Supported authentication methods: plain scram-sha-1 external gssapi cram-md5 digest-md5 login ntlm IDN support: enabled NLS: enabled, LOCALEDIR is /usr/share/locale Keyring support: none System configuration file name: /etc/msmtprc User configuration file name: /root/.msmtprc

[svenakela]

You don't need mail.rc in the container (it didn't exist before either). Mailing is using the sendmail command indirectly and sendmail is re-symlinked in the supervisor startup script of icinga2. You can execute the same test with:

echo "Test" | sendmail example@mail.com but echo "Test" | mail example@mail.com should work too.

I just did a test in two different containers and both are sending me e-mails with both sendmail and mail so I believe you might have another problem. I miss the mail.rc in several nodes, happily spamming our mailboxes. :)

[svenakela]

BTW: I noticed, that the files msmtprc and aliases have to be at /etc/ , not at /etc/msmtp Maybe you check this, and correct the wiki as required.

That's a typo in the README, the docker-compose file is correct though. I can edit that.

[svenakela]

@GrexAut Can you please verify that your container has a valid sendmail symlink? If you do following ls you should get

# ls -al /usr/sbin/sendmail 
lrwxrwxrwx 1 root root 14 Oct  7 17:55 /usr/sbin/sendmail -> /usr/bin/msmtp

[GrexAut]

@svenakela The original notification mail scripts using mailbin "mail" not, "sendmail", see: https://github.com/Icinga/icinga2/blob/master/etc/icinga2/scripts/mail-host-notification.sh#L7

Output of ls -l /usr/sbin/sendmail

# ls -l /usr/sbin/sendmail
lrwxrwxrwx 1 root root 14 Oct  8 14:36 /usr/sbin/sendmail -> /usr/bin/msmtp

[swtrustly]

Then you should not need mail.rc because mail itself uses sendmail as the mailing agent. As I already wrote, my nodes are working without mail.rc. I will build a new container on a fresh machine to see if I can copy your behaviour.

[GrexAut]

Please test, my test was as expected: mail.rc is required, see:

starting container with following paramters:

docker run --rm --name icinga2-test \
-h SRV-MON-Test \
-v /root/msmtp-test/aliases:/etc/aliases:ro \
-v /root/msmtp-test/msmtprc:/etc/msmtprc:ro \
-v /root/msmtp-test/mail.rc:/etc/mail.rc:ro \
-e TZ=Europe/Vienna \
--net mynetwork \
--ip 172.18.0.10 \
-t jordan/icinga2

The command inside the container works: echo "Test" | mail -s "Test" example@mail.com

Starting container with following parameters:

docker run --rm --name icinga2-test \
-h SRV-MON-Test \
-v /root/msmtp-test/aliases:/etc/aliases:ro \
-v /root/msmtp-test/msmtprc:/etc/msmtprc:ro \
-e TZ=Europe/Vienna \
--net mynetwork \
--ip 172.18.0.10 \
-t jordan/icinga2

root@SRV-MON-Test:/# echo "Test" | mail -s "Test" example@mail.com
mail: cannot send message: Process exited with a non-zero status
root@SRV-MON-Test:/# echo "set sendmail=\"/usr/bin/msmtp -t\"" > /etc/mail.rc
root@SRV-MON-Test:/# echo "Test" | mail -s "Test" example@mail.com
root@SRV-MON-Test:/#

[svenakela]

I took out the latest code base:

$ git clone https://github.com/jjethwa/icinga2.git
$ cd icinga2

I started a new container with following command:

$ docker run --rm --name icinga2-test \
-h SRV-MON-Test \
-v /home/svenakela/icinga2/msmtp/aliases:/etc/aliases:ro \
-v /home/svenakela/icinga2/msmtp/msmtprc:/etc/msmtprc:ro \
-e TZ=Europe/Vienna \
-t jordan/icinga2

Added credentials for a Gmail account in msmtp/msmtprc and then:

$ docker exec -it icinga2-test bash
# echo "TESTING FROM FRESH" | mail svenakela@mydomain.org

and the mail pops in my mailbox within seconds.

[GrexAut]

I tried it with docker pull (latest) from docker hub. Are there any difference?

[svenakela]

The latest code does not exist as a released Docker image yet. I believe you have to get the latest code base from github. It will work for you if you build the container locally. Maybe @jjethwa could make a new release with the updates?

No wait, you run latest...

[svenakela]

@GrexAut, can you please post your msmtprc (mail and password obfuscated). I think I found another typo. It doesn't work if the config is wrong and by using sendmail directly instead of mail you get a hint of it. I guess you are missing a defaults as the top line in the msmtprc file:

# Set default values for all following accounts.
defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
...

[jjethwa]

Yep @svenakela latest was built ~20 hours ago so @GrexAut should be good to go with pulling latest.

[svenakela]

I ran a clean Docker file and took the msmtp config from the README, and then I got the same error. Using sendmail directly there is a better error message. I then realized that the default header was missing in the config, I added that line and everything worked as expected. So this is probably another README typo from me.

[GrexAut]

This is my msmtprc file:

defaults

account barracuda
host 10.xxx.xxx.xxx
port 25
tls off

from icinga2@mytld.com

account default: barracuda

I deliver the mails to our mail gateway Is it wrong?

[svenakela]

I am pretty sure there is something with your config. I have tested on a local SMTP server but with a different config than yours. Login to your docker container and execute one of following, they will likely tell what the problem is. Either:

echo "HELO" | sendmail -v yourmail@mytld.com

or

echo "HELO" | mail -d yourmail@mytld.com

[GrexAut]

Hi,

sorry for my late response.

root@SRV-MON-TEST:/# echo "HELO" | sendmail -v yourmail@mytld.com loaded system configuration file /etc/msmtprc ignoring user configuration file /root/.msmtprc: No such file or directory falling back to default account using account default from /etc/msmtprc host = 10.xxx.xxx.xxx port = 25 source ip = (not set) proxy host = (not set) proxy port = 0 timeout = off protocol = smtp domain = localhost auth = none user = (not set) password = (not set) passwordeval = (not set) ntlmdomain = (not set) tls = off tls_starttls = on tls_trust_file = system tls_crl_file = (not set) tls_fingerprint = (not set) tls_key_file = (not set) tls_cert_file = (not set) tls_certcheck = on tls_min_dh_prime_bits = (not set) tls_priorities = (not set) auto_from = off maildomain = (not set) from = icinga2@mytld.com add_missing_from_header = on add_missing_date_header = on remove_bcc_headers = on dsn_notify = (not set) dsn_return = (not set) logfile = (not set) logfile_time_format = (not set) syslog = (not set) aliases = (not set) reading recipients from the command line <-- 220 smtp.mytld.local ESMTP mailgw ready. --> EHLO localhost <-- 250-smtp.mytld.local localhost [unknown] pleased to meet you. <-- 250 DSN --> MAIL FROM:icinga2@mytld.com <-- 250 OK --> RCPT TO:yourmail@mytld.com <-- 250 OK --> DATA <-- 354 Start mail input; end with . --> From: icinga2@mytld.com --> Date: Tue, 22 Oct 2019 09:22:33 +0200 --> HELO --> . <-- 250 OK --> QUIT <-- 221 Closing connection. root@SRV-MON-TEST:/# echo "HELO" | mail -d yourmail@mytld.com mail: unrecognized option '-d' root@SRV-MON-TEST:/# echo "HELO" | mail yourmail@mytld.com mail: cannot send message: Process exited with a non-zero status root@SRV-MON-TEST:/#

Sendmail works, mail does not work. Sendmail is not complaining about wrong configuration.

[svenakela]

Is it possible for you to hand me the Docker image?

[samo4b]

By any chance do you want to use icinga director with the provided mail-host-notification.sh from the /etc/icinga2/scripts folder?

I was also trying to get this to work but i also gotten the return code 36. As i looked into the logs the full command with all parameters was shown and as i executed it on command line everything worked fine.

I found a blog in my language which explained exactly the problem with the scripts and i used the scripts from that person and modified them for my needs. Now everything works fine without any changes to the system itself just different scripts.

Also i already submitted a bugfix request myself for a part of those scripts because i also gotten another error while testing.

See https://github.com/Icinga/icinga2/issues/7609 But this does not fix the error 36 it was only for the [: unexpected operator

The scripts from icinga are only a base and made to be modified. I found a blog for my language which exactly stated the problem with the scripts and icinga director and my notifications work now without a problem.

[jjethwa]

Hi @samo4b

Thanks for reporting the issue and you did the right thing by submitting it to the main Icinga2 repository. Do you know which line is exiting with the return code 36? Maybe try adding a set -x or set -xv in the script to see if it provides any more debugging information?

[samo4b]

Hi @jjethwa

Unfortunately adding set -x or set -xv to the script does not give me any additional information. But the error code appears after the script tries to execute the "mail" command. This is all i can see in the error log:

[2019-11-14 08:35:03 +0000] information/Notification: Completed sending 'Custom' notification 'server@example.com!Host Notifications' for checkable 'server@example.com' and user 'sam test' using command 'mail-host-notification'.

[2019-11-14 08:35:03 +0000] warning/PluginNotificationTask: Notification command for checkable 'server@example.com' and notification 'server@example.com!Host Notifications' (PID: 25237, arguments: '/etc/icinga2/scripts/mail-host-notification.sh' '-4' 'server@example.com' '-6' '' '-b' 'icingaadmin' '-c' 'Test Message' '-d' '2019-11-14 08:35:03 +0000' '-l' 'server@example.com' '-n' 'server@example.com' '-o' 'CRITICAL - Host Unreachable (server@example.com)' '-r' 'sam@example.com' '-s' 'DOWN' '-t' 'CUSTOM') terminated with exit code 36, output: mail: cannot send message: Process exited with a non-zero status

When i copy exactly those arguments (text was modified due to sensible information) into the command line the mail is sent correctly! So i assume the problem lies within icinga director somewhere.

In the blog which helped me the person stated that icinga director has currently no ENV variable support.

[jjethwa]

Hi @samo4b

That's really strange that the command ran when you tried it at the prompt. Did you run it as root or the icinga user?

[samo4b]

Hi @jjethwa

I ran the command under the root user from command line but the notifications from the icigina director over icingaweb2 UI run with a different user i think it's nagios.

[jjethwa]

Hi @samo4b

Can you try running it as the nagios or icinga user to see if you get an error?

[arpitshah20]

Hi @jjethwa, I want to send email notification through msmtp with use of icinga director

• msmtp configured properly and tested with cli, its working.

From UI icinga2 triggers this script /etc/icinga2/scripts/mail-service-notification.sh which runs succesfully and icinga2 logs says this [2020-02-13 10:06:38 +0000] information/ExternalCommandListener: Executing external command: [1581588398] SEND_CUSTOM_SVC_NOTIFICATION;ip-172-31-19-79 .ec2.internal;check-load;2;icingaadmin;sadasdad so the logs says everything ok but the mail is not sent. Did you guys found any workaround for this?

[jjethwa]

Hi @arpitshah20

Can you try executing the /etc/icinga2/scripts/mail-service-notification.sh script at the CLI with some test arguments to see if you receive the email or an error? Also, make sure that the custom service notification is set up to send emails, it could be disabled depending on your config,

[arpitshah20]

Hi @jjethwa, Thanks for the quick response. I tried executing the script /etc/icinga2/scripts/mail-service-notification.sh with cli but gives some command error as sendmail has '-s' option for sending an email to sender (i mean the sender address) Below command is executed and I am sharing the output for the same. /etc/icinga2/scripts/mail-service-notification.sh -r <some domain email> -d 1581589504 -e check-load -l <some private ip> -n nginx-plus-instance -u check-load -o "OK - load average: 0.44, 0.54" -s "ok" -t "custom" Output: sendmail: invalid option -- 's' Let me know where I am getting wrong on cli script. P.S: I have changed the script execution from mail command to sendmail command. And yes director has the custom service notification set up for services, even the warning and critical notifications are also not sent.

[jjethwa]

Hi @arpitshah20

Do you mean you changed the mail command to sendmail inside the mail-service-notification.sh script itself? You shouldn't need to do this. As @svenakela pointed out in the comments above, both mail and sendmail work properly, however mail has a different set of flags that are passed so you would need to modify the flags if switching to sendmail.

Edit: I would not recommend switching because then you're going to have to maintain the script yourself instead of using the defaults provided by Icinga

[arpitshah20]

yes I changed that command from mail to sendmail in the script itself. Ohh is it, ok I didn't knew that. Let me change back and post the results again back from cli with reverting mail command to it. After changing the command to mail again /etc/icinga2/scripts/mail-service-notification.sh -r <some domain email> -d 1581589504 -e check-load -l <some private ip> -n nginx-plus-instance -u check-load -o "OK - load average: 0.44, 0.54" -s "ok" -t "custom" Output: mail: cannot send message: Process exited with a non-zero status I am not able to see any logs as well.

[svenakela]

Wasn't there a Locale issue with the notification scripts? https://github.com/jjethwa/icinga2/issues/200#issuecomment-553426643

[arpitshah20]

Hi @svenakela, yes it was there. I fixed it. I changed the comparison string of double equal to single equal. I think that was the only issue right ?

[jjethwa]

@arpitshah20 maybe try adding a set -x to the script so you can see the mail command that is generated and run it manually?

[arpitshah20]

@jjethwa Sure, good idea. Let me try that and post here. Ran this cli command with set -x /etc/icinga2/scripts/mail-service-notification.sh -r <some email> -d 1581589504 -e check-load -l <some private ip> -n nginx-plus-instance -u check-load -o "OK - load average: 0.44, 0.54" -s "ok" -t "custom" Output: basename /etc/icinga2/scripts/mail-service-notification.sh PROG=mail-service-notification.sh hostname ICINGA2HOST=icinga2 MAILBIN=mail which mail [ -z /usr/bin/mail ] getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt USEREMAIL= getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt LONGDATETIME=1581589504 getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt SERVICENAME=check-load getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt HOSTNAME=ip-172-31-19-79.ec2.internal getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt HOSTDISPLAYNAME=nginx-plus-instance getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt SERVICEDISPLAYNAME=check-load getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt SERVICEOUTPUT=OK - load average: 0.44, 0.54 getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt SERVICESTATE=ok getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt NOTIFICATIONTYPE=custom getopts 4:6:b:c:d:e:f:hi:l:n:o:r:s:t:u:v: opt shift 18 eval PAR=$LONGDATETIME PAR=1581589504 [ ! 1581589504 ] eval PAR=$HOSTNAME PAR= [ ! ] eval PAR=$HOSTDISPLAYNAME PAR=nginx-plus-instance [ ! nginx-plus-instance ] eval PAR=$SERVICENAME PAR=check-load [ ! check-load ] eval PAR=$SERVICEDISPLAYNAME PAR=check-load [ ! check-load ] eval PAR=$SERVICEOUTPUT PAR=OK - load average: 0.44, 0.54 [ ! OK - load average: 0.44, 0.54 ] eval PAR=$SERVICESTATE PAR=ok [ ! ok ] eval PAR=$USEREMAIL PAR= [ ! ] eval PAR=$NOTIFICATIONTYPE PAR=custom [ ! custom ] SUBJECT=[custom] check-load on nginx-plus-instance is ok! cat NOTIFICATION_MESSAGE= Service Monitoring on icinga2 check-load on nginx-plus-instance is ok! Info: OK - load average: 0.44, 0.54 When: 1581589504 Service: check-load Host: [ -n ] [ -n ] [ -n ] [ -n ] [ = true ] [ -n ] mail -s [custom] check-load on nginx-plus-instance is ok! /usr/bin/printf %b Service Monitoring on icinga2 check-load on nginx-plus-instance is ok! Info: OK - load average: 0.44, 0.54 When: 1581589504 Service: check-load Host: mail: cannot send message: Process exited with a non-zero status

Also I used the created mail command from cli, it just stucks on the prompt. mail -s [custom] check-load on nginx-plus-instance is ok! <some email> Output: Cc: I need to interrupt multiple times to stop the above command.

[jjethwa]

Thanks @arpitshah20

When you ran it manually, you forgot to add the printf part that has the message, that's why it was waiting for input at the CLI. Try something like:

/usr/bin/printf "%b" "This is a test message" | mail -s "[custom] check-load on nginx-plus-instance is ok!" <YOUR_EMAIL_ADDRESS>

[arpitshah20]

Thanks @jjethwa for the responses and support. I am able to send email notification from director now. The tricky part was I missed the /etc/mail.rc and /etc/icinga2/conf.d/scripts/users.conf configuration

• I added the file /etc/mail.rc as per this comment https://github.com/jjethwa/icinga2/issues/200#issuecomment-539877463
• Changed the email to my domain email in this file /etc/icinga2/conf.d/scripts/users.conf
• Added notification template, user template, notification apply rule and users from director. yes that was it. Mail works like charm!! :+1: :tada: Thankyou all.

[jjethwa]

That's great news, @arpitshah20 😃

[trickert76]

I'm not sure if this is the same issue in my case, but I like to give you my solution.

First - try the mail-host-notification.sh and mail-service-notification.sh as the nagios user which may have different privileges then root with

sudo -u nagios /etc/icinga2/scripts/mail-service-notification.sh ...

inside the container. Sometimes this helps.

Then the msmtp tries to be like the default mail command. But the mail-host-notification and mail-service-notification.sh make a difference if the command is run under Debian and then uses different attributes (especially -a "From: .."). In that case msmtp interpretes the -a as "accountId" which is wrong. Because I run the host on Debian and the image is based on Debian the mail-host-notification and mail-service-notifcation cannot work with msmtp without changing the script.

So I removed the if condition inside both scripts and - now it works

@@ -155,18 +156,9 @@
## Send the mail using the $MAILBIN command.
## If an explicit sender was specified, try to set it.
if [ -n "$MAILFROM" ] ; then
+  /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -r "$MAILFROM" -s "$SUBJECT" $USEREMAIL
+else
+  /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -s "$SUBJECT" $USEREMAIL
+fi

-  ## Modify this for your own needs!
-
-  ## Debian/Ubuntu use mailutils which requires `-a` to append the header
-  if [ -f /etc/debian_version ]; then
-    /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -a "From: $MAILFROM" -s "$SUBJECT" $USEREMAIL
-  ## Other distributions (RHEL/SUSE/etc.) prefer mailx which sets a sender address with `-r`
-  else
-    /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -r "$MAILFROM" -s "$SUBJECT" $USEREMAIL
-  fi
-
-else
-  /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" \
-  | $MAILBIN -s "$SUBJECT" $USEREMAIL
-fi

[jjethwa]

Thanks for your findings @trickert76

The fixes for the notification scripts might be something we can automate in the /opt/run container init script. Maybe we just need to make the if statement always fail instead?

if [ -f /etc/debian_version ]; then

[trickert76]

I'm not sure, if this is a good idea, because /etc/debian_version is more relevant in other cases (I'm not sure, if apt uses it for example). If you only want to change one line with sed or so, just change the mail-line like:

sed -e 's/MAILBIN -a "From: /MAILBIN -r "/g'

That would be more or less idempotent in case of upgrades and could be called from the entrypoint script.

Otherwise - provide your own version of mail-host-notification and mail-service-notifcation and change the notification.conf and link that to your script. Or change the path in constants.conf or even remove the msmtp alias for mail and replace it with a wrapper that switches from -a to -r that simulates the mail-binary in the Debian version.

[jjethwa]

Hi @trickert76

Yes, sorry, I didn't explain that well hehe. I wouldn't touch the file itself, but modify the script check so it fails or something like you suggested.