Genmon sending Blank e-mails

[lps-rocks]

Expected Behavior

Genmon send e-mail with contents

Actual Behavior

E-mails from genmon are blank

Steps to Reproduce (including precondition)

Configure e-mail notifications, test notifications, get blank emails

Screenshot or Pictures relating to the problem (if possible)

https://i.hexhost.net/7e5676c3-1dec-49f3-8cef-6386ce1178ef.jpg

Your Environment

• Genmon Logs: Submitted
• Genmon Version: V1.18.12

Genmon keeps throwing an error also whenever it goes to send a message: 2022-06-22 18:32:54,328 : Error in MyConfig:ReadValue: MyMail: extend_wait: invalid literal for int() with base 10: '' : myconfig.py:94

However, extend_wait exists as a number zero in mymail.conf - so i'm puzzled on this one...

[jgyates]

Hi @lps-rocks

Make sure you are looking at the file /etc/genmon/mymail.conf for the extend_wait parameter and not the file with the rest of the genmon files (usually in your /home/pi/genmon/conf folder). The file in the conf folder is the original copy and is not used at by genmon. The file in /etc/genmon is used by genmon. This error can be removed by either removing this entry or making it a valid integer. This error has no effect on sending mail.

I am guessing your problem is the security settings you have on your email client. You sent a screen shot but I don't know where this is from. To me, it looks like your email client is blocking the email from genmon. What email client are you using? You can validate the email is sent correctly by looking at the sent folder on your email account. For example if you are using a gmail account to send emails from genmon, log into the gmail account in a web browser and look at the Sent folder associated with the email account. This should allow you to validate the email is being sent and the issue is with the client's security settings.

Let me know how it goes.

[lps-rocks]

Make sure you are looking at the file /etc/genmon/mymail.conf for the extend_wait parameter and not the file with the rest of the genmon files (usually in your /home/pi/genmon/conf folder). The file in the conf folder is the original copy and is not used at by genmon. The file in /etc/genmon is used by genmon. This error can be removed by either removing this entry or making it a valid integer. This error has no effect on sending mail.

Thanks, modifying it there made the errors stop.

I am guessing your problem is the security settings you have on your email client. You sent a screen shot but I don't know where this is from. To me, it looks like your email client is blocking the email from genmon. What email client are you using? You can validate the email is sent correctly by looking at the sent folder on your email account. For example if you are using a gmail account to send emails from genmon, log into the gmail account in a web browser and look at the Sent folder associated with the email account. This should allow you to validate the email is being sent and the issue is with the client's security settings.

I've tried viewing the body in multiple platforms, iOS, Outlook, Outlook Web, and they all show no body content in the e-mail. Clicking 'View Source' only shows headers and no body content in this instance. Other emails show both headers and body content.

[jgyates]

If you do not have any errors in your /var/log/mymail.log then this is the recommended next step: You can validate the email is sent correctly by looking at the sent folder on your email account. For example if you are using a gmail account to send emails from genmon, log into the gmail account in a web browser and look at the Sent folder associated with the email account.

[jgyates]

what is the provider of this email (gmail.com, outlook.com, etc)?

[lps-rocks]

I'm not using an e-mail account to send, I'm using an MS Exchange server and genmon is allowed to SMTP relay through it. The recipient accounts are MS365 accounts.

This is also a recent change. We used to get email contents, after one of the genmon updates it started sending blanks.

[lps-rocks]

Specifically, we went from version 1.18.08 to 1.18.12 and started having this problem. Looking at the commit history for mymail.py, there were many changes between those two versions. I'm going to try checking out 1.18.08 mymail and then go forward one commit at a time until the issue it shows up.

[jgyates]

Let me know how the 1.18.08 test goes. That would be a good data point. According to this article:

https://success.phishinsight.trendmicro.com/en/support/solutions/articles/61000292358--we-could-not-verify-the-identity-of-the-sender-warning-message

This message is related to the Spoof Intelligence features of anonymous relays. Have you explicitly allows the sender email used in genmon to be allows to spoof (SNMP relay without a matching account). The above link also has info regarding the security settings to allow the SMTP relay to work fi that is the issue.

[lps-rocks]

Checking out the old version for just mymail.py fixed it, I'm currently iterating up through the commits till I find the one that broke the email bodies.

[lps-rocks]

Found the commit that broke it, 08b23740d76a7967d1f57c1973ad27a782d99d68

There we go, this commit breaks the body.

[lps-rocks]

E-mail bodies work with commit e2fe33c0f11c988314523d767ca671030f0e9cfb and prior.

Since it looks like the commit that breaks has to do with python3 compatibility, I'm running Python 3.9.2

[lps-rocks]

Looks like it's the usage of as_bytes that's mangling the message most likely due to the warnings the python developers give:

https://docs.python.org/3/library/email.compat32-message.html#email.message.Message.as_bytes

Note that this method is provided as a convenience and may not always format the message the way you want. For example, by default it does not do the mangling of lines that begin with From that is required by the unix mbox format. For more flexibility, instantiate a BytesGenerator instance and use its flatten() method directly. For example:

[lps-rocks]

Yeah, there's something weird going on with using a bytes object into session.sendmail for the messages. As soon as I change it back to using msg.as_string() for that call, it works. I validated that I'm getting the same exact output on both as_string and as_bytes, just in different types. This may be a python bug.

[lps-rocks]

Found the problem. The body is not well formatted.

b'Content-Type: multipart/mixed; boundary="===============1314857552=="\nMIME-Version: 1.0\nFrom: Genmon \nTo: \nDate: Thu, 30 Jun 2022 14:01:55 -0400\nSubject: Genmon Test Email\n\n--===============1314857552==\nContent-Type: text/plain; charset="us-ascii"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\n\n\nTime: 02:01:55 PM\nDate: Thu 30-Jun-2022\n\nTest email from genmon\n\n--===============1314857552==--\n'

That's an illegal body.

https://docs.python.org/3.9/library/smtplib.html#smtplib.SMTP.sendmail

msg may be a string containing characters in the ASCII range, or a byte string. A string is encoded to bytes using the ascii codec, and lone \r and \n characters are converted to \r\n characters. A byte string is not modified.

In string format, the call to sendmail fixes the error. In bytes format, it does not.

[lps-rocks]

I see subsequent commits try and fix this but don't entirely do-so. I'm still investigating.

[lps-rocks]

Even in the latest commit I still get:

b'Content-Type: multipart/mixed; boundary="===============0302288506=="\nMIME-Version: 1.0\nFrom: Genmon \nTo: \nDate: Thu, 30 Jun 2022 14:11:03 -0400\nSubject: Genmon Test Email\n\n--===============0302288506==\nContent-Type: text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: base64\n\nDQpUaW1lOiAwMjoxMTowMyBQTQ0KRGF0ZTogVGh1IDMwLUp1bi0yMDIyDQoNClRlc3QgZW1haWwg\nZnJvbSBnZW5tb24NCg==\n\n--===============0302288506==--\n'

The explicitly added carriage returns are missing. So, MIMEMultipart or msg.attach is stripping them out.

[jgyates]

what version of python are you using?

[lps-rocks]

3.9.2 - I found the issue. MimeMultipart and MimeText default to using a compatible policy that is designed for backwards compatibility which strips carriage returns. I'm going to open a PR.

I do have to ask, why bother with python2 compatibility?

[jgyates]

I don't actively test or support python2, however I have not actively removed the support for it from the code. If I come across something that I know will break python 2, I try to add conditionals for the sake of older installations that still run Rasbpian 9. If that ever gets to a point that it prohibits maintainability I will drop the conditionals. In most cases, the issue is something needs to be added to support python3 (specifically how strings are handled) so I just added it and keep the older code with conditionals.

[lps-rocks]

Ah, okay. Makes sense.

I found the issue with why the bodies are created out of spec. The default policy that MIMEMultipart and MimeText use strips the carriage returns in anticipation that smtplib.sendmail will add them back. smtplib.sendmail does this when passing the body in as a string, but as a bytes objects it's left uncorrected and the malformed components are dropped by an SMTP server. I've corrected the policy to be SMTP compliant (it's also RFC-compliant, so it shouldn't affect any other sending mechanisms either).

[jgyates]

Thanks for tracking this down. Great work!