text/x-mail MIME issues

[gh0stwizard]

Issue summary

I have found 2 issues when working with text/x-mail email messages:

1. Unable to view the body of a message with mime type text/x-mail.
2. Unable to release a message with mime type text/x-mail.

I am not including steps to reproduce these bugs, instead see patches below. Hope this will help. I am not exactly sure that this is correct way to fix the issue.

1. Allow view the body of a message

https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L541

--- /var/www/html/mailscanner/detail.php.orig   2019-07-15 14:49:21.311222291 +0300
+++ /var/www/html/mailscanner/detail.php     2019-07-15 14:49:54.533127253 +0300
@@ -533,7 +533,7 @@
                     $item['dangerous'] === 'N' ||
                     $_SESSION['user_type'] === 'A' ||
                     (defined('DOMAINADMIN_CAN_SEE_DANGEROUS_CONTENTS') && true === DOMAINADMIN_CAN_SEE_DANGEROUS_CONTENTS && $_SESSION['user_type'] === 'D' && $item['dangerous'] === 'Y')
-                ) && preg_match('!message/rfc822!', $item['type'])
+                ) && preg_match('!message/rfc822|text/x-mail!', $item['type'])
             ) {
                 echo '  <td><a href="viewmail.php?token=' . $_SESSION['token'] .'&amp;id=' . $item['msgid'] . '">' .
                     substr($item['path'], strlen($quarantinedir) + 1) .

2. Allow releasing a message

https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3584

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-15 13:21:48.935886617 +0300
@@ -3493,7 +3493,7 @@
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
         foreach ($num as $key => $val) {
-            if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+            if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
                 exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
                 if ($retval === 0) {

Installation

Version and method

• MailWatch Version: 1.2.7-dev (EFA 3.0.2.6)
• Install type: EFA https://efa-project.org
• Updated from an older MailWatch or fresh install: vendor (EFA)

Thank you!

[gh0stwizard]

I have added a fix for releasing messages from Message Operations list. Full patch is below.

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-16 10:27:39.143386292 +0300
@@ -3461,7 +3461,7 @@
             // Loop through each selected file and attach them to the mail
             foreach ($num as $key => $val) {
                 // If the message is of rfc822 type then set it as Quoted printable
-                if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+                if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                     $mime->addAttachment($list[$val]['path'], 'message/rfc822', 'Original Message', true, '');
                 } else {
                     // Default is base64 encoded
@@ -3492,8 +3492,9 @@
         // Use sendmail to release message
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
-        foreach ($num as $key => $val) {
-            if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+        $count = count($list);
+        for ($val = 0; $val < $count; $val++) {
+            if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
                 exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
                 if ($retval === 0) {

The issue with the foreach loop is due the fact how exactly the second argument is passed to quarantine_release in the /var/www/html/mailscanner/do_message_ops.php file. Pay attention on how $itemnum is counted. It means that $itemnum is always equal to [ 0 ] array: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/do_message_ops.php#L96-L113

But in the case when a message has an attachment like .zip and the message is considered as Bad Content by some reason, then the quarantined directory contains multiple files, for instance (got from the real blocked message):

• my.cnf | text/plain; charset=us-ascii | 20190716/3C0A11231E8.AD0E2/my.cnf
• diags.tar | application/x-tar; charset=binary | 20190716/3C0A11231E8.AD0E2/diags.tar
• message | message/rfc822; charset=us-ascii | 20190716/3C0A11231E8.AD0E2/message

The file order is major issue here. So, the loop foreach ($num as $key => $val) { will check first file in the order only (my.cnf to be exactly). Because $num in the loop is the value $itemnum from do_message_ops.php and this value always equal to [0] (precisely, $num = 0; $itemnum = array($num);). Hope I explained this clear.

In the time of writing this message I've realized that call of quarantine_learn() also is incorrect in the do_message_ops.php file. There is need a fix either in do_message_ops.php and/or both quarantine_release(), quarantine_learn(). Specifically, quarantine_learn() must pass to the spamassassin command only complete the email message file, IMHO. But quarantine_learn() does not contain such a check.

UPDATE

I have rewrote the patch taking into account that the value $num of quarantine_release($list, $num, $to, $rpc_only = false) is an array of selected items. But I see there is a mime type check, so probably the version below is better:

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-16 11:24:46.860228320 +0300
@@ -3461,7 +3461,7 @@
             // Loop through each selected file and attach them to the mail
             foreach ($num as $key => $val) {
                 // If the message is of rfc822 type then set it as Quoted printable
-                if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+                if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                     $mime->addAttachment($list[$val]['path'], 'message/rfc822', 'Original Message', true, '');
                 } else {
                     // Default is base64 encoded
@@ -3492,8 +3492,11 @@
         // Use sendmail to release message
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
+        if (count($num) < count($list)) {
+            $num = array_keys($list);
+        }
         foreach ($num as $key => $val) {
-            if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+            if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
                 exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
                 if ($retval === 0) {

[Skywalker-11]

Is there a standard or similar for the structure of text/x-mail? I could only find a reference to "x-..." prefix in general referencing unstandardized protocolls.

[gh0stwizard]

@Skywalker-11 I did not dig so deep, but see http://lists.roundcube.net/pipermail/dev/2013-January/022097.html

Ok, while writing this message I have found out what is a cause. On CentOS 6.10 there is file-libs package version 5.04. I got source from https://www.darwinsys.com/file/ and dig in inside:

$ grep -nrR 'x-mail' file-5.04
file-5.04/src/names.h:70:       { "mail",                                       "text/x-mail" },

As expected file returns:

$ file -bi text_x-mail.message.txt
text/x-mail; charset=us-ascii

I have posted an example email message here: https://pastebin.com/2LEhbzhz

Steps to reproduce:

1. Include any non-empty file with any blocked extension in MailScanner.
2. That's all.

[gh0stwizard]

@Skywalker-11 Here is a real cause, compare this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3498 and this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3516-L3517

On the detail page of the message you see message/rfc822 because of this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L380 https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L503 https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L524-L533

Conclusion. Depending on if a message is blocked or not quarantine_list_items() returns either hard-coded value message/rfc822, either what file -bi FILE has returned.

UPDATE

Easiest workaround instead of fixing all regular expressions with message/rfc822 is below. Changes:

1. Use text/x-mail as an alias for message/rfc822 in quarantine_list_items() function.
2. Fix quarantine_release() to look all extracted files within a message. If the [mime] type of a file is message/rfc822 then release the message. Tested on only when QUARANTINE_USE_SENDMAIL is true (see conf.php for details).
3. Fix quarantine_learn() to look all extracted files within a message. If the [mime] type of a file is message/rfc822 then pass the path of the file to spamassassin, e.g. complete message and not extracted from the message attachments.

--- /var/www/html/mailscanner/functions.php.orig        2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php  2019-07-16 16:55:31.642773396 +0300
@@ -3394,7 +3394,11 @@
                     $quarantined[$count]['to'] = $row->to_address;
                     $quarantined[$count]['file'] = $f;
                     $file = escapeshellarg($quarantine . '/' . $f);
-                    $quarantined[$count]['type'] = ltrim(rtrim(`/usr/bin/file -bi $file`));
+                    $type = ltrim(rtrim(`/usr/bin/file -bi $file`));
+                    if (preg_match('!^text/x-mail!', $type)) {
+                        $type = 'message/rfc822';
+                    }
+                    $quarantined[$count]['type'] = $type;
                     $quarantined[$count]['path'] = $quarantine . '/' . $f;
                     $quarantined[$count]['md5'] = md5($quarantine . '/' . $f);
                     $quarantined[$count]['dangerous'] = $infected;
@@ -3492,6 +3496,9 @@
         // Use sendmail to release message
         // We can only release message/rfc822 files in this way.
         $cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
+        if (count($num) < count($list)) {
+            $num = array_keys($list);
+        }
         foreach ($num as $key => $val) {
             if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
                 debug($cmd . $list[$val]['path']);
@@ -3566,7 +3573,14 @@
     if (!$rpc_only && is_local($list[0]['host'])) {
         //prevent sa-learn process blocking complete apache server
         session_write_close();
+        if (count($num) < count($list)) {
+            $num = array_keys($list);
+        }
         foreach ($num as $key => $val) {
+            if (!preg_match('!message/rfc822!', $list[$val]['type'])) {
+                continue;
+            }
+
             $use_spamassassin = false;
             $isfn = '0';
             $isfp = '0';

[Schroeffu]

Have had exactly the same problem with mails encoded text/plain; charset=us-ascii in mailwatch v1.2.10. Changing the two lines initially explained to:

(detail.php) ) && preg_match('!message/rfc822|text/plain!', $item['type']) (function.php) if (preg_match('/message\/rfc822|text\/plain/', $list[$val]['type'])) {

and change conf.php to define('QUARANTINE_USE_SENDMAIL', true); (usually i am using "false") then the release works.

But i am using usually QUARANTINE_USE_SENDMAIL false. This way the attachement called "message" is in a corrupt format, cant open it with any app (ok with atom f.e. but the content is not read-able by outlook. in a text editor atom i see the mail content is base64 encoded -.-

Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64)

[Schroeffu]

Today i've had the same issue with another mail release again. still no fix published ? :-(

[jensd0e]

To add to this issue, I had to change detail.php and functions.php to include another type. detail.php: ) && preg_match('!message/rfc822|text/plain|text/x-mail|text/html!', $item['type'])

functions.php: if (preg_match('/^(message\/rfc822|text\/plain|text\/x-mail|text\/html)/', $list[$val]['type'])) {

Is there a more generic solution to this problem?

[shawniverson]

@jensd0e I don't understand why you would do this for text/plain and text/html. Both of these represent the the text and html mime parts of a multipart mime email message that should have mime type message/rfc822. Can you share with me some more details, perhaps screenshots of what you see in MailWatch?

Also, thank you for calling my attention to this old issue. I will work on a fix for text/x-mime, but I need to understand how you are ending up with a message with text/plain and text/html.

[jensd0e]

@shawniverson I'm so glad, that you responded to my problem :-) Now, I'm happy to help. First things first: PHP is far from things I do at all. Reading and finding the "error" took me some hours, since I didn't understood what was going at first. And looking at the code that decides, weather a mail is released or not was bringing me here. The messages in question had file types different from message/rfc822: text/plain and text/html.

Here are the details for one of the mails:

• The email had two *.docx with a long names and two PDFs with long names attached to it.
• The message was of type 'text/plain; charset=us-ascii'.
• Only one attachment was reported in MailWatch. That was the first attached file.
• The reason for quarantine was 'Very long filenames are good signs of attacks against Microsoft e-mail packages (26_start_of_filename.doc)'. Only the start of the filename and the file extension was kept in the message. Actually the file was a *.docx file with a long filename.
• The attachment was of type 'application/vnd.openxmlformats-officedocument.wordprocessingml.document; charset=binary'.

Why the other three attachments very excluded from the report, I don't know. Another mail I had to release had text/html as file type. If I can help to find the underlying cause and fix it, please let me know, how I can help.

[shawniverson]

@jensd0e Are you able to get me a screenshot of the quarantine file list in the details pane on one of these quarantined emails?

[jensd0e]

@shawniverson I send you a PM with screenshots. Just did a test and I'm able to reproduce this with any file type attached, as long as the file name triggers the mailscanner rule.

[shawniverson]

@jensd0e thank you for the screenshots. Here is why you are having some trouble.

So you have a DKIM-Signature: line ahead of your Received: header as the first line of the message file. This is fine, but the file command by default does not detect this as message/rfc822 . To fix this without having to recode mailwatch, add this line to /etc/magic:

0       string/t                DKIM-Signature:         RFC 822 mail text
!:mime  message/rfc822

[jensd0e]

Thanks for your help. I really appreciate your support.