Closed gh0stwizard closed 1 day ago
I have added a fix for releasing messages from Message Operations
list. Full patch is below.
--- /var/www/html/mailscanner/functions.php.orig 2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php 2019-07-16 10:27:39.143386292 +0300
@@ -3461,7 +3461,7 @@
// Loop through each selected file and attach them to the mail
foreach ($num as $key => $val) {
// If the message is of rfc822 type then set it as Quoted printable
- if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+ if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
$mime->addAttachment($list[$val]['path'], 'message/rfc822', 'Original Message', true, '');
} else {
// Default is base64 encoded
@@ -3492,8 +3492,9 @@
// Use sendmail to release message
// We can only release message/rfc822 files in this way.
$cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
- foreach ($num as $key => $val) {
- if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+ $count = count($list);
+ for ($val = 0; $val < $count; $val++) {
+ if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
debug($cmd . $list[$val]['path']);
exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
if ($retval === 0) {
The issue with the foreach
loop is due the fact how exactly the second argument is passed to quarantine_release
in the /var/www/html/mailscanner/do_message_ops.php
file. Pay attention on how $itemnum
is counted. It means that $itemnum
is always equal to [ 0 ]
array: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/do_message_ops.php#L96-L113
But in the case when a message has an attachment like .zip
and the message is considered as Bad Content
by some reason, then the quarantined directory contains multiple files, for instance (got from the real blocked message):
my.cnf | text/plain; charset=us-ascii | 20190716/3C0A11231E8.AD0E2/my.cnf
diags.tar | application/x-tar; charset=binary | 20190716/3C0A11231E8.AD0E2/diags.tar
message | message/rfc822; charset=us-ascii | 20190716/3C0A11231E8.AD0E2/message
The file order is major issue here. So, the loop foreach ($num as $key => $val) {
will check first file in the order only (my.cnf
to be exactly). Because $num
in the loop is the value $itemnum
from do_message_ops.php
and this value always equal to [0]
(precisely, $num = 0; $itemnum = array($num);
). Hope I explained this clear.
In the time of writing this message I've realized that call of quarantine_learn()
also is incorrect in the do_message_ops.php
file. There is need a fix either in do_message_ops.php
and/or both quarantine_release()
, quarantine_learn()
. Specifically, quarantine_learn()
must pass to the spamassassin
command only complete the email message
file, IMHO. But quarantine_learn()
does not contain such a check.
UPDATE
I have rewrote the patch taking into account that the value $num
of quarantine_release($list, $num, $to, $rpc_only = false)
is an array of selected items. But I see there is a mime type check, so probably the version below is better:
--- /var/www/html/mailscanner/functions.php.orig 2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php 2019-07-16 11:24:46.860228320 +0300
@@ -3461,7 +3461,7 @@
// Loop through each selected file and attach them to the mail
foreach ($num as $key => $val) {
// If the message is of rfc822 type then set it as Quoted printable
- if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+ if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
$mime->addAttachment($list[$val]['path'], 'message/rfc822', 'Original Message', true, '');
} else {
// Default is base64 encoded
@@ -3492,8 +3492,11 @@
// Use sendmail to release message
// We can only release message/rfc822 files in this way.
$cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
+ if (count($num) < count($list)) {
+ $num = array_keys($list);
+ }
foreach ($num as $key => $val) {
- if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
+ if (preg_match('/message\/rfc822|text\/x-mail/', $list[$val]['type'])) {
debug($cmd . $list[$val]['path']);
exec($cmd . $list[$val]['path'] . ' 2>&1', $output_array, $retval);
if ($retval === 0) {
Is there a standard or similar for the structure of text/x-mail? I could only find a reference to "x-..." prefix in general referencing unstandardized protocolls.
@Skywalker-11 I did not dig so deep, but see http://lists.roundcube.net/pipermail/dev/2013-January/022097.html
Ok, while writing this message I have found out what is a cause. On CentOS 6.10 there is file-libs
package version 5.04. I got source from https://www.darwinsys.com/file/ and dig in inside:
$ grep -nrR 'x-mail' file-5.04
file-5.04/src/names.h:70: { "mail", "text/x-mail" },
As expected file
returns:
$ file -bi text_x-mail.message.txt
text/x-mail; charset=us-ascii
I have posted an example email message here: https://pastebin.com/2LEhbzhz
Steps to reproduce:
@Skywalker-11 Here is a real cause, compare this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3498 and this: https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3516-L3517
On the detail page of the message you see message/rfc822
because of this:
https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L380
https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L503
https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L524-L533
Conclusion. Depending on if a message is blocked or not quarantine_list_items()
returns either hard-coded value message/rfc822
, either what file -bi FILE
has returned.
UPDATE
Easiest workaround instead of fixing all regular expressions with message/rfc822
is below. Changes:
text/x-mail
as an alias for message/rfc822
in quarantine_list_items()
function.quarantine_release()
to look all extracted files within a message. If the [mime] type of a file is message/rfc822
then release the message. Tested on only when QUARANTINE_USE_SENDMAIL
is true (see conf.php
for details).quarantine_learn()
to look all extracted files within a message. If the [mime] type of a file is message/rfc822
then pass the path of the file to spamassassin
, e.g. complete message and not extracted from the message attachments.--- /var/www/html/mailscanner/functions.php.orig 2019-07-15 13:01:13.838325817 +0300
+++ /var/www/html/mailscanner/functions.php 2019-07-16 16:55:31.642773396 +0300
@@ -3394,7 +3394,11 @@
$quarantined[$count]['to'] = $row->to_address;
$quarantined[$count]['file'] = $f;
$file = escapeshellarg($quarantine . '/' . $f);
- $quarantined[$count]['type'] = ltrim(rtrim(`/usr/bin/file -bi $file`));
+ $type = ltrim(rtrim(`/usr/bin/file -bi $file`));
+ if (preg_match('!^text/x-mail!', $type)) {
+ $type = 'message/rfc822';
+ }
+ $quarantined[$count]['type'] = $type;
$quarantined[$count]['path'] = $quarantine . '/' . $f;
$quarantined[$count]['md5'] = md5($quarantine . '/' . $f);
$quarantined[$count]['dangerous'] = $infected;
@@ -3492,6 +3496,9 @@
// Use sendmail to release message
// We can only release message/rfc822 files in this way.
$cmd = QUARANTINE_SENDMAIL_PATH . ' -i -f ' . MAILWATCH_FROM_ADDR . ' ' . escapeshellarg($to) . ' < ';
+ if (count($num) < count($list)) {
+ $num = array_keys($list);
+ }
foreach ($num as $key => $val) {
if (preg_match('/message\/rfc822/', $list[$val]['type'])) {
debug($cmd . $list[$val]['path']);
@@ -3566,7 +3573,14 @@
if (!$rpc_only && is_local($list[0]['host'])) {
//prevent sa-learn process blocking complete apache server
session_write_close();
+ if (count($num) < count($list)) {
+ $num = array_keys($list);
+ }
foreach ($num as $key => $val) {
+ if (!preg_match('!message/rfc822!', $list[$val]['type'])) {
+ continue;
+ }
+
$use_spamassassin = false;
$isfn = '0';
$isfp = '0';
Have had exactly the same problem with mails encoded text/plain; charset=us-ascii in mailwatch v1.2.10. Changing the two lines initially explained to:
(detail.php)
) && preg_match('!message/rfc822|text/plain!', $item['type'])
(function.php)
if (preg_match('/message\/rfc822|text\/plain/', $list[$val]['type'])) {
and change conf.php to
define('QUARANTINE_USE_SENDMAIL', true);
(usually i am using "false") then the release works.
But i am using usually QUARANTINE_USE_SENDMAIL false. This way the attachement called "message" is in a corrupt format, cant open it with any app (ok with atom f.e. but the content is not read-able by outlook. in a text editor atom i see the mail content is base64 encoded -.-
Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64)
Today i've had the same issue with another mail release again. still no fix published ? :-(
To add to this issue, I had to change detail.php and functions.php to include another type.
detail.php:
) && preg_match('!message/rfc822|text/plain|text/x-mail|text/html!', $item['type'])
functions.php:
if (preg_match('/^(message\/rfc822|text\/plain|text\/x-mail|text\/html)/', $list[$val]['type'])) {
Is there a more generic solution to this problem?
@jensd0e I don't understand why you would do this for text/plain and text/html. Both of these represent the the text and html mime parts of a multipart mime email message that should have mime type message/rfc822. Can you share with me some more details, perhaps screenshots of what you see in MailWatch?
Also, thank you for calling my attention to this old issue. I will work on a fix for text/x-mime, but I need to understand how you are ending up with a message with text/plain and text/html.
@shawniverson I'm so glad, that you responded to my problem :-) Now, I'm happy to help. First things first: PHP is far from things I do at all. Reading and finding the "error" took me some hours, since I didn't understood what was going at first. And looking at the code that decides, weather a mail is released or not was bringing me here. The messages in question had file types different from message/rfc822: text/plain and text/html.
Here are the details for one of the mails:
Why the other three attachments very excluded from the report, I don't know. Another mail I had to release had text/html as file type. If I can help to find the underlying cause and fix it, please let me know, how I can help.
@jensd0e Are you able to get me a screenshot of the quarantine file list in the details pane on one of these quarantined emails?
@shawniverson I send you a PM with screenshots. Just did a test and I'm able to reproduce this with any file type attached, as long as the file name triggers the mailscanner rule.
@jensd0e thank you for the screenshots. Here is why you are having some trouble.
So you have a DKIM-Signature: line ahead of your Received: header as the first line of the message file. This is fine, but the file command by default does not detect this as message/rfc822 . To fix this without having to recode mailwatch, add this line to /etc/magic:
0 string/t DKIM-Signature: RFC 822 mail text
!:mime message/rfc822
Thanks for your help. I really appreciate your support.
Issue summary
I have found 2 issues when working with
text/x-mail
email messages:text/x-mail
.text/x-mail
.I am not including steps to reproduce these bugs, instead see patches below. Hope this will help. I am not exactly sure that this is correct way to fix the issue.
https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/detail.php#L541
https://github.com/mailwatch/MailWatch/blob/a47b54993f8a7b174cb88fe65eeb0d2eb83e2960/mailscanner/functions.php#L3584
Installation
Version and method
Thank you!