Configuration not straight

[hotstepper13]

I appreciate the work that you have done with the project. I was forced to reinstall my very old debian server and now I use ubuntu. With this project I was able resetup my mail stack without much conversion regarding existing accounts. Thank you.

After it took me nearly a day to figure this out, I would to report this here as it may affect other, too.

I wanted to have the SPAM_REPORT hearder always in my mails, even if they are ham so i may look why spam was not identified.

Within vexim-acl-check-content.conf I have found the following configuration which tells me that it should do exactly what i want:

  # Always add X-Spam-Score and X-Spam-Report headers, using SA system-wide settings
  # (user "nobody"), no matter if over threshold or not.
  warn  message     = X-Spam-Score: $spam_score ($spam_bar)
        spam        = maildeliver:true
  warn  message     = X-Spam-Report: $spam_report
        spam        = maildeliver:true
  accept hosts      = 127.0.0.1:+relay_from_hosts
  accept authenticated  = *

However, non spam mails doesnt contain the header (used my corporate mail account to test it).

I found the following passage in the exim configuration example (docs/configure)

virtual_domains:
  driver = redirect
  domains = +local_domains
  address_data = ${lookup mysql{\
        select smtp, users.sa_tag*10 AS sa_tag, users.on_spamassassin AS on_spamassassin, \
        users.uid AS uid, users.gid AS gid, quota from users,domains \
        where localpart = '${quote_mysql:$local_part}' \
                and domain = '${quote_mysql:$domain}' \
                and domains.enabled = '1' \
                and users.enabled = '1' \
                and users.domain_id = domains.domain_id}{$value}fail}
  allow_fail
  data = ${extract{smtp}{$address_data}}
  headers_add = ${if and { \
                    {>{$spam_score_int}{${extract{sa_tag}{$address_data}}}} \
                    {eq{1}{${extract{on_spamassassin}{$address_data}}}} \
                    } {X-Spam-Flag: YES\n}{} }
  # Whether to use old or new headers_remove behavior.
  .ifndef OLD_HEADERS_REMOVE
    headers_remove = ${if or { { <{$spam_score_int}{1} } \
                              { and {{<{$spam_score_int}{${extract{sa_tag}{$address_data}}} } \
                                     {eq {1}{${extract{on_spamassassin}{$address_data}}}} \
                                    } \
                              } { eq {0}{${extract{on_spamassassin}{$address_data}}}} \
                            }  {X-Spam-Score}}
    headers_remove = ${if or { { <{$spam_score_int}{1} } \
                              { and {{<{$spam_score_int}{${extract{sa_tag}{$address_data}}} } \
                                     {eq {1}{${extract{on_spamassassin}{$address_data}}}} \
                                    } \
                              } { eq {0}{${extract{on_spamassassin}{$address_data}}}} \
                            }  {X-Spam-Report}}
  .else
    headers_remove = ${if or { { <{$spam_score_int}{1} } \
                              { and {{<{$spam_score_int}{${extract{sa_tag}{$address_data}}} } \
                                     {eq {1}{${extract{on_spamassassin}{$address_data}}}} \
                                    } \
                              } { eq {0}{${extract{on_spamassassin}{$address_data}}}} \
                            }  {X-Spam-Score:X-Spam-Report}}
  .endif
  .ifdef VEXIM_LOCALPART_SUFFIX
    local_part_suffix = VEXIM_LOCALPART_SUFFIX
    local_part_suffix_optional
  .endif
  retry_use_local_part
  file_transport = virtual_delivery
  reply_transport = address_reply
  pipe_transport = address_pipe

To archive my goal I have had to comment out the second passage in the OLD_HEADERS_REMOVE:

virtual_domains:
  driver = redirect
  domains = +local_domains
  address_data = ${lookup mysql{\
        select smtp, users.sa_tag*10 AS sa_tag, users.on_spamassassin AS on_spamassassin, \
        users.uid AS uid, users.gid AS gid, quota from users,domains \
        where localpart = '${quote_mysql:$local_part}' \
                and domain = '${quote_mysql:$domain}' \
                and domains.enabled = '1' \
                and users.enabled = '1' \
                and users.domain_id = domains.domain_id}{$value}fail}
  allow_fail
  data = ${extract{smtp}{$address_data}}
  headers_add = ${if and { \
                    {>{$spam_score_int}{${extract{sa_tag}{$address_data}}}} \
                    {eq{1}{${extract{on_spamassassin}{$address_data}}}} \
                    } {X-Spam-Flag: YES\n}{} }
  # Whether to use old or new headers_remove behavior.
  .ifndef OLD_HEADERS_REMOVE
    headers_remove = ${if or { { <{$spam_score_int}{1} } \
                              { and {{<{$spam_score_int}{${extract{sa_tag}{$address_data}}} } \
                                     {eq {1}{${extract{on_spamassassin}{$address_data}}}} \
                                    } \
                              } { eq {0}{${extract{on_spamassassin}{$address_data}}}} \
                            }  {X-Spam-Score}}
#    headers_remove = ${if or { { <{$spam_score_int}{1} } \
#                              { and {{<{$spam_score_int}{${extract{sa_tag}{$address_data}}} } \
#                                     {eq {1}{${extract{on_spamassassin}{$address_data}}}} \
#                                    } \
#                              } { eq {0}{${extract{on_spamassassin}{$address_data}}}} \
#                            }  {X-Spam-Report}}
  .else
    headers_remove = ${if or { { <{$spam_score_int}{1} } \
                              { and {{<{$spam_score_int}{${extract{sa_tag}{$address_data}}} } \
                                     {eq {1}{${extract{on_spamassassin}{$address_data}}}} \
                                    } \
                              } { eq {0}{${extract{on_spamassassin}{$address_data}}}} \
                            }  {X-Spam-Score:X-Spam-Report}}
  .endif
  # using local_part_suffixes enables possibility to use user-"something" localparts
  # which could cause you trouble if you're creating email-adresses with dashes in between.
  .ifdef VEXIM_LOCALPART_SUFFIX
    local_part_suffix = VEXIM_LOCALPART_SUFFIX
    local_part_suffix_optional
  .endif
  retry_use_local_part
  file_transport = virtual_delivery
  reply_transport = address_reply
  pipe_transport = address_pipe

Now i have the result i wanted. One additional note: While googling regarding this issue, i found many hints regarding amavis and to set the tag score to -99. The new tables doesnt support signed values and even after i changed the table shema, and set a tag score to -3 (my test messages got a 0) I didnt got the report in my headers.

I am not sure if that is intended or not so it is just reporting.

Best regards

Frank

[Udera]

You could have used the mailing list to ask instead of spending so much time on it :-) If you only want to disable it for certain accounts, you could just increase the spam_tag and spam_refuse points to high values (e.g. 99) so that the header is never removed.

I started a document with some optional configuration options. In some environments you might want to have a different setting, so I started a document with such options: https://github.com/vexim/vexim2/wiki/Optional-configurations-after-the-main-setup You could add your case.

Just one more question: Did you install everything from scratch? If you did some migration, did you use the migration script we are working on: https://github.com/vexim/vexim2/pull/72 ? Which version of Ubuntu did you use? It's just we want to release vexim2.3 soon, and we should test on a set of different environments. If you just installed it on Ubuntu x.y, we rather test it on a different version.

[hotstepper13]

Uh.. sorry. I think i made thing not clear. I didnt want to disable it, i wanted to ENABLE it for ALL accounts. The configuration in the check_content configuration let me assume that this is the intended behavior but the configuration regarding the headers remove has removed the header all the time. So even if a mail was considered as ham, i want the option to look why spamassasing think that this is ham and not spam.

I am using Ubuntu 14.04.4 and installed from scrath but with an existing vexim db. I am not sure which version I have used on my old server but I had to do some strange things in order to make everythin work. The migration perl wasnt working that well as i have had to drop a column in my old database and afterwards i have had to change uid,git and mailpath manually.

I used everything from master.

Also the eximconfiguration files are not really well prepared. The documentation stated to create a user called vexim (btw. in the past the user was called vmail which fits better in my opinion) and I followed the documentation. The exim configurations are sometimes working with different users (e.g. maildeliver).

I have had a lot of permission problems to deal with in order to allow everything access to the needed directories (especially clamav and Spamassassin). Because I have had many additional things to be aware of (like setting up a local dns server for dnsbl) I didnt know exactly where i have had to change the configuration to allow it working in the intended way.

But you already stated at the start of the documentation that this are instructions for someone who might have some knowledge about exim and how to deal with configuration. Therefore it might be my own problem as I didnt have that deep knowledge.

Regarding your additional options: I installed Rainloop as webmailer. The cool thing is that this will allow me access the spam filtered files as it works out of the box with the .spam folder. That is a nice feature to get rid of spam but be able to have a look if you missing some files (as long as you have enough discspace to store them).

But in my opinion there should be a hint that the installation of caching dns server (like bind) is very recommended as long as you want to use dnsbl´s like spamhaus. Btw. dsbl.org is gone and should be removed from the example configuration. Instead you could use ix.dnsbl.manitu.net for dnsbl. This is a list operated in germany by a small provider and a well known IT magazine.

I could try to do some changes but I am not aware how to do this. Can i just create a branch here, do my changes and create a pull request or do i have to fork the whole stuff?

Best regards Frank

[hotstepper13]

Ok... I think i got it. I started the creation of some pull requests to fix some painpoints I have had. Hopefully this fits into your strategy.

[Udera]

I am using Ubuntu 14.04.4 and installed from scrath but with an existing vexim db. I am not sure which version I have used on my old server but I had to do some strange things in order to make everythin work. The migration perl wasnt working that well as i have had to drop a column in my old database and afterwards i have had to change uid,git and mailpath manually.

We have modified the table slightly (removed/added some fields, changed size of fields). You can manually compare the database settings: https://github.com/vexim/vexim2/blob/master/setup/mysql.sql

Also the eximconfiguration files are not really well prepared. The documentation stated to create a user called vexim (btw. in the past the user was called vmail which fits better in my opinion) and I followed the documentation. The exim configurations are sometimes working with different users (e.g. maildeliver).

The original files fit the FreeBSD default users. Now we have a mixture of debian/FreeBSD default users. We should perhaps choose one as default and put the other into comments.

I have had a lot of permission problems to deal with in order to allow everything access to the needed directories (especially clamav and Spamassassin). Because I have had many additional things to be aware of (like setting up a local dns server for dnsbl) I didnt know exactly where i have had to change the configuration to allow it working in the intended way.

The spamassassin-part is unfortunately incomplete. We need to work over it. I'd like to add a part that you can use some folders to train the Bayesian spam filter.

[rimas-kudelis]

Regarding headers_remove: I thought we were only removing these headers when forwarding mail to other servers, but apparently, we're also removing them when the e-mail is considered ham. That's unexpected to me.

Does anyone think there would be any harm if we just dropped that headers_remove behavior and instead only removed these headers unconditionally when forwarding e-mail to other servers? That would not only simplify our config file (and let us get rid of the OLD_HEADERS_REMOVE variable, but also let people like Frank examine why their spam wasn't seen as such.

[Udera]

That's the reason we have two different levels for spamassassin: sa_tag and sa_remove. If you want to keep the header in, you can use sa_tag=0. The currently added spam-report is quite detailed, I wouldn't like to have it in each mail. For a spam mail of 17.7 kB, this spam-report is about 1.9 kB. If you add this to each mail ...

[rimas-kudelis]

Thing is, sa_tag also acts as a threshold to add the X-Spam-Flag: YES header, which marks the message as spam automatically, so if you set sa_tag to 0, you effectively mark all of your e-mails as spam (except the ones with negative points, if that's possible, but then these emails don't get the other headers either).

[Udera]

I'm not really seeing the purpose of the spamassassin-headers when the spam score is below the sa_tag-level. Could be perhaps interesting during training of the spam-filter and to identify problems when having false negatives. I didn't use this very often, in some cases I could lookup the spam-score of the mail in the mail-logs.

[rimas-kudelis]

I've just looked at a spam message in my Junk folder. Here are the example headers we're talking about:

X-Spam-Level: **
X-Spam-Status: No, score=2.5 required=5.0 tests=BAYES_50,
    FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FREEMAIL_REPLYTO_END_DIGIT,
    HEADER_FROM_DIFFERENT_DOMAINS,HK_RANDOM_ENVFROM,HTML_MESSAGE,MIME_HTML_ONLY,
    MISSING_MID autolearn=no autolearn_force=no version=3.4.0

or this:

X-Spam-Flag: YES
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.7 required=5.0 tests=BAYES_50,
    FREEMAIL_FORGED_REPLYTO,FROM_NO_USER,HTML_MESSAGE,MISSING_MID,SUBJ_ALL_CAPS
    autolearn=no autolearn_force=no version=3.4.0

I don't think these are big enough to be worthy of size concerns, and AFAIR size was the main reason for their removal, no? I understand we may not want to pass these headers on when sending e-mail to other servers (to avoid leaking our filters, and since the receiving server should test them for spam independently anyway), but for local delivery, I think they're OK to stay.

[Udera]

But we use the X-Spam-Report which looks like this:

X-Spam-Report: Spam detection software, running on the system "mail.example.com",
has identified this incoming email as possible spam.  The original
message has been attached to this so you can view it or label
similar future email.  If you have any questions, see
the administrator of that system for details.

Content preview:  Dear user, Please see attached the confirmation of transaction
   conducted from your account. Please sign it and forward the copy to us for
   approval. Best regards, [...] 

Content analysis details:   (10.9 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 3.6 RCVD_IN_PBL            RBL: Received via a relay in Spamhaus PBL
                            [130.25.222.171 listed in zen.spamhaus.org]
 1.6 RCVD_IN_BRBL_LASTEXT   RBL: No description available.
                            [130.25.222.171 listed in bb.barracudacentral.org]
 0.7 SPF_NEUTRAL            SPF: sender does not match SPF record (neutral)
 1.4 GENERIC_IXHASH         BODY: iXhash found @ generic.ixhash.net
 0.4 RDNS_DYNAMIC           Delivered to internal network by host with
                            dynamic-looking rDNS
 3.2 HELO_DYNAMIC_IPADDR    Relay HELO'd using suspicious hostname (IP addr
                            1)

For some mails, this might be longer than the text inside the mail itself.

[Udera]

There is a PR to work on the spamassassin-integration: https://github.com/vexim/vexim2/pull/203

[rimas-kudelis]

Just to explain my older comment, apparently, X-Spam-Level and X-Spam-Status are the headers added by SpamAssassin when checking messages, but Exim does NOT use these headers, at least when hooked with SpamAssassin the way we have it hooked. The reason I saw these headers in my e-mails was my use of sa-exim package, which integrates SA and Exim differently.

I've filed a feature request for Exim asking to support this thing, but this won't be available anytime soon, if ever.

Regarding @hotstepper13's worries, I think #203 will make this feature a bit more straightforward: if we go with that approach, headers will not be added at ACL time and later removed during delivery, but will instead only be added during delivery when necessary. While this doesn't mean they will be added to every message by default, it will hopefully make it easier to configure their addition.

[rimas-kudelis]

Apparently, adding the following two lines to /etc/spamassassin/local.cf allows to pretty much emulate the X-Spam-Status behavior by changing SpamAssassin report template. I think it might be worth the effort to at least document this:

clear_report_template

report _YESNO_, score=_SCORE_ required=_REQD_ tests=_TESTS_ autolearn=_AUTOLEARN_ version=_VERSION_

The only difference of this template from output above is that it won't be wrapped into multiple lines as is usual.