Mail client IP marked with RCVD_IN_ZEN_LASTEXTERNAL and SHORTCIRCUIT

bjoernv commented 3 years ago

Currently spamassassin-dqs does not always ignores the mail client (sender) IP. Mail client IP addresses are often marked in blocklists like ZEN. May be the bug is triggered by special Received: header formats.

The following mail shows the problem. The senders IP 5.61.176.191 is found in ZEN (can be shown with "spamassassin --debug").

Date: Sun, 25 Apr 2021 22:45:33 +0200
Subject: test
From: <example@vodafone.com>
Content-Type: text/plain; charset=utf-8

test

The mail gets these flags from spamassassin:

X-Spam-Flag: YES
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=100.5 required=5.0 tests=RCVD_IN_ZEN_LASTEXTERNAL,
        SHORTCIRCUIT shortcircuit=spam autolearn=disabled version=3.4.6
X-Spam-Report: 
        *  100 SHORTCIRCUIT Not all rules were run, due to a shortcircuited
        *      rule
        *  0.5 RCVD_IN_ZEN_LASTEXTERNAL The last untrusted relay is listed in
        *      Spamhaus ZEN

ricalfieri commented 3 years ago

Hi,

this is not a bug in SpamAssassin or the plugin, but a misconfiguration on your side.

You need to configure your system to avoid RBLs for SMTP submissions.

bjoernv commented 3 years ago

this is not a bug in SpamAssassin or the plugin, but a misconfiguration on your side. You need to configure your system to avoid RBLs for SMTP submissions.

No, you are wrong here. The SMTP submission side is managed by Vodafone. Vodafone is a big telecommunication company like AT&T or German Telekom. I can not change their mailservers. Spamassassin is running on the receiver side.

This is the data flow for the example mail:

Thunderbird mailer (at home with IP 192.168.1.2] (unknown [5.61.176.191] --> SMTP submission with SMTP AUTH (Vodafone, no Spamassassin --> SMTP receiver (my company mail server with Spamassassin) or my local Linux desktop SMTP server (with Spamassassin, mail fetched by fetchmail - in this example)

ricalfieri commented 3 years ago

Please paste a complete sample on pastebin so I can take a look at it

bjoernv commented 3 years ago

Here is a complete sample

https://paste.opensuse.org/c24fe76b

This is the data flow:

Thunderbird mailer (at home with IP [192.168.1.2] (unknown [5.61.176.191])) From: bjoernv@arcor.de, To: bjoernv@arcor.de --> Vodafone (MX for arcor.de, smtp.vodafone.de) --> Fetchmail to local Sendmail (at home with IP 192.168.1.2)

ricalfieri commented 3 years ago

Thanks, I just ran spamassassin on my test server (SA 3.4.6 with latest plugin installed) and this is the result:

Content preview: test 1

Content analysis details: (-3.0 points, 5.0 required)

pts rule name description

-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [bjoernv[at]arcor.de]

I really think that there is some misconfiguration on your side since I cannot reproduce the error

bjoernv commented 3 years ago

pts rule name description

What do you mean?

ricalfieri commented 3 years ago

It's the standard output when running spamassassin -t

I'll close this issue as it is not a bug of the plugin. If you need help with spamassassin I suggest you to post your issues on the mailing list spamassassin-users: https://cwiki.apache.org/confluence/display/SPAMASSASSIN/MailingLists

bjoernv commented 3 years ago

Output of spamassassin -t is:

Pkte Regelname              Beschreibung
---- ---------------------- --------------------------------------------------
 100 SHORTCIRCUIT           Not all rules were run, due to a shortcircuited
                            rule
 0.5 RCVD_IN_ZEN_LASTEXTERNAL The last untrusted relay is listed in
                            Spamhaus ZEN

RCVD_IN_ZEN_LASTEXTERNAL is defined in /etc/mail/spamassassin/sh.cf from spamassassin-dqs. So only spamassassin-dqs users can see the issue.

ricalfieri commented 3 years ago

I know it is defined there, but it uses standard SA configuration parameters, it's not a lookup done by the plugin itself.

The only thing you can do is fix your configuration

bjoernv commented 3 years ago

Personally I doubt, that Spamassassin does not understand the Received: header of the Vodafone mail servers correctly. /usr/lib/perl5/vendor_perl/5.32.1/Mail/SpamAssassin/Message/Metadata/Received.pm (from Spamassasin base) has a lot of rules for mail servers from Gmail, AOL, etc. I will try to find the bug here. spamassassin-dqs needs to correctly distinguish between ESMTPSA (SMTP with authentication) Received: and other Received: headers and this does not work correctly for Vodafone.

ricalfieri commented 3 years ago

It's not the job of a rule to distinguish between submissions or email received from external servers.

Other than that, there are hundreds of customers running spamassassin-dqs and nobody complained about this issue, and since vodafone.de is a pretty big provider if there were a bug I bet we would have ton of reports.

bjoernv commented 3 years ago

Now I found the main reason for the problems. zendqs-lastexternal selects an IP using this code (I added some debugging statements).

/usr/lib/perl5/vendor_perl/5.32.1/Mail/SpamAssassin/Plugin/DNSEval.pm

    # use the external IP set, instead of the trusted set; the user may have
    # specified some third-party relays as trusted.  Also, don't use
    # @originating; those headers are added by a phase of relaying through
    # a server like Hotmail, which is not going to be in dialup lists anyway.
    dbg("dns: MY fullexternal IPs: ".join(", ", @fullexternal));
    @ips = $self->ip_list_uniq_and_strip_private(@fullexternal);
    if ($1 eq "lastexternal") {
      dbg("dns: MY ips IPs: ".join(", ", @ips));
      @ips = (defined $ips[0]) ? ($ips[0]) : ();
      dbg("dns: MY lastexternal IPs: ".join(", ", @ips));
    } else {
        pop @ips if (scalar @ips > 1);
    }

For the Vodafone example mail it only finds on external IP (5.61.176.191) in the following list: 10.110.1.51, 10.110.1.21, 10.2.0.38, 5.61.176.191

Unfortunately IP 5.61.176.191 is my dialup IP and is listed in ZEN. Shortcircuit makes Spamassassin fail this mail as Spam directly.

Vodafone has a mail header configuration issue. The following Vodafone header is wrong because it resolves the host smtp.vodafone.de with the private IP 10.2.0.38.

Received: from smtp.vodafone.de (smtpa07.fra-mediabeam.com [10.2.0.38]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id 6EC5B12153A for bjoernv@arcor.de; Tue, 27 Apr 2021 19:43:59 +0200 (CEST)

This is the corrected header

Received: from smtp.vodafone.de (smtp.vodafonemail.de [2.207.150.234]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id 6EC5B12153A for bjoernv@arcor.de; Tue, 27 Apr 2021 19:43:59 +0200 (CEST)

If I manually correct the Vodafone header (can be also done automatically, then Spamassassin selects the IP 2.207.150.234 with zendqs-lastexternal and the mail is not marked as Spam anymore

I found these solutions until Vodafone fixes their mailservers:

Comment out zendqs-lastexternal in /etc/mail/spamassassin/sh.cf from spamassassin-dqs or
Decrease the score for RCVD_IN_ZEN_LASTEXTERNAL and comment out RCVD_IN_ZEN_LASTEXTERNAL in the Mail::SpamAssassin::Plugin::Shortcircuit block in /etc/mail/spamassassin/sh.cf from spamassassin-dqs

spamhaus / spamassassin-dqs

Mail client IP marked with RCVD_IN_ZEN_LASTEXTERNAL and SHORTCIRCUIT #36