friedPotat0 / Spam-Scores

Thunderbird add-on to display spam scores according to mail headers. Supports spam/ham score headers of Rspamd and SpamAssassin. The add-on adds a column with the overall spam score to the mail list view and shows details of any matched spam/ham rule.
https://addons.thunderbird.net/de/thunderbird/addon/spam-scores/
Other
42 stars 6 forks source link

Error parsing detail rules with content preview header #34

Closed friedPotat0 closed 2 years ago

friedPotat0 commented 2 years ago

I tested the new add-on version 1.3.1. Unfortunately, it isn't quite good yet. The error repeats, when the number 30 appears in Content Preview. Maybe, here's the problem?

Example 1: obraz

Source 1:

MIME-Version: 1.0
Content-Type: multipart/alternative;
    boundary="_=aspNetEmail=_4d1094a12e524faf8923828cde92cb72"
Precedence: bulk
List-Unsubscribe: <http://info.topdelivery.net.pl/appreg/panel/Redirect.aspx?link_id=4833DA5C-36E3-4DF7-9695-D7E616F146EF&mail_id=01a6f998-3ea6-497a-9954-568e7f737fc1&d=14A96185-339C-41B4-8E28-CB069D47DA14&cntct_id=DWMTLRQjAFkMe1tBaW5AQhN9FDcRdhRoAQhCel0QPSJCSkZ4&p1=FBNFExdQBDNLWxYpa1trICl6ZXRddFkOBWkWbWULFQhLCCggHXRfdC1BEh4ZRXRKAVwNdxIIOGkbIHh0AXRpURIYDDpjC0QOAg8EbBpOXVESKiMuKnUoOGQZUBVtCnZmS2YLWGpcHDJNZWVpGSZPGnAcDWV4GgEMeB5xdHl7Yh4%2fVw9WKHwGZARbCFFZenk%3d&p2=EHhWL0UAUkReXB4IcUVuVltnfFZefxJiUz4zdGIHM1tLcglXAWExBENWehEeTx5fAR0mMkxZH0MsAgk8NjQyCF5CRhh4VhBlHhMZaExOCVsfVk1eDi1fS3R%2bUkd1AHgNWxUKVmEKbxVPFX88TFIiExppemdsFA4BfgoG&site=aHR0cCUzYSUyZiUyZmluZm8udG9wZGVsaXZlcnkubmV0LnBsJTJmYXBwcmVnJTJmcGFuZWwlMmZSZWdpc3Rlck91dFBhZ2UuYXNweCUzZm1haWxfaWQlM2QlMjMlMjNtYWlsX2lkJTIzJTIzJTI2YW1wJTNiZCUzZDE0QTk2MTg1LTMzOUMtNDFCNC04RTI4LUNCMDY5RDQ3REExNA%3d%3d>
Feedback-ID: :53537:65841:net.pl
X-Sid: 20211003.150020.1495@topdelivery.net.pl
Message-ID: <14x65841.53537.1159794393@info.topdelivery.net.pl>
X-Spam-Subject: ***SPAM***  =?utf-8?B?TmllIHByemVnYXAhIE5vd2/Fm2NpIC0yNSUgeiBrb2RlbSBGUkVTSDI1IQ==?=
X-Spam-Status: Yes, score=13.3
X-Spam-Score: 133
X-Spam-Bar: +++++++++++++
X-Spam-Report: Spam detection software, running on the system "xxxxx",
 has identified this incoming email as possible spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 root\@localhost for details.
 Content preview:  SprawdĹşsporstylestory_mailing-sale DARMOWA DOSTAWA OD 200
    PLN Â Â  / Â Â  30 DNI NA ZWROT Â Â  / Â Â  BEZPIECZNE ZAKUPY Â Â  / Â Â 
    RATY 0%Kampania realizowana przez Redgroup do bazy partnera Kampani [...]

 Content analysis details:   (13.3 points, 4.0 required)
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  4.5 URIBL_DBL_SPAM         Contains a spam URL listed in the Spamhaus DBL
                             blocklist
                             [URIs: topdelivery.net.pl]
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                             [score: 0.0000]
  5.0 URIBL_BLACK            Contains an URL listed in the URIBL blacklist
                             [URIs: topdelivery.net.pl]
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level
                             mail domains are different
  0.0 HTML_FONT_LOW_CONTRAST BODY: HTML font color similar or
                             identical to background
  0.0 HTML_IMAGE_RATIO_02    BODY: HTML has a low ratio of text to image
                             area
  0.0 HTML_MESSAGE           BODY: HTML included in message
  0.5 KAM_REALLYHUGEIMGSRC   RAW: Spam with image tags with ridiculously
                              huge http urls
 -0.1 DKIM_VALID             Message has at least one valid DKIM or DK signature
  0.1 DKIM_SIGNED            Message has a DKIM or DK signature, not necessarily
                             valid
 -0.1 DKIM_VALID_EF          Message has a valid DKIM or DK signature from
                             envelope-from domain
  5.0 KAM_VERY_BLACK_DBL     Email that hits both URIBL Black and Spamhaus
                              DBL
X-Spam-Flag: YES

Example 2: obraz

Source 2:

MIME-Version: 1.0
Content-Type: multipart/alternative;
    boundary="_=aspNetEmail=_4d1094a12e524faf8923828cde92cb72"
Precedence: bulk
List-Unsubscribe: <http://info.topdelivery.net.pl/appreg/panel/Redirect.aspx?link_id=4833DA5C-36E3-4DF7-9695-D7E616F146EF&mail_id=01a6f998-3ea6-497a-9954-568e7f737fc1&d=14A96185-339C-41B4-8E28-CB069D47DA14&cntct_id=DWMTLRQjAFkMe1tBaW5AQhN9FDcRdhRoAQhCel0QPSJCSkZ4&p1=FBNFExdQBDNLWxYpa1trICl6ZXRddFkOBWkWbWULFQhLCCggHXRfdC1BEh4ZRXRKAVwNdxIIOGkbIHh0AXRpURIYDDpjC0QOAg8EbBpOXVESKiMuKnUoOGQZUBVtCnZmS2YLWGpcHDJNZWVpGSZPGnAcDWV4GgEMeB5xdHl7Yh4%2fVw9WKHwGZARbCFFZenk%3d&p2=EHhWL0UAUkReXB4IcUVuVltnfFZefxJiUz4zdGIHM1tLcglXAWExBENWehEeTx5fAR0mMkxZH0MsAgk8NjQyCF5CRhh4VhBlHhMZaExOCVsfVk1eDi1fS3R%2bUkd1AHgNWxUKVmEKbxVPFX88TFIiExppemdsFA4BfgoG&site=aHR0cCUzYSUyZiUyZmluZm8udG9wZGVsaXZlcnkubmV0LnBsJTJmYXBwcmVnJTJmcGFuZWwlMmZSZWdpc3Rlck91dFBhZ2UuYXNweCUzZm1haWxfaWQlM2QlMjMlMjNtYWlsX2lkJTIzJTIzJTI2YW1wJTNiZCUzZDE0QTk2MTg1LTMzOUMtNDFCNC04RTI4LUNCMDY5RDQ3REExNA%3d%3d>
Feedback-ID: :53537:65841:net.pl
X-Sid: 20211003.150020.1495@topdelivery.net.pl
Message-ID: <14x65841.53537.1159794393@info.topdelivery.net.pl>
X-Spam-Subject: ***SPAM***  =?utf-8?B?TmllIHByemVnYXAhIE5vd2/Fm2NpIC0yNSUgeiBrb2RlbSBGUkVTSDI1IQ==?=
X-Spam-Status: Yes, score=13.3
X-Spam-Score: 133
X-Spam-Bar: +++++++++++++
X-Spam-Report: Spam detection software, running on the system "xxxxx",
 has identified this incoming email as possible spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 root\@localhost for details.
 Content preview:  SprawdĹşsporstylestory_mailing-sale DARMOWA DOSTAWA OD 200
    PLN Â Â  / Â Â  30 DNI NA ZWROT Â Â  / Â Â  BEZPIECZNE ZAKUPY Â Â  / Â Â 
    RATY 0%Kampania realizowana przez Redgroup do bazy partnera Kampani [...]

 Content analysis details:   (13.3 points, 4.0 required)
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  4.5 URIBL_DBL_SPAM         Contains a spam URL listed in the Spamhaus DBL
                             blocklist
                             [URIs: topdelivery.net.pl]
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                             [score: 0.0000]
  5.0 URIBL_BLACK            Contains an URL listed in the URIBL blacklist
                             [URIs: topdelivery.net.pl]
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level
                             mail domains are different
  0.0 HTML_FONT_LOW_CONTRAST BODY: HTML font color similar or
                             identical to background
  0.0 HTML_IMAGE_RATIO_02    BODY: HTML has a low ratio of text to image
                             area
  0.0 HTML_MESSAGE           BODY: HTML included in message
  0.5 KAM_REALLYHUGEIMGSRC   RAW: Spam with image tags with ridiculously
                              huge http urls
 -0.1 DKIM_VALID             Message has at least one valid DKIM or DK signature
  0.1 DKIM_SIGNED            Message has a DKIM or DK signature, not necessarily
                             valid
 -0.1 DKIM_VALID_EF          Message has a valid DKIM or DK signature from
                             envelope-from domain
  5.0 KAM_VERY_BLACK_DBL     Email that hits both URIBL Black and Spamhaus
                              DBL
X-Spam-Flag: YES

Originally posted by @MXEH in https://github.com/friedPotat0/Spam-Scores/issues/33#issuecomment-933195315

friedPotat0 commented 2 years ago

I will check how to avoid the parsing the content preview part of the X-Spam-Report header. Otherwise, it will always be possible to send a mail with content falsely recognised as a spam rule.

MXEH commented 2 years ago

Another example.

obraz

MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Subject: ***SPAM***  =?UTF-8?Q?Limit_w_rachunku_do_500_000_PLN_z_por=C4=99czeniem_UE_bez_ZUS,_?=
 =?UTF-8?Q?US.?=
X-Spam-Status: Yes, score=14.1
X-Spam-Score: 141
X-Spam-Bar: ++++++++++++++
X-Spam-Report: Spam detection software, running on the system "xxxxx",
 has identified this incoming email as possible spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 root\@localhost for details.
 Content preview:  Witam serdecznie, zwracam się z zapytaniem o możliwość
    zaprezentowania informacji nt kredytów bankowych dla firm – obrotowych,
    inwestycyjnych do 500 000 PLN bez zabezpieczeń. Jeśli temat finansowania
    jest dla Państwa interesujący prosimy o odpowiedź "TAK" – prześlemy
    materiały. Możecie Państwo również podać swój numer telefonu - oddzwonię.

 Content analysis details:   (14.1 points, 4.0 required)
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                             [score: 0.0000]
  4.5 URIBL_DBL_SPAM         Contains a spam URL listed in the Spamhaus DBL
                             blocklist
                             [URIs: inwestycyjne-finansowanie.com.pl]
  5.0 URIBL_BLACK            Contains an URL listed in the URIBL blacklist
                             [URIs: inwestycyjne-finansowanie.com.pl]
 -0.0 SPF_PASS               SPF: sender matches SPF record
  1.7 FUZZY_CREDIT           BODY: Attempt to obfuscate words in spam
 -0.1 DKIM_VALID             Message has at least one valid DKIM or DK signature
 -0.1 DKIM_VALID_AU          Message has a valid DKIM or DK signature from
                             author's domain
  0.1 DKIM_SIGNED            Message has a DKIM or DK signature, not necessarily
                             valid
 -0.1 DKIM_VALID_EF          Message has a valid DKIM or DK signature from
                             envelope-from domain
  5.0 KAM_VERY_BLACK_DBL     Email that hits both URIBL Black and Spamhaus
                              DBL
X-Spam-Flag: YES
MXEH commented 2 years ago

Another example for better problem analysis. In this case, retrieved two values from Content preview.

obraz

MIME-Version: 1.0
Content-Type: multipart/alternative;
    boundary="_=aspNetEmail=_6f2eb5506cca4b0ba0de63e9c639793c"
Precedence: bulk
List-Unsubscribe: <http://info.topdelivery.net.pl/appreg/panel/Redirect.aspx?link_id=7814AF6C-D0F9-402F-85B3-7793A032D29A&mail_id=58dbd2ed-bb95-4bf3-ac7d-842612654997&d=14A96185-339C-41B4-8E28-CB069D47DA14&cntct_id=DWMTLRQjAFkMe1tBaW5AQhN9FDcRdhRoAQhCel0QPSJCSkZ4&p1=FBNFExdQBDNLWxYpa1trICl6ZXRddFkOBWkWbWULFQhLCCggHXRfdC1BEh4ZRXRKAVwNcxB0OGlvUHB0AHAYJBIYAT1mC0QNfH8EHRw7ViZkWiFdWQ5YOGQZUBJoD3dmS2YIWGhbGTVOZWUeHyVFGnBoeRR4GwJ3fh4CdAV%2bEhhKVH9SVHsGZARbCVRWeX8%3d&p2=EHhWL0UAUkReXB4IcUVuVltnfFZefxJiUz4zdGIHM1tLcglXAWExBENWehEeTx5fAR0mMkxZH0MsAgk8NjQyCF5CRhh4VhBlHhMcYUkaC1BDCk0PCXVcS3QlAxV1WCIPCxUHVGtZaUFOE3xjFlQiExppemtiGgACfwoN&site=aHR0cCUzYSUyZiUyZmluZm8udG9wZGVsaXZlcnkubmV0LnBsJTJmYXBwcmVnJTJmcGFuZWwlMmZSZWdpc3Rlck91dFBhZ2UuYXNweCUzZm1haWxfaWQlM2QlMjMlMjNtYWlsX2lkJTIzJTIzJTI2YW1wJTNiZCUzZDE0QTk2MTg1LTMzOUMtNDFCNC04RTI4LUNCMDY5RDQ3REExNA%3d%3d>
Feedback-ID: :54066:65841:net.pl
X-Sid: 20211011.150020.3132@topdelivery.net.pl
Message-ID: <14x65841.54066.1197977298@info.topdelivery.net.pl>
X-Spam-Subject: ***SPAM***  =?utf-8?B?Qm9ueSB3YXJ0b8WbY2lvd2UgMTAwesWCIGR6aXNpYWogZGxhIENpZWJpZQ==?=
X-Spam-Status: Yes, score=13.4
X-Spam-Score: 134
X-Spam-Bar: +++++++++++++
X-Spam-Report: Spam detection software, running on the system "xxxxx",
 has identified this incoming email as possible spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 root\@localhost for details.
 Content preview:  Bony wartoĹ?ciowe 100zĹ? dzisiaj dla Ciebie BON WARTOĹ?CIOWY
    50 ZŁ (NR 15) NA DZISIEJSZE ZAKUPY W BINGOSPA.EU BON WARTOĹ?CIOWY 50 ZŁ
    (NR 16) NA NASTĘPNE ZAKUPY Kampania realizowana przez Redgroup do bazy partnera
    Kampanie Online LTD, na zlecenie: PrzedsiÄ?biorstwo Prywatne IMPEX P. Grabowski,
    M. Szpakowski [...] 
 Content analysis details:   (13.4 points, 4.0 required)
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  4.5 URIBL_DBL_SPAM         Contains a spam URL listed in the Spamhaus DBL
                             blocklist
                             [URIs: topdelivery.net.pl]
  5.0 URIBL_BLACK            Contains an URL listed in the URIBL blacklist
                             [URIs: topdelivery.net.pl]
 -1.9 BAYES_00               BODY: Bayes spam probability is 0 to 1%
                             [score: 0.0000]
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level
                             mail domains are different
  0.0 HTML_MESSAGE           BODY: HTML included in message
  0.2 KAM_TRACKIMAGE         RAW: Message has a remote image explicitly meant
                             for tracking
  0.5 KAM_REALLYHUGEIMGSRC   RAW: Spam with image tags with ridiculously
                              huge http urls
  0.1 DKIM_SIGNED            Message has a DKIM or DK signature, not necessarily
                             valid
 -0.1 DKIM_VALID             Message has at least one valid DKIM or DK signature
 -0.1 DKIM_VALID_EF          Message has a valid DKIM or DK signature from
                             envelope-from domain
  5.0 KAM_VERY_BLACK_DBL     Email that hits both URIBL Black and Spamhaus
                              DBL
X-Spam-Flag: YES
MXEH commented 2 years ago

Problem not solved?

friedPotat0 commented 2 years ago

Sorry that I haven't had the time to close the open issues yet. The problem will be fixed in the next update, which will definitely be released until the end of next week.

friedPotat0 commented 2 years ago

Should be fixed as a part of the merge request c1ca2a96294c765782687acd603b0c85bbb3b78f. It will be uploaded as a new version in the official Thunderbird add-on store this week.

image