Spamassassin Concepts

Spamassassin Concepts is a plug in that canonicalises emails into their basic concepts. These concepts are then available to Bayes to work with as well as being available via meta scoring. It offers a simple alternative to the native SA meta rules, by being easier to write and understand.

SA plugin adapted from http://wiki.junkemailfilter.com/index.php/Concept_Parsing_Spam_Filter

Installation

• Download zip / Clone repo
• Move to contents to spamassassin location
  • Usually /etc/mail/spamassassin or /etc/spamassassin
• Update Concepts.cf with correct path to the concepts directory
• Do the test as below
• Make sure Bayes is enabled
• Restart Spamassassin

Testing Installation

With a test email run spamassassin in debug mode, searching for the Concepts keyword

$ spamassassin -D -t testemail 2>&1 | grep Concepts
...
[12201] dbg: plugin: loading Mail::SpamAssassin::Plugin::Concepts from /etc/spamassassin/Concepts.pm
...
[12201] dbg: Concepts: 256 concepts loaded

This means the plugin has loaded.

You should also get an output with a digest of the concepts gained from your example email

[12201] dbg: Concepts: metadata: X-SA-Concepts: service madam dear stranger invest sir reply2me please email-adr

For problems please follow the error messages

Usage

In day to day life you do not need to interact with Concepts - Bayes will automatically keep up to date with what is considered good and bad as long as you classify your spam and ham carefully.

Be careful with manual scoring!!! By design concepts can hit both ham and spam emails - including combinations you couldn't imagine!

If you would like to interact with Concepts you can in the form of meta rules to give matching email additional weighting

## JUST AN EXAMPLE!!!
header __CONCEPTS_GOD X-SA-Concepts =~ /\bgod\b/
header __CONCEPTS_AFRICA X-SA-Concepts =~ /\bafrica\b/
header __CONCEPTS_LOTSMONEY X-SA-Concepts =~ /\blotsofmoney\b/
header __CONCEPTS_STRANGER X-SA-Concepts =~ /\bstranger\b/
header __CONCEPTS_DYING X-SA-Concepts =~ /\bdying\b/

#scores 4 or more
meta GOD_IN_AFRICA_LOTSAMONEY_STRANGERS_DYING (__CONCEPTS_GOD + __CONCEPTS_AFRICA + __CONCEPTS_LOTSMONEY + __CONCEPTS_STRANGER + __CONCEPTS_DYING >= 4)
describe GOD_IN_AFRICA_LOTSAMONEY_STRANGERS_DYING Combo
score GOD_IN_AFRICA_LOTSAMONEY_STRANGERS_DYING 1.0 

Example

Normally logs will not show, however by switching logging to debug mode you will see similar in your logs

May 22 23:12:22 spamfilter spamd[7459]: metadata: X-SA-Concepts: hotwords law sale https contact time-ref order privacy internet store apple report invoice all-rights email-adr price receipt
May 22 23:12:23 spamfilter spamd[1998]: metadata: X-SA-Concepts: friend law re optout important online time-ref all-rights please news home privacy
May 22 23:12:26 spamfilter spamd[3128]: metadata: X-SA-Concepts: https optout day-of-week
May 22 23:12:33 spamfilter spamd[1998]: metadata: X-SA-Concepts: friend https linkedin twitter america opportunity contact experience club search-eng optout winner time-ref facebook prize mailto re important online click all-rights email-adr please newsletter day-of-week partner
May 22 23:12:35 spamfilter spamd[3128]: metadata: X-SA-Concepts: time-ref https optout
May 22 23:12:38 spamfilter spamd[1998]: metadata: X-SA-Concepts: hotwords money optout discount online click news hot-adj
May 22 23:12:39 spamfilter spamd[1998]: metadata: X-SA-Concepts: alert service
May 22 23:12:43 spamfilter spamd[1998]: metadata: X-SA-Concepts: hotwords offer search-eng optout facebook details discount doitnow hello price watches https sale twitter sentfrom contact time-ref deal great home deals store score health camera click please email-adr
May 22 23:12:43 spamfilter spamd[3128]: metadata: X-SA-Concepts: hotwords law offer claim optout online time-ref click compensation best
May 22 23:12:44 spamfilter spamd[3129]: metadata: X-SA-Concepts: internet law voip time-ref click all-rights email-adr please
May 22 23:12:46 spamfilter spamd[3129]: metadata: X-SA-Concepts: https contact time-ref notice order email-adr day-of-week
May 22 23:12:47 spamfilter spamd[9513]: metadata: X-SA-Concepts: thankyou reply2me contact details online woman please
May 22 23:12:47 spamfilter spamd[10502]: metadata: X-SA-Concepts: friend google india linkedin boost america phone-num money search-eng optout time-ref home invest internet store apple member re important details email-adr please asian security news newsletter day-of-week
May 22 23:12:47 spamfilter spamd[7459]: metadata: X-SA-Concepts: offer search-eng optout facebook enjoy details discount doitnow hello price watches https sale twitter sentfrom contact time-ref deal woman great home deals store health camera click game please email-adr

Concept file format

Consider the following file named viagra

1
viagra
v[jl1]agra

The file is parsed in the following way.

• The concept name will be the file name. Please refrain from special characters here - alpha-numeric and -_ are fine.
• The first line of the file dictates how many matches within the file need to be true to mark the email as containing the concept name
• The rest of the file contains regex lines that are each wrapped internally with the following: /$regex/ig
• The body (and other parts) are check against the regexes specified.

What is checked

The email body is pre-stripped of all of it's HTML and other un-needed tags to become plain text. To that we add the subject as well as a few other header items.

This text is concatenated and then checked against the regex lines specified in concept files.

You can add additional header fields with the concepts_headers directive.

Change Log

Version 0.03

• Code optimisation

Version 0.02

• Bug fixes from SA user list - David Jones

Version 0.01

• Initial release
• 250 concept files - Thanks Marc Perkel
• Native tags and Bayes integration
• Native meta rules based on concepts