Yomi: send only some file types

nethesis / dev

Nethesis issue tracker

http://www.nethesis.it

4 stars 1 forks source link

Yomi: send only some file types #6031

Closed filippocarletti closed 3 years ago

filippocarletti commented 3 years ago

We send all files to the Yomi sandbox for analysis, except a list of unsupported types. Unfortunately, the sandbox has two issues with some files:

false positives
errors during scanning

These problems lead to delayed deliveries and increased costs. Also, debugging email delivery problems is complex, especially when the email has many attachments of different types.

Proposed solution

Send only a list of well-known file types to the sandbox
Adjust spam scoring using a simple formula proportional to sandbox score
Log all file names

YOMI score range is 0-1, the proposed formula is SPAM_SCORE=YOMI_SCORE*10-1.

nethbot commented 3 years ago

in 7.9.2009/nethesis-testing:

nethserver-yomi-1.1.2-1.4.gaf9043e.ns7.noarch.rpm x86_64

gsanchietti commented 3 years ago

Test case 1

Update an existing machine or install a new one
Install the package from nethesis-testing
Verify that rspamd sends to the sandbox only the files with mime type listed inside /usr/share/nethserver-yomi/mime_type_graylist

Test case 2

After test case 1
Verify that the following prop under yomi key has been deleted: VirusWeight, SuspiciousWeight, SkipMimeTypes, CleanWeight

Test case 3

After test case 1
Verify the name of the uploaded file is present inside /var/log/maillog

filippocarletti commented 3 years ago

I can't test all mime types, but I sent many different files, and the sandbox received the expected types.

~]# config show yomi
yomi=configuration
    Debug=disabled
    SuspiciousLevel=0.4
    Url=https://sb.nethesis.it/api
    VirusLevel=0.8
    status=enabled

maillog contains the filename, but it's repeated twice:

Jun 10 10:02:50 ns7-ent rspamd[4062]: <4ed400>; lua; yomi.lua:84: yomi: attachment documento4_773.xls: MIME type application/vnd.ms-excel, size: 244736 bytes
Jun 10 10:02:50 ns7-ent rspamd[4062]: <4ed400>; lua; yomi.lua:84: yomi: attachment documento4_773.xls has hash f5622d0d69b8717dba28989c4f3fe160ecf0f4f45c4c7d9a481e4780b8ea6ae3
Jun 10 10:02:52 ns7-ent rspamd[4062]: <4ed400>; lua; yomi.lua:84: yomi: attachment documento4_773.xls: MIME type application/vnd.ms-excel, size: 244736 bytes
Jun 10 10:02:52 ns7-ent rspamd[4062]: <4ed400>; lua; yomi.lua:84: yomi: attachment documento4_773.xls has hash f5622d0d69b8717dba28989c4f3fe160ecf0f4f45c4c7d9a481e4780b8ea6ae3

nethbot commented 3 years ago

in 7.9.2009/nethesis-testing:

nethserver-yomi-1.1.2-1.5.gafa3158.ns7.noarch.rpm x86_64

nethbot commented 3 years ago

in 7.9.2009/nethesis-testing:

nethserver-yomi-1.1.2-1.6.ga330788.ns7.noarch.rpm x86_64

nethbot commented 3 years ago

in 7.9.2009/nethesis-testing:

nethserver-yomi-1.1.2-1.7.g31dc166.ns7.noarch.rpm x86_64

nethbot commented 3 years ago

in 7.9.2009/nethesis-testing:

nethserver-yomi-1.1.2-1.8.g51abe5b.ns7.noarch.rpm x86_64

nethbot commented 3 years ago

in 7.9.2009/nethesis-testing:

nethserver-yomi-1.1.2-1.9.g160838e.ns7.noarch.rpm x86_64

nethbot commented 3 years ago

in 7.9.2009/nethesis-updates:

nethserver-yomi-2.0.0-1.ns7.noarch.rpm x86_64