search

sdgathman / pymilter

Python bindings for libmilter api
GNU General Public License v2.0
41 stars 21 forks source link

Virus detected in tgz by many scan engines! #33

Open chillout2k opened 5 years ago

chillout2k commented 5 years ago

https://www.virustotal.com/gui/file/f3580dd12c961cc46b77c90ec1fa4b9f73af15b4b340c44efc008bd30edeecae/detection

Maybe it´s due to the included test cases?

chillout2k commented 5 years ago

Source link: https://files.pythonhosted.org/packages/ca/b0/0e314563fc802cd7f8f98c858acf5def0ba85acc5fb2cef6db83d1b70431/pymilter-1.0.4.tar.gz

chillout2k commented 5 years ago

VT file relations: https://www.virustotal.com/gui/file/f3580dd12c961cc46b77c90ec1fa4b9f73af15b4b340c44efc008bd30edeecae/relations

It seems to be the test cases... I love this module, but, unfortunately, in this state the package is not ready for production in our environment :-/

Please fix this!!! Thanx ;)

kitterma commented 5 years ago

It is in the test cases. It's not actually a virus. There's nothing to worry about.

That said, this periodically comes up in Debian too where someone is worried if this is a virus. Maybe is more trouble than it's worth.

chillout2k commented 5 years ago

Maybe... but, we´ve got a restrictive compliance process (which I fully agree to)... Nevertheless, if 22 of 60 virus scanners complain, so this is IMO a case and I don´t want to downplay it.

From my point of view it would be more clear to externalize those "alarming/suspicious" tests: download them if needed while building...

Am 09.07.19 um 13:55 schrieb Scott Kitterman:

It is in the test cases. It's not actually a virus. There's nothing to worry about.

That said, this periodically comes up in Debian too where someone is worried if this is a virus. Maybe is more trouble than it's worth.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sdgathman/pymilter/issues/33?email_source=notifications&email_token=AKKO5ZRPTJHGELD3TWMFK7LP6R4ELA5CNFSM4H7EKKJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZQBB4Y#issuecomment-509612275, or mute the thread https://github.com/notifications/unsubscribe-auth/AKKO5ZWJGEPSJUVK7B2KOIDP6R4ELANCNFSM4H7EKKJA.

chillout2k commented 5 years ago

counter-question: why do you need suspicious virus tests packed together with a generic libmilter wrapper?

sdgathman commented 5 years ago

The main requirement of the test cases is that the headers be exactly as sent by the virus. I collected these because they are malformed and exploit bugs in header processing to deliver the payload.

The only modification I made to the original virus is to the payload (deleting all but first line). If that is what the engines are triggering on, then I think that any valid base64 data will suffice for testing. BUT, in some cases the virus relied on invalid termination of the base64 data, so we need to keep that intact as well.

sdgathman commented 5 years ago

Most people do not use the bare wrapper (milter module), but use the Milter.Base class instead, with the OO and decorator framework. These need to reasonably handle malformed emails.

chillout2k commented 5 years ago

The problem is, that we need to use pip3 to install the module as well as "all" dependencies. I use the code in the documented way:

  1. pip3 install pymilter
  2. import Milter...

Please see one of my projects. In private context "virus complains" are no problem, but in enterprise context it becomes a policy/compliance problem.

sdgathman commented 5 years ago

Ok, so can we find out which denatured viruses the programs are triggering on? I'm guessing that replacing the first line of the base64 payload with the standard "fake virus" signature (and keeping the last line in case of any malformed ending) may solve the problem.

sdgathman commented 5 years ago

I think I named the denatured viruses virusXX. Many of them have cleartext vbs script payloads. I'll start with a patch to the base64 payloads.

sdgathman commented 5 years ago

Do we even need EICAR? pymilter and libraries is not concerned with scanning for signatures, but just robustly parsing screwy headers.

chillout2k commented 5 years ago

Some of the scanners trigger on the presence of iframes. Others detected the love letter as well as VBA exploit

chillout2k commented 5 years ago

Base64 could be detected as obfuscation of VBA as well... If you really need that suspicious stuff in the repo/distro, a dirty workaround could be to encrypt the suspicious payload of the test mails with a hardcoded key and decrypt them on-the-fly while running the tests. In this case, on demand scanners will not detect it while scanning every 24 hours across the hard disk...

sdgathman commented 5 years ago

Heh. That is what more advanced viruses do. Encrypt the payload with a hardcoded key - which is different every time it infects a new victim.

chillout2k commented 5 years ago

@sdgathman if you only want to parse headers, why do you provide suspicious payload (body)? Further, if you add eicar, all scanners will trigger ;)

chillout2k commented 5 years ago

It's just a dirty hack ;)

chillout2k commented 5 years ago

IMO the cleanest way would be to replace the suspicious body-stuff with some harmless content, as your header tests do not rely on it... Finally you want to pass virus scans, right!? If you do so, my (enterprise) build pipeline will pass too ;-) Thank you very much for the library, great peace of software!

sdgathman commented 5 years ago

I went with your dirty hack. I pushed a commit to move the viruses to an encrypted zip file with password "denatured". Now I have to fix the unit testing machinery.

sdgathman commented 5 years ago

Can you check that a tar of that commit is "clean" according to your signature scanners? https://github.com/sdgathman/pymilter/commit/edc2f7337564ceec5cb5f998336707d8df28b81f

chillout2k commented 5 years ago

https://www.virustotal.com/gui/url/aa0d8e252020dae6038029a2b2e27846ac318a9558cbd70365420702fdb97018/detection

chillout2k commented 5 years ago

No surprise... Looks good! You can do it too with virustotal.com ;-)

sdgathman commented 5 years ago

I meant the entire tarball, since some engines may be triggering on the spam as well. But I can run it myself.

sdgathman commented 5 years ago

Ok, the full tarball checks out, so I did keep the actual viruses properly named. I went with the quick and dirty approach because I remember spending a lot of time denaturing the viruses without spoiling the test of parsing loopholes. Sometimes the size of a base64 attachment was critical, sometimes a malformed ending. I don't remember enough details of all the tricks to safely further denature the viruses without spoiling the tests.

As of that commit, the unit testing is broken. As soon as it is working again, I can do a new release.

sdgathman commented 5 years ago

If I add any more test cases, it is definitely best practice to denature it until it passes virustotal, but still triggers the bug or loophole.

chillout2k commented 5 years ago

Thanx man, good luck! Let me know if you need help!

sdgathman commented 5 years ago

@chillout2k Commit a1714f4838786b0359677dbc7c62d3dc647a6b19 fixes the test cases to extract from the zipfile.

DaedalusEncounter commented 1 year ago

Can you revisit separating these malware tests outside pymilter? AMP is still flagging this and causing needless security cases.

pymilter-1.0.5.tar.gz (78f1c55a…2a2d804d)[Mail] was Created by bash[common filename] (00f8cbc5…a78c92e9)[ELF Executable] .
Detected as Html.Exploit.Iframe::95.sbx.tg.

Detected inner file (0E8A0C6B…989E9E97)[Text (ASCII)] within pymilter-1.0.5.tar.gz (2fac54d5…6e460c3f)[GZip] as Auto.2FAC54.251641.in02.

https://www.virustotal.com/gui/file/2fac54d5b5ccca87786ad25eb0057a40bee6d5d3f71d7a6d5fe495ab6e460c3f/detection

Shows 22 scanners flagging it as malicious. Looks like the top files triggering the alerts are

2023-07-12 16 / 59 Email pymilter-1.0.5/test/honey 2020-08-01 5 / 59 Email pymilter-1.0.5/test/zip1 2020-08-01 5 / 59 Email pymilter-1.0.5/test/ziploop 2023-08-18 1 / 48 JavaScript pymilter-1.0.5/pymilter.egg-info/dependency_links.txt

sdgathman commented 1 year ago

Sigh. I guess the entire test directory needs to be moved to the encrypted zip.

DaedalusEncounter commented 1 year ago

Thanks for re-opening this issue.