Scanning for URL in mails

[thegrillmaster]

Hello everybody, I have a question again :) Attachments are processed very well with PeekabooAV and Cuckoo. But we also have many mails with only one URL where malware can be downloaded. Is there a possibility to do this with Peekaboo? So that the whole mail is scanned? With the Peekaboo-Rules I put the following on the greylist: message/rfc822, but this causes false-positive, because in the sandbox instead of opening Outlook a CMD is executed. Does anyone have an idea how this could be implemented? Thanks for your input.

[michaelweiser]

Thanks for reaching out! Can you give a bit more detail what you're trying to achieve by scanning the email text as well with regards to the URL(s) in those mails? Do you expect the mail client to do something with that mail so that Cuckoo can detect potentially malicious actions? Or do you expect Cuckoo to also click on that URL visible in the mail client? That would be really cool, particularly in conjunction with inetsim, and Cuckoo already has a clicking feature in the agent.

We're tracking the feature of scanning the whole mail in #88 but haven't made much progress. We should continue discussion there if it's a distinct feature you're looking for and not just a possible implementation of a URL scanning feature.

[yoshimo]

Many administrators decided to block potentially malicious attachments like doc or docm, so malware campaigns give you a link to a page where the bill, the requested document or the application letter is stored. This bypasses the first layer of defense. So if we manage to download whatever file is behind this link without revealing being a malware scanner, we might be able to prevent troubles for users.

[michaelweiser]

Understood. But how would giving the email to Outlook in the sandbox trigger a download of the linked document?

Also: We not only do not want to reveal that we're a malware scanner. We also do not want to reveal which email addresses in our organisation are alive and which are not. Also we do not want to be trickable into doing denial of service attacks on web servers. There's also one-time links, e.g. for validating password resets or subscribing to newsletters - we certainly do not want to click on those.

One concept we've been toying with is a plugin for the email scanner (AMaViS, rspamd, ...) that extracts URLs from emails, places them in a database and replaces them with a warning and link to a web portal in the email. If the user decides that they really need this attachment, they can go to the portal and trigger the download, which will then be scanned by the portal for malware (maybe using Cuckoo) before delivering it back to the user. Solves above problems but breaks email signatures because it modifies them. Pointers to any existing software very welcome.

[thegrillmaster]

Thanks for your answers. For us, an extract of the URL from the mail would be sufficient, whether with Amavis, Peekaboo or Cuckoo, doesn't matter... we compare this extracted URL with IRMA or the Virustotal-Private-Api-Key for known harmfulness and that the mail can be classified as good or bad. To reduce the number of queries to Virustotal, a database where the known URLs are stored would be very helpful. Of course we also don't want Urls to be clicked in the mail. But an automated possibility to read URLs from mails and give them to compare IRMA or Virustotal, that would be a wonderful addition. I am very grateful for your ideas!

[michaelweiser]

Both rspamd[1,2] and spamassassin[3] have modules for extracting URLs from messages and checking them against DNS blocklists and even reputation. Architecturally they seem like the most natural place to do this kind of thing.

[1] https://rspamd.com/doc/modules/surbl.html [2] https://rspamd.com/doc/modules/url_reputation.html [3] https://spamassassin.apache.org/full/3.0.x/dist/doc/Mail_SpamAssassin_Plugin_URIDNSBL.html

[thegrillmaster]

We use Spamassassin with various modules before the sandbox to also check URLs against RBLs. The sandbox is at the end of the chain and for my feeling there are still too many mails with dangerous links.

[michaelweiser]

Well yes, RBLs do not provide threat intelligence, only domain reputation. What I meant was that such a module would provide a good starting point for extracting URLs from mails in AMaViS, rspamd or as a separate little virus scanner - if vendors of such threat intelligence don't provide their own plugins already. Architecturally it IMO belongs there, since it's 100% static analysis and Peekaboo is focusing on feeding samples into behavioural analysis.

If we were to add this kind of analytics to Peekaboo it would likely be to help us decide what to put into behavioural analysis (Cuckoo) and what not to or to augment behavioural analysis results to make them more conclusive (like Cuckoo does itself). Once added, it could of course also be used as sole classifier without any behavioural analysis but IMO we shouldn't add it just for that. I'd like to avoid reinventing the wheel there.

This brings me back to your original question how to have the whole email analysed: Have you had a look how Cuckoo's IRMA and VirusTotal modules could help you there? I can see some handling of URLs in the VirusTotal module code but have no experience with the functionality. Also, presumably, it would currently only get to see URLs extracted from attachment samples we submit to Cuckoo. I'd be interesting to know if this does get involved in static analysis of an rfc822 email sample already without the need of an actual behavioural analysis.

[yoshimo]

I am wondering if the place for "clicking" links and downloading the files behind it really is in Peekaboo.

It gets email attachments from AMaViSd, checks them, uses Cuckoo for behavioral checks, and evaluates and rates fully automatic

Wouldn't it be up to a plugin for AMaViSd/RSpamD to grab the files from behind the link and feed it back to the AV engine similiarly to extracting the attachments?

[michaelweiser]

I am wondering if the place for "clicking" links and downloading the files behind it really is in Peekaboo.

My current understanding is that @thegrillmaster is looking for a way to extract URLs from mails and that automatically downloading and analysing them is out of scope. I do not think that either functionality fits particularly well into Peekaboo, considering that other tools already do at least part of this. We certainly do not have any functionality to that effect currently.

Wouldn't it be up to a plugin for AMaViSd/RSpamD to grab the files from behind the link and feed it back to the AV engine similiarly to extracting the attachments?

Yes, if you really wanted to do that, that would be a way to do it. It could in principle also be part of behavioural analysis by opening the email in the email client and have the Cuckoo agent click on the link and let events take their course. For the reasons above (https://github.com/scVENUS/PeekabooAV/issues/144#issuecomment-595743260) I'd counsel to do neither and rather go with @thegrillmaster 's approach or the one outlined in the same comment.

[Jack28]

I close this for now feel free to reopen, comment and/or share your solution