ET Telemetry ruleset clarifications

[0nnyx]

Important notices Our forum is located at https://forum.opnsense.org , please consider joining discussions there in stead of using GitHub for these matters.

Before you ask a new question, we ask you kindly to acknowledge the following:

• [x] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• [x] I have searched the existing issues, open and closed, and I'm convinced that mine is new.

1. From https://docs.opnsense.org/manual/etpro_telemetry.html, ET Telemetry provides "ET Pro ruleset free of charge". Per Proofpoint, ETPro ruleset includes the ETOpen rules. However, comparing Open rules and Telemetry rules, a lot of ET Open rules are missing from Telemetry. When excluding all disabled rules from ET Open and the categories not covered by Telemetry, ET Open contains more than 32k rules while ET Telemetry contains less than 20k rules. Referring to the comparison document https://www.proofpoint.com/sites/default/files/data-sheets/pfpt-us-ds-etpro-vs-etopen-ruleset.pdf, if ET Telemetry matches ETPRO, Telemetry should reach at least 70k rules.

There is then some obvious rule pruning/filtering done. I've read on the forum the intent is to eliminate some false positives and noisy rules. However, some became quite a lot and I can't believe recent high confidence malware rules are to be considered false positives or noisy.

2. Also, from my observation, Telemetry systematically lags 1 version/day behind both ET Open and Pro. If you check the daily rule updates at https://community.emergingthreats.net/, most new ET Open and Pro rules won't be available yet on the Telemetry ruleset while the new ET Open rules will be present on the Open ruleset. Example : Getting "opnsense-rules.tar.gz", "version": "10445" when available contains the added rules from v10444 + a few but not all ET Open and Pro additions from v10445. I saw this behavior over several versions.

At best, I would prefer to receive the full ET Pro ruleset (also containing the full Open ruleset) without the 1 version/day delay and do the rule filtering myself. If providing the full ET Pro ruleset isn't an option unlike implied on the docs, I would at least want to get the full ET Open ruleset without delay + the extra ETPro rules deemed useful ideally without delay. At least, I'd want to know the filtering criterias applied by Proofpoint to generate opnsense-rules.tar.gz from the original ET Pro ruleset. Such transparency would allow people to decide which ruleset suits them best.

[AdSchellevis]

Hi @0nnyx,

We did add a supplemental list a long time ago for (a selection of) the et-open rules in case one wants to combine the older and more diagnostics rules from the open ruleset, you can find it here https://github.com/opnsense/plugins/tree/master/security/intrusion-detection-content-et-open . For questions around Proofpoints ET-Pro telemetry offering I'll gladly refer to their support which is available at support@emergingthreats.net, we merely offer access to the rules.

Best regards,

Ad

[0nnyx]

Hello Ad, Appreciate your feedback.

I was aware about os-intrusion-detection-content-et-open. This plugin doesn't make ET Telemetry cover the base ruleset of ET Open, meaning the rules filenames starting with "emerging-". Instead, it adds IP blocklists in suricata rule format (drop dshield etc). Telemetry still misses many recent and valid Open rules (malware exploit worm etc).

Right now, we need to choose between ET Open and ET Telemetry (+ the extra IP blocklists from os-intrusion-detection-content-et-open) but can't have both at the same time. It would make more sense imo to get the full ET Open by default, then have the option to enable Telemetry only for the extra ET Pro rules. If only getting rules starting with ETPro through Telemetry isn't possible, Suricata can handle duplicate rules (with the same sid). Therefore, you could still provide the full ET Open ruleset after installing the Telemetry plugin. This way we could have both full ET Open rulesets and the extra ETPro rules then let Suricata filter the duplicate rules. What do you think about this approach ?

The main reason for this post is that the ET Telemetry documentation is misleading and inaccurate. It effectively allows to use a subset / limited version of the ET Pro ruleset but not "the ET Pro ruleset". Note that my intent here isn't to complain. I would honestly not expect to get the full ET Pro for free with telemetry. However, the wording from https://docs.opnsense.org/manual/etpro_telemetry.html should be made more clear imo that Telemetry doesn't equal ETPro. The only page more or less correct about it is https://shop.opnsense.com/etpro-telemetry-faq/ stating ETPro Telemetry Edition is a tuned version of the ETPro ruleset.

I will contact ET support for more clarifications on the tuning then post the reply back here if I get any.

Cheers

[AdSchellevis]

I was aware about os-intrusion-detection-content-et-open. This plugin doesn't make ET Telemetry cover the base ruleset of ET Open, meaning the rules filenames starting with "emerging-".

It's easy to extend, you can always open a PR to add the missing files. Given there's overlap, I wouldn't like to offer et-open and et-pro-telemetry both by default as the errors in the logs usually lead to new tickets nobody wants to answer.

[0nnyx]

The ET team provided useful clarifications :

• ET Telemetry has no hard cap on rule volume. Severity is not part of the selection criteria. Rules get included if they are marked as recommended (manual function done by researchers) AND Rules have hits within their sensor and sandbox network within 180 days OR were written within 30 days AND are NOT marked TLP Red. Telemetry covers then ET recommended disclosable rules less than 1 month old or triggered over the past 6 months.
• ET Telemetry doesn't include the complete ET Open due to lack of need/request. The supplemental ET Open is therefore useful for people wanting a more complete ruleset.
• ET team agreed about Telemetry ruleset lagging behind both Open and Pro and might improve the rule update process.