Add Rule Fields for Observers and Agents

[MikePaquette]

ECS needs a place to capture information about rules used to create detections, for example, web proxy rules, firewall rules, IDS/IPS signatures, EDR rules, WAF rules, SIEM rules, etc.

Proposed Fields

• [ ] rule.id - A rule ID that is unique within the scope of the entity using the rule for detection of this event
• [ ] rule.type - a categorization value keyword used by the entity using the rule for detection of this event
• [ ] rule.name - A keyword name for the rule used to create this event
• [ ] rule.description - a text description of the rule used to create this event
• [ ] rule.reference- Text URL pointing to additional information about the rule used to create this event
• [ ] rule.version - Rule version of the rule used to created this event. Used by systems that allow multiple rule versions to exist.
• [ ] rule.ruleset - name of the ruleset, policy, or group in which the rule used to create this event is a member.

Usage Example

• observer.type:"proxy"
• observer.name:"1_proxysg"
• observer.vendor:"symantec"
• observer.product:"s200"
• observer.os.name:"sgos"
• observer.os.version:"7.1"
• observer.geo.name:"3rd_floor_mdf"
• observer.rule.id:"00012345"
• observer.rule.type:"url"
• observer.rule.name: "no gambling at work"
• observer.rule.description:"blocks access to known gambling web sites
• observer.rule.reference:"http://sitereview.bluecoat.com/#/lookup-result/betthefarm.xyz"
• observer.rule.version:"1"
• observer.rule.ruleset:"Standard Employee Browsing Policy"

Related Fields In security use cases, there may be other identifiers used for classifying rules, such as the MITRE ATT&CK framework. If this is the case, such information should be populated in the ECS threat.* fields.

Field Reuse Other field sets must not be nested under the rule.* field set The rule.* fields must not be used at the top level.

Nested usage	Description
observer.rule.*	Fields describing rules running on an observer used to detect this event
agent.rule.*	Fields describing rules running on an agent used to detect this event

[webmat]

Thanks Mike! Got a few questions / challenges ;-)

ECS needs a place to capture information about rules used to create detections

Is this meant to capture the ensuing alert events or events about the rules? The way the above is phrased sounds like you'd want the event to be about the rules themselves (creating the rule, enabling/disabling the rule).

I don't see the advantage of making the rules nested in two places, in this case. Can you expand on why you're proposing nesting them?

I agree the rules are set either on an agent or on an observer. However nesting the rules under both means that you can't analyze both agent-based rules and observer-based rules together, because they have different field names (agent.rule.name v observer.rule.name). This is a problem IMO, especially because the same piece of software can be used either as an agent or as an observer. This in turn means the sets of rules that can be configured for a single piece of software can't be analyzed together, depending on the configuration (e.g. Packetbeat on a host -- an agent -- or Packetbeat on a tap -- an observer).

If we keep this at the top level, we can associate the rule with which agent/observer triggered it using another method. And having it at the top level will mean we can analyze all of the rules together.

[MikePaquette]

thanks @webmat

Is this meant to capture the ensuing alert events or events about the rules? The way the above is phrased sounds like you'd want the event to be about the rules themselves (creating the rule, enabling/disabling the rule).

Nope. This field set is meant to capture details about the rule that was used by the observer or agent to generate the alert. It is not about rule creation/enabling/disabling.

I don't see the advantage of making the rules nested in two places, in this case. Can you expand on why you're proposing nesting them?

I was thinking about future ECS Apps that might have their own rules that operate on alerts generated from observer or agent rules. These apps might generate their own detection events that contain both the original rule and their own rule, and thus they would have two rules associated with one event. So I was thinking that these apps could also re-use the newly defined rule.* fields in their outputs.

Even with that said, I think you raise a great point about not nesting rule.* fields under observer and agent. I'll retract that proposal :-).

Question: Would it make sense putting these fields under event.rule.* or better at the top level?

[enotspe]

Rule fields are very needed. I vote for top level.

One suggestion: rule.comment should also be considered.

[webmat]

@enotspe Thanks for chiming in! I'm curious about rule.comment: the systems that let you configure alert rules with comments can also pass along the comments in their ensuing events? If so, that's great, and I think this added field would make sense.

@MikePaquette Thanks for reconsidering the nesting :-) Although I agree with @enotspe and I think this field set would work best at the top level. If it's at event.rule.* it doesn't get listed in the main field sets here. That situation is fine for transport details like log.syslog.*, but I think rules and alerts should be front and center.

I understand that the rules are what will end up causing alerts to be triggered. But the ensuing events are the alert themselves. Why wouldn't we be calling the field set alert? Someone trying to map their IDS alerts, looking for the right place to put their alert details may not even realize that "rule" is meant for that. Also, the word "rule" is really generic and may end up being useful in other situations than alert rules (although I don't have an example that comes to mind yet).

Overall I love the proposed fields inside the field set. I think I would also add .category. I haven't taken the time to go around various source alerts, but I looked at this Suricata example alert, and category is one that I think we don't have a direct mapping for.

[enotspe]

the systems that let you configure alert rules with comments can also pass along the comments in their ensuing events?

At least FortiOS does.

[enotspe]

One more suggestion:

rule.universally_unique_identifier

https://en.wikipedia.org/wiki/Universally_unique_identifier

[webmat]

If there's such a thing as a UUID for alert rules, I agree with including it.

Although I think it's another case of the convention/habits being so strongly in favour of the contraction, that I'd want this to be rule.uuid rather than rule.universally_unique_identifier; adding yet another exception to our rule of trying to keep with full words :-)

[webmat]

Saw relevant fields that would map to rule.* when glancing at this PR yesterday https://github.com/elastic/beats/pull/14553. Let's make sure this field set can capture them :-)

[dainperkins]

@enotspe would rule.uuid be in addition to rule.id? I'm not opposed to adding it if it makes sense, but I can't recall any alerts I've ever seen that included anything more than a locally significant rule id. Looking at beats info now @webmat - will pop an updated pre-PR list in comments shortly for review, and will try to get the PR in by EOD

[dainperkins]

1 more thought (probably esoteric or aesthetic) but would rule.ruleset be better served as rule.group or rule.policy to represent a higher level grouping of individual rules that are applied as a whole based on some criteria?
change rule.ruleset to rule.policy?

along those lines should we have something to represent the locally significant groupings as defined in the observer / agent (thinking along the lines of Web Filter / DLP that has local groups defined to apply policies (those groups likely being made up of enterprise IAM users/groups) rule.group

[MikePaquette]

@enotspe @webmat I am not opposed to adding rule.comment but I think that rule.name, rule.description, and rule.reference cover it pretty well. I'm happy to leave it to whomever writes the PR to make the in/out decision on rule.comment.

@webmat I am strongly opposed to creating an alert.* field set. ECS provides for (and will soon define) event.kind:"alert" as the way to indicate that an event is an alert. At this higher level (event.kind) of categorization, we can then use all the other fields sets to further describe the alert, without resulting in a duplication of fields under an alert object.

@dainperkins @enotspe I too am not familiar with a globally unique ID (UUID) being associated with a rule, and even if one exists, it's probably not common, so I suggest we go ahead with just a the rule.id field as defined in this issue above.

@webmat Regarding field re-use, I am :+1 to allowing rule at the top level of the naming hierarchy.

@dainperkins I prefer not to change rule.ruleset to rule.group or rule.policy, primarily because I think ruleset is more general, and mapping policy or group to ruleset seems pretty natural. Also, group is already a defined object in ECS, and I think this would be confusing to ECS users, who would wonder if its a rule group, or the group of, say users, affected by a rule.

Thanks all for the great thought put into this!

[dainperkins]

I went thru the Beats GCP Fw fields, a number of these seem very specific to cloud use cases, but I'm thinking we should be considering them - just not sure if it should be in rule, or cloud...

Bold seems to fit, or be a good idea for addition, italics I'm questioning if we don't just add cloud entries to the event type...

I guess my question is should we be e.g. inventorying rules or using this just to identify which rules we're triggered and why, or should we be thinking about both since its not that much more work - define the full scope and use event.type: inventory or similar and set a smaller set of rule info for event.type: alert ?

Field	Description	Notes
rule_details	Description of the firewall rule that matched this connection.	rule.description
rule_details.action	Action that the rule performs on match.	rule.action
rule_details.destination_range	List of destination ranges that the firewall applies to.	rule.destination
rule_details.direction	Direction of traffic that matches this rule.	seems intrinsic?
rule_details.ip_port_info	List of ip protocols and applicable port ranges for rules.	rule.ports & rule.application
rule_details.priority	The priority for the firewall rule.	rule.priority
rule_details.reference	Reference to the firewall rule.	rule.reference
rule_details.source_range	List of source ranges that the firewall rule applies to.	rule.source
rule_details.source_service_account	List of all the source service accounts that the firewall rule applies to.	_rule.sourceuser
rule_details.source_tag	List of all the source tags that the firewall rule applies to.	_rule.sourcetag
rule_details.target_service_account	List of all the target service accounts that the firewall rule applies to.	_rule.targetuser
rule_details.target_tag	List of all the target tags that the firewall rule applies to.	_rule.sourcetag

[MikePaquette]

Good question @dainperkins I'd suggest we keep the scope as a place to capture information about rules used to create detection events and alerts, not as a full specification of fields for all kinds of rules.

In other words, let's not add fields for destination, direction, ports, source, dest, target, etc. at this time.

Also, we've already defined rule.description and rule.reference above, and I'd suggest skipping rule.priority and rule.action at this point. I think those fields would fit into existing event fields.

Care to create a PR based on this scope?

[enotspe]

major firewalls vendors support uuid for at least for firewall policies (other objects are supported as well). When you have 100s of rules to manage per fw, and as few as 10s of firewalls, then uuid becomes really handy.

Fortinet Policy UUID Palo Alto Rule UUID Checkpoint rule_uid

I suggest we keep rule.uuid as extended

[MikePaquette]

@enotspe thanks for the example. The Palo Alto example is clear. I am 👍 for including rule.uuid in this field set and updating the definitions as follows:

• [ ] rule.id - A rule ID that is unique within the scope of the agent, observer, or other entity using the rule for detection of this event
• [ ] rule.uuid - A rule ID that is unique within the scope of a set or group of agents, observers, or other entities using the rule for detection of this event.

[dainperkins]

Opened PR 665 - https://github.com/elastic/ecs/pull/665