Requirement on comparison expression 'AND' should apply to sub-expressions

[clenk]

Suggestion from @chisholm:

In section 9.5, change

The Comparison Expressions of an Observation Expression that use AND MUST use the same base Object Path, e.g., file:.

to

All operands of AND in an Observation Expression MUST be satisfiable with the same type of object.

---

Original comment (https://github.com/oasis-open/cti-python-stix2/pull/456):

The spec is lacking in describing requirements on comparison expressions. For AND, it says:

The Comparison Expressions of an Observation Expression that use AND MUST use the same base Object Path, e.g., file:.

That sentence seems to assume that the operands to an AND are never sub-expressions. If they are, it doesn't make sense to talk about their "base object path". A sub-expression may be of arbitrary complexity, with lots of base object paths inside it.

The approach seems to be to make expressions which are syntactically valid but can never match, a spec violation. In keeping with this idea, one might use it to better define requirements on comparison expression AND, e.g.:

All operands of AND MUST be satisfiable with the same type of object.

The "satisfiable" idea is applicable to sub-expressions, and you can always tell what the applicable types are just by inspecting the pattern. For OR, it's the union of all operand types. For AND, it's the intersection. For example:

(type-a:a=1 OR type-b:b=2) AND (type-a:a=2 OR type-b:b=1)

The applicable types in the AND's lhs subexpression are {type-a, type-b}. It's same on the right side. Their intersection is {type-a, type-b}, which is non-empty, therefore the comparison expression is legal. The AND operands are all satisfiable with the same type of object, and the possible types include type-a and type-b. For:

(type-a:a=1 OR type-b:b=2) AND (type-a:a=2 OR type-c:b=1)

We have {type-a, type-b} intersect {type-a, type-c}, which is {type-a}. So the only object type we can satisfy both AND operands with is type-a. But since one exists, it's a legal pattern.

[jordan2175]

We talked about this on an editors call and agree. This change has been made.

[jmgnc]

This IMO, is a significant change, and needs to be discussed further (it makes previously invalid patterns valid). This text that is being changed was added because people did not feel that this was clear enough:

That is, when resolving object paths of each Comparison Expression, the : MUST start from the same SCO. Different SCO's may ultimately be used in matching, but they MUST be referenced from the same, single SCO.

The problem is that the example given:

(type-a:a=1 OR type-b:b=2) AND (type-a:a=2 OR type-b:b=1)

If it was compared to a observed data object like:

{
"objects": [
   { "type": "type-a", "a": 1 },
   { "type": "type-b", "b": 1 },
]
}

It could be argued that the object matches the above pattern, but the specification requires that this object NOT match the above pattern, and this clarification, IMO, adds confusion, as the pattern is "satisfiable", but does not make it clear that the above MUST NOT match due to the above quoted text.

This also introduces a significant change between STIX 2.0 and 2.1 making the languages incompatible. The TC explicitly chose not to make significant changes to patterning for 2.1, so IMO this change should be reverted. I am fine w/ further clarification of the rules, but this is a material change IMO.

[chisholm]

This IMO, is a significant change, and needs to be discussed further (it makes previously invalid patterns valid).

Your concern here is about pattern validity (i.e. spec compliance), whereas:

It could be argued that the object matches the above pattern, but the specification requires that this object NOT match the above pattern

this concern is about how patterns match. You may be conflating the two. My intent was to clarify the criteria for pattern validity. I intended no change to how patterns match observations. In your example, the pattern does not match. As far as I am aware, that has not changed (but I am not as up-to-date regarding the details of the latest standardization activities). It is nevertheless a valid pattern.

[allant0]

Given JMG's analysis and explanation on this being a material change then I would suggest we defer to a future release. It would seem that there are crude work-arounds to this problem within systems that are performing pattern matching where they could combine the result of two patterns themselves outside of the STIX pattern grammar. They have to have this type of capability anyway where multiple patterns may be getting matched to a condition and therefore handling conflicts and multiple hits of patterns has to be done regardless of what is fixed in this specific case. I suggest deferral.

[jordan2175]

We will talk about this again next week.

[jordan2175]

After talking about this again, we made one more minor tweak. The new text is:

All operands of AND in an Observation Expression MUST be satisfiable with the same object.