[FR] Fail Earlier on min_compat validation

[Mikaayenson]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

When we use features in our rules that are implemented with a minimum compatibility (e.g. field(metadata=dict(metadata=dict(min_compat="8.13")))), currently the only way to detect this restriction is to merge a rule into our repo, let it backport, and watch the errors on the unsupported release branch.

For example:

1) Imagine a rule uses the alert suppression field which has different minimum compatibility stack versions based on when the feature was implemented in kibana. 2) Then we merge a rule without a min_stack_version. 3) This rule will backport as far as possible, which could inadvertently overlap with a stack not supported. 4) That rule will pass on main but fail on a branch, which then requires us to revert the PR, minstack the rule properly, then open a new PR .

This is possible because the current logic checks incompatible field versions against the package_version which is the latest version defined in the stack_schema_map file. It's also possible because we dont consider the rules min_stack_version at all in this validation.

class StackCompatMixin:
    """Mixin to restrict schema compatibility to defined stack versions."""

    @validates_schema
    def validate_field_compatibility(self, data: dict, **kwargs):
        """Verify stack-specific fields are properly applied to schema."""
        package_version = Version.parse(load_current_package_version(), optional_minor_and_patch=True)
        schema_fields = getattr(self, 'fields', {})
        incompatible = get_incompatible_fields(list(schema_fields.values()), package_version)
        if not incompatible:
            return

        package_version = load_current_package_version()
        for field, bounds in incompatible.items():
            min_compat, max_compat = bounds
            if data.get(field) is not None:
                raise ValidationError(f'Invalid field: "{field}" for stack version: {package_version}, '
                                      f'min compatibility: {min_compat}, max compatibility: {max_compat}')

Desired Solution

We should:

1) Pull the package_version based on the rules min_stack_version if available, and otherwise use the latest supplied as-is.

package_version = min_stack_version or Version.parse(load_current_package_version(), optional_minor_and_patch=True)

2) Check that package_version against all the fields min_compat and max_compat bounds to ensure it's in the range. 3) Pull the latest backport range and fail if any min_compat will fall outside the backport range (for any fields used in the rule)

Considered Alternatives

What we've been doing today is catching this when it fails in back ports. Alternatively we could try to soft enforce this with guidelines per PR.

Additional Context

No response

[shashank-elastic]

@mikaayenson I was going through the originally PR of StackCompatMixin for better context and saw a unit test in place, so wanted to test out!

1. Went to Prebuilt Rule Set identified a sample rule that has restricted filed types
2. Identified Potential Suspicious Clipboard Activity Detected rule that has new_terms filed.
3. Now Added alert suppression for new_terms filed that is available only in 8.14.0 and above refer
4. Also Query Rule Data has the right min_compat set

5. So The Rule being tested had no min_stack_version set for testing purposes
   Rule Details

[metadata]
creation_date = "2024/08/28"
maturity = "production"
updated_date = "2024/08/28"

[rule]
actions = []
author = ["Elastic"]
building_block_type = "default"
description = """
This rule monitors for the usage of the most common clipboard utilities on unix systems by an uncommon process group
leader. Adversaries may collect data stored in the clipboard from users copying information within or between
applications.
"""
filters = []
from = "now-7140s"
index = ["logs-endpoint.events.*", "endgame-*", "auditbeat-*", "logs-auditd_manager.auditd-*"]
interval = "60m"
language = "kuery"
license = "Elastic License v2"
max_signals = 100
name = "Potential Suspicious Clipboard Activity Detected [Duplicate]"
note = "None"
risk_score = 21
rule_id = "e74912de-eb3f-440d-9759-dc6fe09579b8"
setup = "None"
severity = "low"
tags = [
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Collection",
    "Rule Type: BBR",
    "Data Source: Elastic Defend",
    "Data Source: Elastic Endgame",
    "Data Source: Auditd Manager",
]
timeline_id = "76e52245-7519-4251-91ab-262fb1a1728c"
timeline_title = "Generic Process Timeline"
timestamp_override = "event.ingested"
to = "now"
type = "new_terms"

query = '''
event.category:process and host.os.type:"linux" and
event.type:"start" and event.action:("exec" or "exec_event" or "executed" or "process_started") and
process.name:("xclip" or "xsel" or "wl-clipboard" or "clipman" or "copyq")
'''

[rule.alert_suppression]
group_by = ["host.id"]
missing_fields_strategy = "doNotSuppress"

[rule.meta]
from = "59m"
kibana_siem_app_url = ""

[rule.new_terms]
field = "new_terms_fields"
value = ["host.id", "process.group_leader.executable"]
[[rule.new_terms.history_window_start]]
field = "history_window_start"
value = "now-7d"

[rule.alert_suppression.duration]
unit = "m"
value = 5

6. TestIncompatibleFields.test_rule_backports_for_restricted_fields unit test fails, and this wont go till backports!

Details


```console Received JSON data in run script Running pytest with args: ['-p', 'vscode_pytest', '--rootdir=/Users/shashankks/elastic_workspace/detection-rules', '/Users/shashankks/elastic_workspace/detection-rules/tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields'] ============================= test session starts ============================== platform darwin -- Python 3.12.5, pytest-8.1.1, pluggy-1.4.0 rootdir: /Users/shashankks/elastic_workspace/detection-rules configfile: pyproject.toml plugins: typeguard-3.0.2 collected 1 item tests/test_all_rules.py F [100%] =================================== FAILURES =================================== _______ TestIncompatibleFields.test_rule_backports_for_restricted_fields _______ self = def test_rule_backports_for_restricted_fields(self): """Test that stack restricted fields will not backport to older rule versions.""" invalid_rules = [] for rule in self.all_rules: invalid = rule.contents.check_restricted_fields_compatibility() if invalid: invalid_rules.append(f'{self.rule_str(rule)} {invalid}') if invalid_rules: invalid_str = '\n'.join(invalid_rules) err_msg = 'The following rules have min_stack_versions lower than allowed for restricted fields:\n' err_msg += invalid_str > self.fail(err_msg) E AssertionError: The following rules have min_stack_versions lower than allowed for restricted fields: E e74912de-eb3f-440d-9759-dc6fe09579b8 - Potential Suspicious Clipboard Activity Detected [Duplicate] -> {'alert_suppression': {'min_stack_version': Version(major=8, minor=10, patch=0, prerelease=None, build=None), 'min_allowed_version': Version(major=8, minor=14, patch=0, prerelease=None, build=None)}} tests/test_all_rules.py:1203: AssertionError =========================== short test summary info ============================ FAILED tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields ============================== 1 failed in 51.35s ============================== Finished running tests! ```

This made me wonder what more is required for the feature?

[Mikaayenson]

Right. Look at the actual error. If we added a min_stack of 8.10 to the rule, it would not fail. This would still allow us to try to back port to 8.11 where the field would then fail since the feature isn't available until 8.14, but again that's after we already merged to main.

[shashank-elastic]

@Mikaayenson

I dont think thats the case.

Example : I min stacked the sample rule for 8.12.0 [ the filed is available in 8.14.0 and above ]

The unit test still fails

```console =================================== FAILURES =================================== _______ TestIncompatibleFields.test_rule_backports_for_restricted_fields _______ self = def test_rule_backports_for_restricted_fields(self): """Test that stack restricted fields will not backport to older rule versions.""" invalid_rules = [] for rule in self.all_rules: invalid = rule.contents.check_restricted_fields_compatibility() if invalid: invalid_rules.append(f'{self.rule_str(rule)} {invalid}') if invalid_rules: invalid_str = '\n'.join(invalid_rules) err_msg = 'The following rules have min_stack_versions lower than allowed for restricted fields:\n' err_msg += invalid_str > self.fail(err_msg) E AssertionError: The following rules have min_stack_versions lower than allowed for restricted fields: E e74912de-eb3f-440d-9759-dc6fe09579b8 - Potential Suspicious Clipboard Activity Detected [Duplicate] -> {'alert_suppression': {'min_stack_version': Version(major=8, minor=12, patch=0, prerelease=None, build=None), 'min_allowed_version': Version(major=8, minor=14, patch=0, prerelease=None, build=None)}} tests/test_all_rules.py:1203: AssertionError =========================== short test summary info ============================ FAILED tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields ============================== 1 failed in 49.46s ============================== Finished running tests! ```

I have tried min stacking to all unavailable versions until min supported version until 8.10, in all of the cases we see a unit test failure.

8.13

Details

```console =================================== FAILURES =================================== _______ TestIncompatibleFields.test_rule_backports_for_restricted_fields _______ self = def test_rule_backports_for_restricted_fields(self): """Test that stack restricted fields will not backport to older rule versions.""" invalid_rules = [] for rule in self.all_rules: invalid = rule.contents.check_restricted_fields_compatibility() if invalid: invalid_rules.append(f'{self.rule_str(rule)} {invalid}') if invalid_rules: invalid_str = '\n'.join(invalid_rules) err_msg = 'The following rules have min_stack_versions lower than allowed for restricted fields:\n' err_msg += invalid_str > self.fail(err_msg) E AssertionError: The following rules have min_stack_versions lower than allowed for restricted fields: E e74912de-eb3f-440d-9759-dc6fe09579b8 - Potential Suspicious Clipboard Activity Detected [Duplicate] -> {'alert_suppression': {'min_stack_version': Version(major=8, minor=13, patch=0, prerelease=None, build=None), 'min_allowed_version': Version(major=8, minor=14, patch=0, prerelease=None, build=None)}} tests/test_all_rules.py:1203: AssertionError =========================== short test summary info ============================ FAILED tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields ============================== 1 f ```

8.11

Details

=================================== FAILURES =================================== _______ TestIncompatibleFields.test_rule_backports_for_restricted_fields _______ self = def test_rule_backports_for_restricted_fields(self): """Test that stack restricted fields will not backport to older rule versions.""" invalid_rules = [] for rule in self.all_rules: invalid = rule.contents.check_restricted_fields_compatibility() if invalid: invalid_rules.append(f'{self.rule_str(rule)} {invalid}') if invalid_rules: invalid_str = '\n'.join(invalid_rules) err_msg = 'The following rules have min_stack_versions lower than allowed for restricted fields:\n' err_msg += invalid_str > self.fail(err_msg) E AssertionError: The following rules have min_stack_versions lower than allowed for restricted fields: E e74912de-eb3f-440d-9759-dc6fe09579b8 - Potential Suspicious Clipboard Activity Detected [Duplicate] -> {'alert_suppression': {'min_stack_version': Version(major=8, minor=11, patch=0, prerelease=None, build=None), 'min_allowed_version': Version(major=8, minor=14, patch=0, prerelease=None, build=None)}} tests/test_all_rules.py:1203: AssertionError =========================== short test summary info ============================ FAILED tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields ============================== 1 failed in 48.92s ============================== Finished running tests!

8.10

Details

=================================== FAILURES =================================== _______ TestIncompatibleFields.test_rule_backports_for_restricted_fields _______ self = def test_rule_backports_for_restricted_fields(self): """Test that stack restricted fields will not backport to older rule versions.""" invalid_rules = [] for rule in self.all_rules: invalid = rule.contents.check_restricted_fields_compatibility() if invalid: invalid_rules.append(f'{self.rule_str(rule)} {invalid}') if invalid_rules: invalid_str = '\n'.join(invalid_rules) err_msg = 'The following rules have min_stack_versions lower than allowed for restricted fields:\n' err_msg += invalid_str > self.fail(err_msg) E AssertionError: The following rules have min_stack_versions lower than allowed for restricted fields: E e74912de-eb3f-440d-9759-dc6fe09579b8 - Potential Suspicious Clipboard Activity Detected [Duplicate] -> {'alert_suppression': {'min_stack_version': Version(major=8, minor=10, patch=0, prerelease=None, build=None), 'min_allowed_version': Version(major=8, minor=14, patch=0, prerelease=None, build=None)}} tests/test_all_rules.py:1203: AssertionError =========================== short test summary info ============================ FAILED tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields ============================== 1 failed in 48.75s ============================== Finished running tests!

[Mikaayenson]

Oh right! So we may be able to close this out with the cover in the unit test. Perhaps even I noticed something looked odd, I only looked at the schema validation which is only half of the cover.

No min stack defaults to the earliest supported, otherwise uses the rules.

Thanks for taking a look!

[shashank-elastic]

The Unit test TestIncompatibleFields::test_rule_backports_for_restricted_fields covers the use case and has been retested.

No additional fixes are required.