elastic / detection-rules

https://www.elastic.co/guide/en/security/current/detection-engine-overview.html
Other
1.91k stars 492 forks source link

[FR] [DAC] Add support for CCS in Beats Validation #3853

Closed eric-forte-elastic closed 2 months ago

eric-forte-elastic commented 3 months ago

Issue Links

https://github.com/elastic/detection-rules/issues/3708

Summary

This PR adds some additional support for validation of rules using cross cluster search(CCS). This updates the method of parsing beats schemas to also allow for a CCS prefix.

For more context, see this specific error that we are attempting to resolve: https://github.com/elastic/infosec/issues/16183#issuecomment-2092652160

Testing

If you have a rule that utilizes CCS in Kibana you can use an export command similar to the following to export it to test python -m detection_rules kibana export-rules -d custom_rules/rules -sv

If you do now have one setup, you can test locally using view-rule on the following test rule file. What we are testing is that the schema validation appropriately uses schemas when the indexes are using CCS prefixes.

Test Rule

```toml [metadata] creation_date = "2024/07/01" maturity = "production" updated_date = "2024/07/01" [rule] actions = [] author = [] description = "INFOSEC CSS Test Rule" enabled = true exceptions_list = [] false_positives = [] filters = [] from = "now-18060s" index = ["infosec_test:logs-*", "infosec_test:auditbeat-*"] interval = "5h" language = "eql" license = "" max_signals = 100 name = "INFOSEC CSS Test Rule" references = [] related_integrations = [] required_fields = [] risk_score = 21 risk_score_mapping = [] rule_id = "521629d1-61e4-4203-8a16-a08d5dc20281" setup = "" severity = "low" severity_mapping = [] tags = [] threat = [] to = "now" type = "eql" query = ''' process where process.name == "FakeRoot" ''' [rule.meta] from = "1m" kibana_siem_app_url = "https://dev-deployment-2c684a.kb.us-central1.gcp.cloud.es.io:9243/app/security" ```

Result Prior to Fix

```shell detection-rules on  3708-frdac-support-ccs-index-patters-with-ecs-fields [!?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte took 2s ❯ python -m detection_rules view-rule custom_rules/rules/infosec_css_test_rule.toml Loaded config file: /home/forteea1/Code/clean_mains/additional_mains/detection-rules/.detection-rules-cfg.json █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ Error loading rule in /home/forteea1/Code/clean_mains/additional_mains/detection-rules/custom_rules/rules/infosec_css_test_rule.toml Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/__main__.py", line 35, in main() File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/__main__.py", line 32, in main root(prog_name="detection_rules") File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/main.py", line 231, in view_rule rule = RuleCollection().load_file(rule_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule_loader.py", line 438, in load_file return self.load_dict(obj, path=path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule_loader.py", line 414, in load_dict contents = TOMLRuleContents.from_dict(obj) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/mixins.py", line 142, in from_dict return schema.load(obj) ^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 910, in load all_loaded = super().load(data, many=many, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 723, in load return self._do_load( ^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 885, in _do_load self._invoke_schema_validators( File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 1186, in _invoke_schema_validators self._run_validator( File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 775, in _run_validator validator_func(output, partial=partial, many=many) File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule.py", line 1302, in post_conversion_validation data.validate_query(metadata) File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule.py", line 721, in validate_query return validator.validate(self, meta) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule_validators.py", line 314, in validate validation_checks["stack"] = self.validate_stack_combos(data, meta) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule_validators.py", line 341, in validate_stack_combos beat_types, beat_schema, schema = self.get_beats_schema(data.index or [], ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/utils.py", line 295, in wrapped _cache[func_key][cache_key] = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule.py", line 683, in get_beats_schema beat_schema = beats.get_schema_from_kql(self.ast, beat_types, version=beats_version) if beat_types else None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/beats.py", line 284, in get_schema_from_kql return get_schema_from_datasets(beats, modules, datasets, version=version) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/beats.py", line 241, in get_schema_from_datasets filtered.update(get_beat_root_schema(beats_schema, beat)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/beats.py", line 163, in get_beat_root_schema raise KeyError(f"Unknown beats module {beat}") ```

Result Post-Fix

```shell detection-rules on  3708-frdac-support-ccs-index-patters-with-ecs-fields [!?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte ❯ python -m detection_rules view-rule custom_rules/rules/infosec_css_test_rule.toml Loaded config file: /home/forteea1/Code/clean_mains/additional_mains/detection-rules/.detection-rules-cfg.json █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ { "actions": [], "author": [], "description": "INFOSEC CSS Test Rule", "enabled": true, "exceptions_list": [], "false_positives": [], "filters": [], "from": "now-18060s", "index": [ "infosec_test:logs-*", "infosec_test:auditbeat-*" ], "interval": "5h", "language": "eql", "license": "", "max_signals": 100, "meta": { "from": "1m", "kibana_siem_app_url": "https://dev-deployment-2c684a.kb.us-central1.gcp.cloud.es.io:9243/app/security" }, "name": "INFOSEC CSS Test Rule", "query": "process where process.name == \"FakeRoot\"\n", "references": [], "related_integrations": [], "required_fields": [], "risk_score": 21, "risk_score_mapping": [], "rule_id": "521629d1-61e4-4203-8a16-a08d5dc20281", "setup": "", "severity": "low", "severity_mapping": [], "tags": [], "threat": [], "to": "now", "type": "eql", "version": 1 } ```

One you can export/view your rule successfully. Now we need to test to make sure the correct schemas are being used even with the CCS indexes. To do this, modify your rule so that the query should not pass validation, modify process.name to be process.myname. This should cause an error in the schema validation as process.myname is not present in the schema. Additionally, the appropriate schema versions should be shown as what is being used for validation. See below for more details.

Detailed Output

```shell detection-rules on  3708-frdac-support-ccs-index-patters-with-ecs-fields [!?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte ❯ python -m detection_rules view-rule custom_rules/rules/infosec_css_test_rule.toml Loaded config file: /home/forteea1/Code/clean_mains/additional_mains/detection-rules/.detection-rules-cfg.json █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ Error loading rule in /home/forteea1/Code/clean_mains/additional_mains/detection-rules/custom_rules/rules/infosec_css_test_rule.toml Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/__main__.py", line 35, in main() File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/__main__.py", line 32, in main root(prog_name="detection_rules") File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/main.py", line 231, in view_rule rule = RuleCollection().load_file(rule_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule_loader.py", line 438, in load_file return self.load_dict(obj, path=path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule_loader.py", line 414, in load_dict contents = TOMLRuleContents.from_dict(obj) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/mixins.py", line 142, in from_dict return schema.load(obj) ^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 910, in load all_loaded = super().load(data, many=many, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 723, in load return self._do_load( ^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 885, in _do_load self._invoke_schema_validators( File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 1186, in _invoke_schema_validators self._run_validator( File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 775, in _run_validator validator_func(output, partial=partial, many=many) File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule.py", line 1302, in post_conversion_validation data.validate_query(metadata) File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule.py", line 721, in validate_query return validator.validate(self, meta) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/rule_validators.py", line 321, in validate raise validation_checks["stack"] eql.errors.EqlSchemaError: Error at line:1,column:15 Field not recognized for process event process where process.myname == "FakeRoot" ^^^^^^^^^^^^^^ stack: 8.15.0, beats: 8.13.4,ecs: 8.11.0, endgame: 8.4.0 ```