[FR] Added Schema Check for Data View ID and Index

[eric-forte-elastic]

Issues

https://github.com/elastic/detection-rules/issues/3817

Summary

This PR adds a Schema check for QueryRuleData that checks to make sure only Index or Data View ID are set for a given rule and not both. Currently we only support rules using one or the other and this will catch cases where rules may inadvertently reference both.

Testing

Use the rule from https://github.com/elastic/detection-rules/pull/3510, and an index field so that it has both the index and data_view_id set.

Example Rule

```toml [metadata] creation_date = "2024/03/13" maturity = "development" updated_date = "2024/03/13" [rule] author = ["me"] index = ["logs-endpoint.events.*"] data_view_id = "logs-*" description = "ere add aims aewerdas" language = "eql" name = "test" risk_score = 21 rule_id = "8f6eb3b6-e9f2-4c10-a72b-cf48b4e90c2d" severity = "low" type = "eql" query = ''' process where true ''' ```

Now run view-rule and check to make sure there is now a validation error.

Validation Error

```shell detection-rules on  3817-bug-schema-should-not-allow-index-and-dataview [!?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte took 2s ❯ python -m detection_rules view-rule rules/linux/command_and_control_development.toml Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ Error loading rule in /home/forteea1/Code/clean_mains/detection-rules/rules/linux/command_and_control_development.toml Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 35, in main() File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 32, in main root(prog_name="detection_rules") File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/main.py", line 224, in view_rule rule = RuleCollection().load_file(rule_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 435, in load_file return self.load_dict(obj, path=path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 412, in load_dict contents = TOMLRuleContents.from_dict(obj) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/mixins.py", line 142, in from_dict return schema.load(obj) ^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 910, in load all_loaded = super().load(data, many=many, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 723, in load return self._do_load( ^^^^^^^^^^^^^^ File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 910, in _do_load raise exc marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'_schema': ['Only one of index or data_view_id should be set.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.'], 'query': ['Unknown field.'], 'data_view_id': ['Unknown field.'], 'language': ['Unknown field.'], 'index': ['Unknown field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]} ```

Example Run