search

elastic / detection-rules

https://www.elastic.co/guide/en/security/current/detection-engine-overview.html
Other
1.91k stars 487 forks source link

[FR] Update schemas to support runtime fields #3509

Closed Mikaayenson closed 2 weeks ago

Mikaayenson commented 6 months ago

Summary

https://github.com/elastic/kibana/pull/130929

Tasks

#### PR Checklist
- [x] Link to the relevant Kibana PR or issue provided
- [ ] Exported detection rule(s) from Kibana to showcase the feature(s)
- [ ] Converted the exported ndjson file(s) to toml in the detection-rules repo
- [ ] Re-exported the toml rule(s) to ndjson and re-imported into Kibana
- [ ] Updated necessary unit tests to accommodate the feature
- [ ] Applied min_compat restrictions to limit the feature to a specified minimum stack version
- [ ] Executed all unit tests locally with a test toml rule to confirm passing
- [ ] Included Kibana PR implementer as an optional reviewer for insights on the feature
- [ ] Implemented requisite downgrade functionality
- [ ] Cross-referenced the feature with product documentation for consistency
- [ ] Incorporated a comprehensive test rule in unit tests for full schema coverage
- [ ] Conducted system testing, including fleet, import, and create APIs

Dependencies and Constraints

...

Mikaayenson commented 1 month ago

Update Aug 15

Now that we support custom schemas as a beta feature, we can probably close this out. We just need to build the runtime field in a stack and try to use DAC features with it to double check.

  1. Following the Example in the summary description of https://github.com/elastic/kibana/pull/130929 add a runtime field and then create a test rule
  2. Setup a custom DAC setup to test
python -m detection_rules custom-rules setup-config custom_rules

then add this config

bbr_rules_dirs:
- rules_building_block
directories:
  action_connector_dir: action_connectors
  action_dir: actions
  exception_dir: exceptions
files:
  deprecated_rules: etc/deprecated_rules.json
  packages: etc/packages.yaml
  stack_schema_map: etc/stack-schema-map.yaml
  version_lock: etc/version.lock.json
rule_dirs:
- rules
testing:
  config: etc/test_config.yaml
bypass_version_lock: True
normalize_kql_keywords: True
auto_gen_schema_file: "etc/schemas/auto_gen.json"
bypass_optional_elastic_validation: True
  1. Use the import / export commands to try importing and exporting the new rule created.
python -m detection_rules export-rules-from-repo  : TOML --> NDJSON
python -m detection_rules import-rules-to-repo  : NDJSON --> TOML
python -m detection_rules kibana export-rules  : Pull directly from Elastic Security --> TOML
python -m detection_rules kibana import-rules : Push from local TOML --> Elastic Security
shashank-elastic commented 2 weeks ago

Image

Image

detection-rules on ξ‚  issue-3509 [$?] is πŸ“¦ v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ python -m detection_rules custom-rules setup-config custom_rules

β–ˆβ–€β–€β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„   β–„      β–ˆβ–€β–€β–„ β–„  β–„ β–„   β–„β–„β–„ β–„β–„β–„
β–ˆ  β–ˆ β–ˆβ–„β–„  β–ˆ  β–ˆβ–„β–„ β–ˆ    β–ˆ   β–ˆ  β–ˆ β–ˆ β–ˆβ–€β–„ β–ˆ      β–ˆβ–„β–„β–€ β–ˆ  β–ˆ β–ˆ   β–ˆβ–„β–„ β–ˆβ–„β–„
β–ˆβ–„β–„β–€ β–ˆβ–„β–„  β–ˆ  β–ˆβ–„β–„ β–ˆβ–„β–„  β–ˆ  β–„β–ˆβ–„ β–ˆβ–„β–ˆ β–ˆ β–€β–„β–ˆ      β–ˆ β–€β–„ β–ˆβ–„β–„β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–„β–„β–ˆ

Created directory: custom_rules/actions
Created directory: custom_rules/action_connectors
Created directory: custom_rules/exceptions
Created directory: custom_rules/rules
Created directory: custom_rules/rules_building_block
Created directory: custom_rules/etc
Created file with default content: custom_rules/etc/deprecated_rules.json
Created file with default content: custom_rules/etc/version.lock.json
Created file with default content: custom_rules/etc/packages.yaml
Created file with default content: custom_rules/etc/stack-schema-map.yaml
Created file with default content: custom_rules/etc/test_config.yaml
Created file with default content: custom_rules/_config.yaml

# For details on how to configure the _config.yaml file,
# consult: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/_config.yaml
# or the docs: /Users/shashankks/elastic_workspace/detection-rules/docs/custom-rules.md
(.venv) 
detection-rules on ξ‚  issue-3509 [$?] is πŸ“¦ v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯

Updated Config

![Image](https://github.com/user-attachments/assets/9ad6f25d-b2e0-4ac6-925c-df13d2b0af94)

Details

python -m detection_rules import-rules-to-repo /Users/shashankks/Downloads/test_run_time_filed.ndjson β–ˆβ–€β–€β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„ β–„ β–ˆβ–€β–€β–„ β–„ β–„ β–„ β–„β–„β–„ β–„β–„β–„ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆβ–€β–„ β–ˆ β–ˆβ–„β–„β–€ β–ˆ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆβ–„β–„β–€ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆ β–„β–ˆβ–„ β–ˆβ–„β–ˆ β–ˆ β–€β–„β–ˆ β–ˆ β–€β–„ β–ˆβ–„β–„β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–„β–„β–ˆ [+] Building rule for /Users/shashankks/elastic_workspace/detection-rules/rules/run_time_filed_mapping_to_data_views.toml actions (multi, comma separated): alert_suppression: building_block_type: event_category_override: exceptions_list (multi, comma separated): false_positives (multi, comma separated): filters (multi, comma separated): index (multi, comma separated): investigation_fields: license: note: references (multi, comma separated): related_integrations (multi, comma separated): required_fields (multi, comma separated): risk_score_mapping (multi, comma separated): rule_name_override: setup: severity_mapping (multi, comma separated): tags (multi, comma separated): add mitre tactic? [y/N]: N throttle: tiebreaker_field: timeline_id: timeline_title: timestamp_field: timestamp_override: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/__main__.py", line 35, in main() File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/__main__.py", line 32, in main root(prog_name="detection_rules") File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/main.py", line 177, in import_rules_into_repo output = rule_prompt( ^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/cli_utils.py", line 237, in rule_prompt raise e File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/cli_utils.py", line 206, in rule_prompt rule = TOMLRule(path=Path(path), contents=TOMLRuleContents.from_dict({'rule': contents, 'metadata': meta})) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/mixins.py", line 142, in from_dict return schema.load(obj) ^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 910, in load all_loaded = super().load(data, many=many, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 722, in load return self._do_load( ^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 884, in _do_load self._invoke_schema_validators( File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 1185, in _invoke_schema_validators self._run_validator( File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 774, in _run_validator validator_func(output, partial=partial, many=many) File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule.py", line 1355, in post_conversion_validation data.validate_query(metadata) File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule.py", line 735, in validate_query return validator.validate(self, meta) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_validators.py", line 363, in validate raise validation_checks["stack"] eql.errors.EqlSchemaError: Error at line:1,column:11 Field not recognized any where `logs-data-view-files` == "logs data view filed" ^^^^^^^^^^^^^^^^^^^^^^ stack: 8.16.0, beats: 8.15.0,ecs: 8.11.0, endgame: 8.4.0 (.venv)

Sample ndjson of the exported rule is like below

``` {"id":"c234a270-5e5d-41c9-9237-f716c36ac34c","updated_at":"2024-09-03T17:07:07.929Z","updated_by":"841510929","created_at":"2024-09-03T17:07:07.929Z","created_by":"841510929","name":"Run Time Filed Mapping to Data Views","tags":[],"interval":"5m","enabled":true,"revision":0,"description":"Test Run Time Filed Mapping to Data Views","risk_score":21,"severity":"low","license":"","output_index":"","meta":{"from":"1m","kibana_siem_app_url":"https://e2erelease.kb.us-west2.gcp.elastic-cloud.com:9243/app/security"},"author":[],"false_positives":[],"from":"now-360s","rule_id":"a97cf517-bb1c-4d46-9522-f449fd3b0873","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":1,"exceptions_list":[],"immutable":false,"rule_source":{"type":"internal"},"related_integrations":[],"required_fields":[],"setup":"","type":"eql","language":"eql","data_view_id":"logs-*","query":"any where `logs-data-view-files` == \"logs data view filed\"","filters":[],"actions":[]} {"exported_count":1,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0,"exported_action_connector_count":0,"missing_action_connection_count":0,"missing_action_connections":[],"excluded_action_connection_count":0,"excluded_action_connections":[]} ```

I have double checked twice my steps, hopefully i am not missing anything here! @Mikaayenson

Mikaayenson commented 2 weeks ago

The only thing im not seeing is if you exported the CUSTOM_RULES_DIR environment variable. Also can you share the contents of auto_gen.json

shashank-elastic commented 2 weeks ago

The only thing im not seeing is if you exported the CUSTOM_RULES_DIR environment variable. Also can you share the contents of auto_gen.json

Yes the environment variable export was missing from the steps, and that fixed the problem

Successful Import of Rule with Run Time Field

```console ❯ python -m detection_rules import-rules-to-repo /Users/shashankks/Downloads/test_run_time_filed.ndjson --required-only β–ˆβ–€β–€β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„ β–„ β–ˆβ–€β–€β–„ β–„ β–„ β–„ β–„β–„β–„ β–„β–„β–„ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆβ–€β–„ β–ˆ β–ˆβ–„β–„β–€ β–ˆ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆβ–„β–„β–€ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆ β–„β–ˆβ–„ β–ˆβ–„β–ˆ β–ˆ β–€β–„β–ˆ β–ˆ β–€β–„ β–ˆβ–„β–„β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–„β–„β–ˆ [+] Building rule for /Users/shashankks/elastic_workspace/detection-rules/custom_rules/rules/run_time_filed_mapping_to_data_views.toml 1 results exported 1 rules converted 0 exceptions exported 0 actions connectors exported (.venv) detection-rules on ξ‚  issue-3509 [$?] is πŸ“¦ v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️ shashank.suryanarayana@elastic.co took 2s ``` AutoGen Schema populated successfully ![Image](https://github.com/user-attachments/assets/de947e7b-2fab-4574-a1e9-dd11ba4628d1)

Successful Export of Rule with Run Time Field

```console ❯ python -m detection_rules export-rules-from-repo -id "a97cf517-bb1c-4d46-9522-f449fd3b0873" Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json β–ˆβ–€β–€β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„ β–„ β–ˆβ–€β–€β–„ β–„ β–„ β–„ β–„β–„β–„ β–„β–„β–„ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆβ–€β–„ β–ˆ β–ˆβ–„β–„β–€ β–ˆ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆβ–„β–„β–€ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆ β–„β–ˆβ–„ β–ˆβ–„β–ˆ β–ˆ β–€β–„β–ˆ β–ˆ β–€β–„ β–ˆβ–„β–„β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–„β–„β–ˆ Exported 1 rules into /Users/shashankks/elastic_workspace/detection-rules/exports/20240903T234353L.ndjson (.venv) detection-rules on ξ‚  issue-3509 [$?] is πŸ“¦ v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️ shashank.suryanarayana@elastic.co ❯ ```

Successful Export of Rule with Run Time Field From Kibana

```console ❯ python -m detection_rules kibana export-rules -r "a97cf517-bb1c-4d46-9522-f449fd3b0873" -d /Users/shashankks/elastic_workspace/detection-rules/custom_rules Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json β–ˆβ–€β–€β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„ β–„ β–ˆβ–€β–€β–„ β–„ β–„ β–„ β–„β–„β–„ β–„β–„β–„ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆβ–€β–„ β–ˆ β–ˆβ–„β–„β–€ β–ˆ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆβ–„β–„β–€ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆ β–„β–ˆβ–„ β–ˆβ–„β–ˆ β–ˆ β–€β–„β–ˆ β–ˆ β–€β–„ β–ˆβ–„β–„β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–„β–„β–ˆ 1 results exported 1 rules converted 0 exceptions exported 0 action connectors exported 1 rules saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules 0 exception lists saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/exceptions 0 action connectors saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/action_connectors (.venv) detection-rules on ξ‚  issue-3509 [$?] is πŸ“¦ v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️ shashank.suryanarayana@elastic.co took 3s ```

Successful Import of Rule with Run Time Field To Kibana

Expected failure as the rule already exists ```console ❯ python -m detection_rules kibana import-rules -id "a97cf517-bb1c-4d46-9522-f449fd3b0873" Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json β–ˆβ–€β–€β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„ β–„ β–ˆβ–€β–€β–„ β–„ β–„ β–„ β–„β–„β–„ β–„β–„β–„ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆβ–€β–„ β–ˆ β–ˆβ–„β–„β–€ β–ˆ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆβ–„β–„β–€ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆ β–„β–ˆβ–„ β–ˆβ–„β–ˆ β–ˆ β–€β–„β–ˆ β–ˆ β–€β–„ β–ˆβ–„β–„β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–„β–„β–ˆ 1 rule(s) failed to import! - a97cf517-bb1c-4d46-9522-f449fd3b0873: (409) rule_id: "a97cf517-bb1c-4d46-9522-f449fd3b0873" already exists (.venv) detection-rules on ξ‚  issue-3509 [$?] is πŸ“¦ v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️ shashank.suryanarayana@elastic.co took 3s ❯ ``` Changed rule id and name just to test ```console ❯ python -m detection_rules kibana import-rules -id "b97cf517-bb1c-4d46-9522-f449fd3b0873" Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json β–ˆβ–€β–€β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„β–„β–„ β–„ β–„ β–ˆβ–€β–€β–„ β–„ β–„ β–„ β–„β–„β–„ β–„β–„β–„ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆ β–ˆβ–€β–„ β–ˆ β–ˆβ–„β–„β–€ β–ˆ β–ˆ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆβ–„β–„β–€ β–ˆβ–„β–„ β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–ˆ β–„β–ˆβ–„ β–ˆβ–„β–ˆ β–ˆ β–€β–„β–ˆ β–ˆ β–€β–„ β–ˆβ–„β–„β–ˆ β–ˆβ–„β–„ β–ˆβ–„β–„ β–„β–„β–ˆ 1 rule(s) successfully imported - b97cf517-bb1c-4d46-9522-f449fd3b0873 (.venv) detection-rules on ξ‚  issue-3509 [$?] is πŸ“¦ v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️ shashank.suryanarayana@elastic.co took 4s ❯ ``` ![Image](https://github.com/user-attachments/assets/79bc738d-ab96-4737-bb13-969d70114e86)

@Mikaayenson With DAC the feature is working as expected. This should be good to close

shashank-elastic commented 2 weeks ago

All verification steps completed