elastic / detection-rules

https://www.elastic.co/guide/en/security/current/detection-engine-overview.html
Other
1.95k stars 497 forks source link

[Bug] missing fields in importable/uploadable indexes #2523

Closed brokensound77 closed 1 year ago

brokensound77 commented 1 year ago

Update code to include all rule fields as well

terrancedejesus commented 1 year ago

Just backing up a bit here for context. The /releases folder is used to store prebuilt rule packages after they are built using the python -m detection_rules dev build-release command. Packages are stored based on the major and minor stack version they were built on, which references packages.yml. After building a package, several files are dropped into releases/x.x/extras. The files titled x.x-enriched-rules-index-importable.ndjson are created to allow rules to be uploaded as documents into ES if need be.

Problem: We were not adding the rule contents themselves to these documents, which should be added after to_api_format() is called for the JSON representation of the rule.

Before: ```sql { "hash":"36617ec8850ae04feba7b8e3f638dbd57f270919fc6fe0f7e8fd1ee32c922bb5", "source":"repo", "datetime_uploaded":"2023-03-01T23:28:34.462204", "status":"unmodified", "package_version":"8.8", "flat_mitre":{ "technique_names":[ ], "technique_ids":[ ], "sub_technique_names":[ ], "tactic_names":[ ], "tactic_ids":[ ], "sub_technique_ids":[ ] }, "relative_path":"apm/apm_403_response_to_a_post.toml" } ```

Solution: If we simply add rule_doc.update(**rule.contents.to_api_format()) within packging.Package.create_bulk_index_body() method it will update the ES doc to include the rule content themselves, which will ultimately be written to x.x-enriched-rules-index-importable.ndjson.

After (With the suggested solution): ```sql { "hash":"36617ec8850ae04feba7b8e3f638dbd57f270919fc6fe0f7e8fd1ee32c922bb5", "source":"repo", "datetime_uploaded":"2023-03-01T23:21:21.260487", "status":"unmodified", "package_version":"8.8", "flat_mitre":{ "tactic_names":[ ], "sub_technique_ids":[ ], "technique_ids":[ ], "sub_technique_names":[ ], "technique_names":[ ], "tactic_ids":[ ] }, "relative_path":"apm/apm_403_response_to_a_post.toml", "description":"A POST request to a web application returned a 403 response, which indicates the web application declined to process the request because the action requested was not allowed.", "name":"Web Application Suspicious Activity: POST Request Declined", "language":"kuery", "index":[ "apm-*-transaction*", "traces-apm*" ], "author":[ "Elastic" ], "query":"http.response.status_code:403 and http.request.method:post\n", "risk_score":47, "license":"Elastic License v2", "false_positives":[ "Security scans and tests may result in these errors. Misconfigured or buggy applications may produce large numbers of these errors. If the source is unexpected, the user unauthorized, or the request unusual, these may indicate suspicious or malicious activity." ], "severity":"medium", "rule_id":"a87a4e42-1d82-44bd-b0bf-d9b7f91fb89e", "timestamp_override":"event.ingested", "type":"query", "tags":[ "Elastic", "APM" ], "references":[ "https://en.wikipedia.org/wiki/HTTP_403" ], "related_integrations":[ { "package":"apm", "version":"^8.0.0" } ], "required_fields":[ { "name":"http.request.method", "type":"keyword", "ecs":true }, { "name":"http.response.status_code", "type":"long", "ecs":true } ], "version":101 } ```