This PR adds support for import exceptions lists into Kibana. While this PR is not directly dependent on export support from https://github.com/elastic/detection-rules/pull/3869, it may make it easier to test. In short, this uses the same rules API import endpoint and converts the TOMLException Objects loaded in from the Toml Exception files to the appropriate API format and includes them along with the rules upload. This matches format the rules and exceptions are provided in Kibana's API export for reference. Additionally, this PR allows you to output exceptions to an ndjson file instead of directly importing to Kibana using the export-rules-from-repo command.
Note: This will require an update to the Kibana Module.
Expected new output if you attempt to import a rules with exceptions list.
Testing this requires a rule and TOML Exceptions file.
Example TOML Rule File
```toml
[metadata]
creation_date = "2024/07/01"
list_name = "Exceptions for rule - Test Exception List"
rule_ids = ["7c22a9d2-5910-4da2-92af-7ff7481bd0f7"]
rule_names = ["Test Exception List"]
updated_date = "2024/07/10"
[[exceptions]]
[exceptions.container]
description = "Exception list containing exceptions for rule with id: 77260f65-d17e-468b-8fe9-305048404e95"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
name = "Exceptions for rule - Test Exception List"
namespace_type = "single"
tags = ["not_default_rule_exception_list"]
type = "rule_default"
[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "8dffbaa5-8068-4b7c-b4cf-ef17f627dc00"
name = "TestShareItem"
namespace_type = "single"
tags = []
type = "simple"
[[exceptions.items.entries]]
field = "Effective_process.pid"
type = "match"
operator = "included"
value = "1"
[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "970945dd-71d5-4128-89a8-7e8689326a19"
name = "Pid not One"
namespace_type = "single"
tags = []
type = "simple"
[[exceptions.items.entries]]
field = "Effective_process.pid"
type = "match"
operator = "included"
value = "1"
[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "d6a0e21c-bf41-4758-a522-cca5df3a2332"
name = "FakeRoot"
namespace_type = "single"
tags = []
type = "simple"
[[exceptions.items.entries]]
field = "process.name"
type = "match"
operator = "included"
value = "FakeRootUpdatedFour"
```
Example TOML Exceptions File
```toml
[metadata]
creation_date = "2024/07/01"
rule_id = "77260f65-d17e-468b-8fe9-305048404e95"
rule_name = "Test Exception List"
updated_date = "2024/07/01"
[[exceptions]]
[exceptions.container]
description = "Exception list containing exceptions for rule with id: 77260f65-d17e-468b-8fe9-305048404e95"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
name = "Exceptions for rule - Test Exception List"
namespace_type = "single"
tags = ["default_rule_exception_list"]
type = "rule_default"
[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "970945dd-71d5-4128-89a8-7e8689326a19"
name = "Pid not One"
namespace_type = "single"
tags = []
type = "simple"
[[exceptions.items.entries]]
field = "Effective_process.pid"
type = "match"
operator = "included"
value = "1"
[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "d6a0e21c-bf41-4758-a522-cca5df3a2332"
name = "FakeRoot"
namespace_type = "single"
tags = []
type = "simple"
[[exceptions.items.entries]]
field = "process.name"
type = "match"
operator = "included"
value = "FakeRootUpdated"
```
Once you have both the exceptions file and toml rule file you can run the following command to import the rules and exception list (the -e flag is the required flag to include exceptions).
To test importing to an ndjson file instead of importing to Kibana use the following command. Note you will need to have rules with exception(s) in toml format already for this to export.
Issues
https://github.com/elastic/detection-rules/issues/3674
Summary
Note: Please consider merging https://github.com/elastic/detection-rules/pull/3869 first as it will make this PR notably smaller.
This PR adds support for import exceptions lists into Kibana. While this PR is not directly dependent on export support from https://github.com/elastic/detection-rules/pull/3869, it may make it easier to test. In short, this uses the same rules API import endpoint and converts the TOMLException Objects loaded in from the Toml Exception files to the appropriate API format and includes them along with the rules upload. This matches format the rules and exceptions are provided in Kibana's API export for reference. Additionally, this PR allows you to output exceptions to an ndjson file instead of directly importing to Kibana using the
export-rules-from-repocommand.Note: This will require an update to the Kibana Module.
Expected new output if you attempt to import a rules with exceptions list.
Details
```shell detection-rules on 3674-frdac-add-import-support [!?] is v0.1.0 via v3.12.4 (detection-rules-build) on eric.forte took 9s ❯ python -m detection_rules kibana import-rules --overwrite -e -f custom_rules/rules/test_exception_list.toml Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ 1 rule(s) successfully imported - 7c22a9d2-5910-4da2-92af-7ff7481bd0f7 ```
Kibana API Docs for Reference
https://www.elastic.co/guide/en/security/current/exceptions-api-get-item.html https://www.elastic.co/guide/en/security/current/exceptions-api-create-container.html
Testing
Importing Steps
To test this code:
Testing this requires a rule and TOML Exceptions file.
Example TOML Rule File
```toml [metadata] creation_date = "2024/07/01" list_name = "Exceptions for rule - Test Exception List" rule_ids = ["7c22a9d2-5910-4da2-92af-7ff7481bd0f7"] rule_names = ["Test Exception List"] updated_date = "2024/07/10" [[exceptions]] [exceptions.container] description = "Exception list containing exceptions for rule with id: 77260f65-d17e-468b-8fe9-305048404e95" list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9" name = "Exceptions for rule - Test Exception List" namespace_type = "single" tags = ["not_default_rule_exception_list"] type = "rule_default" [[exceptions.items]] comments = [] description = "Exception list item" list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9" item_id = "8dffbaa5-8068-4b7c-b4cf-ef17f627dc00" name = "TestShareItem" namespace_type = "single" tags = [] type = "simple" [[exceptions.items.entries]] field = "Effective_process.pid" type = "match" operator = "included" value = "1" [[exceptions.items]] comments = [] description = "Exception list item" list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9" item_id = "970945dd-71d5-4128-89a8-7e8689326a19" name = "Pid not One" namespace_type = "single" tags = [] type = "simple" [[exceptions.items.entries]] field = "Effective_process.pid" type = "match" operator = "included" value = "1" [[exceptions.items]] comments = [] description = "Exception list item" list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9" item_id = "d6a0e21c-bf41-4758-a522-cca5df3a2332" name = "FakeRoot" namespace_type = "single" tags = [] type = "simple" [[exceptions.items.entries]] field = "process.name" type = "match" operator = "included" value = "FakeRootUpdatedFour" ```
Example TOML Exceptions File
```toml [metadata] creation_date = "2024/07/01" rule_id = "77260f65-d17e-468b-8fe9-305048404e95" rule_name = "Test Exception List" updated_date = "2024/07/01" [[exceptions]] [exceptions.container] description = "Exception list containing exceptions for rule with id: 77260f65-d17e-468b-8fe9-305048404e95" list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9" name = "Exceptions for rule - Test Exception List" namespace_type = "single" tags = ["default_rule_exception_list"] type = "rule_default" [[exceptions.items]] comments = [] description = "Exception list item" list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9" item_id = "970945dd-71d5-4128-89a8-7e8689326a19" name = "Pid not One" namespace_type = "single" tags = [] type = "simple" [[exceptions.items.entries]] field = "Effective_process.pid" type = "match" operator = "included" value = "1" [[exceptions.items]] comments = [] description = "Exception list item" list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9" item_id = "d6a0e21c-bf41-4758-a522-cca5df3a2332" name = "FakeRoot" namespace_type = "single" tags = [] type = "simple" [[exceptions.items.entries]] field = "process.name" type = "match" operator = "included" value = "FakeRootUpdated" ```
Once you have both the exceptions file and toml rule file you can run the following command to import the rules and exception list (the
-eflag is the required flag to include exceptions).python -m detection_rules kibana import-rules --overwrite -e -f custom_rules/rules/test_exception_list.tomlTesting
export-rules-from-repoTo test importing to an ndjson file instead of importing to Kibana use the following command. Note you will need to have rules with exception(s) in toml format already for this to export.
Example Output
```shell ❯ python -m detection_rules export-rules-from-repo -e Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ Exported 16 rules into /home/forteea1/Code/clean_mains/detection-rules/exports/20240707T161019L.ndjson ```
Example ndjson
[20240707T161019L.ndjson.txt](https://github.com/user-attachments/files/16120645/20240707T161019L.ndjson.txt)