[DaC] [FR] End to end testing and feedback improvements

[eric-forte-elastic]

Pull Request

Issue link(s):

Summary - What I changed

Please merge https://github.com/elastic/detection-rules/pull/3955 into this PR before merging to main

This PR adds the ability to specify a config variable to auto generate a custom schema file for any missing non-ecs fields. To trigger this behavior add the following line to your _config.yaml

auto_gen_schema_file: <path_to_schema_json_file>

I addition, there are a number of other minor fixes/updates to the import-rules-to-repo process including:

• Added error skipping for rule-prompt and import-rules-to-repo.
• Modified --require-only behavior in import-rules-to-repo to populate all fields but only prompt for required ones.
• Improved extract_error_field function to handle non alpha-numeric characters.
• Introduced CLI argument to remove null values from TOML rule files.
• Added support for rule.actions.frequency.throttle field, required in Kibana.
• Implemented a _add_known_nulls function to handle null fields in TOML files.
• Added support for action connectors.
• Enhanced rule prompt with skip errors support.
• Included rule name in error.txt output for easier debugging.
• Updated config file initialization to include directories for actions, action connectors, and exceptions.
• Added helper functions for auto-generating schema.
• Made Exception List optional within an Exception Container.
• Cleaned up exceptions and action connectors build and parse functions.
• Updated Generic Loader to support Action Connectors and fixed a bug.
• Added support for endpoint integration.
• Fixed formatting issue with single lines ending in a double quote in a TOML file.
• Improved formatting support for threat query, setup, and description.
• Fixed a bug in extract_error_field function related to quotes around the field value.
• Fixed version printing bug in all_versions function.
• Updated rule name definitions for customer rules to provide less restriction.
• Implemented minor bug fixes.

How To Test

Autogen Custom Schema

To test this, first add the appropriate config variable. Next, run view-rule on a rule that has fields that are not currently present in a schema. The rule should validate successfully and your schema file should be updated.

Example: python -m detection_rules view-rule custom_rules/rules/dac_demo_dev_rule_1.toml

Details

```toml [metadata] creation_date = "2024/07/29" maturity = "production" updated_date = "2024/07/29" [rule] actions = [] author = ["DAC User"] description = "Test Rule" enabled = true exceptions_list = [] false_positives = [] from = "now-6m" index = ["the-best-integration-ever*"] interval = "5m" language = "eql" max_signals = 100 name = "DAC Demo Dev Rule 1" references = [] risk_score = 47 risk_score_mapping = [] rule_id = "794d2fc0-ecd0-4963-99da-fd587666b80d" setup = "Test Setup" severity = "medium" severity_mapping = [] tags = [] threat = [] to = "now" type = "eql" query = ''' process where host.os.type.fakeData == "linux" and process.name.okta.thread == "updated" ''' [[rule.related_integrations]] package = "endpoint" version = "^8.2.0" [[rule.required_fields]] ecs = true name = "host.os.type" type = "keyword" [[rule.required_fields]] ecs = true name = "process.name" type = "keyword" ```

This will occur in any command where rule contents are validated against a schema, including import-rules-to-repo, kibana export-rules and others.

Additionally added, is the --skip-errors flag to import-rules-to-repo to follow similar functionality from this same flag for the kibana export-rules command. With this flag one should expect output similar to the following when importing rules.

1347 results exported
1099 rules converted
248 exceptions exported
49 errors saved to /home/forteea1/Code/clean_mains/detection-rules/custom_rules/rules/_errors.txt

Also, rule names will no longer be validated against our naming conversions for custom rules. Any valid string name will now validate.

Testing support for a required field rule.actions.frequency.throttle

Use the following example rule file to import from ndjson and export back to a kibana instance. You will get a connector id is missing error but this is expected and as far of an implmentation of actions as this PR expects. If the desired result is not working you would see a throttle error instead.

test_action_export.ndjson.txt

Example Rule

```toml [metadata] creation_date = "2024/08/03" maturity = "development" updated_date = "2024/08/03" [rule] author = ["INFOSEC_TEST"] description = "Test" filters = [] from = "now-360s" index = [ "apm-*-transaction*", "auditbeat-*", "endgame-*", "filebeat-*", "logs-*", "packetbeat-*", "traces-apm*", "winlogbeat-*", "-*elastic-cloud-logs-*", ] interval = "5m" language = "kuery" max_signals = 100 name = "TestActionRule" note = "None" revision = 3 risk_score = 21 rule_id = "e818bf1b-dcc8-4746-80fa-a155a94a7f6b" setup = "None" severity = "low" to = "now" type = "query" version = 1 query = ''' process.command_line : "FakeRoot" ''' [[rule.actions]] action_type_id = ".email" group = "default" id = "elastic-cloud-email" uuid = "c405d3e6-f47d-4ee2-bfc2-9fa786051c66" [rule.actions.frequency] notifyWhen = "onActiveAlert" summary = true [rule.actions.params] message = "Rule {{context.rule.name}} generated {{state.signals_count}} alerts" subject = "TestEmail" to = ["eric.forte@elastic.co"] [[rule.actions]] action_type_id = ".webhook" group = "default" id = "478b2165-83fb-480d-8a4a-bb47cfcafd4c" uuid = "fd2c83b6-414b-4e31-939a-27f399c06203" [rule.actions.frequency] notifyWhen = "onActiveAlert" summary = true [rule.actions.params] body = "{}" [rule.meta] from = "1m" kibana_siem_app_url = "https://dev-deployment-2c684a.kb.us-central1.gcp.cloud.es.io:9243/s/infosec/app/security" ```

Note you may have to download this to watch the example, needed it to be mp4 due to length and not sure how well github renders mp4.

https://github.com/user-attachments/assets/1d2ed2a4-7066-4ca9-a117-5a6442a20e22

Checklist

• [ ] Added a label for the type of pr: bug, enhancement, schema, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• [ ] Added the meta:rapid-merge label if planning to merge within 24 hours
• [ ] Secret and sensitive material has been managed correctly
• [ ] Automated testing was updated or added to match the most common scenarios
• [ ] Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[protectionsmachine]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• [ ] Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• [ ] Include additional context or screenshots.
• [ ] Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• [ ] Code follows established design patterns within the repo and avoids duplication.
• [ ] Code changes do not introduce new warnings or errors.
• [ ] Variables and functions are well-named and descriptive.
• [ ] Any unnecessary / commented-out code is removed.
• [ ] Ensure that the code is modular and reusable where applicable.
• [ ] Check for proper exception handling and messaging.

Testing

• [ ] New unit tests have been added to cover the enhancement.
• [ ] Existing unit tests have been updated to reflect the changes.
• [ ] Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• [ ] Validate that any rules affected by the enhancement are correctly updated.
• [ ] Ensure that performance is not negatively impacted by the changes.
• [ ] Verify that any release artifacts are properly generated and tested.

Additional Checks

• [ ] Ensure that the enhancement does not break existing functionality.
• [ ] Review the enhancement with a peer or team member for additional insights.
• [ ] Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• [ ] Confirm that all dependencies are up-to-date and compatible with the changes.