Support toml lint for investigate transforms

[shashank-elastic]

Pull Request

Issue link(s): https://github.com/elastic/detection-rules/issues/4033

Summary - What I changed

• Added a code condition to handle lists of dictionaries in toml linting.
• Also added toml-lint commands to test_cli.bash script
• Update the test_toml.json to include transform fields

How To Test

• Try to run toml-lint on the rule with transform.investigate fields python -m detection_rules toml-lint -f rules/windows/command_and_control_common_webservices.toml
  No Errors Found

❯ python -m detection_rules toml-lint -f rules/windows/command_and_control_remote_file_copy_powershell.toml       
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

TOML file linting complete
(.venv) 
detection-rules on  issue-4033 [$!?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ 

• Try toml-lint on any existing rule without investigate transforms to check existing behaviour
  Existing Functionality is not broken

❯ python -m detection_rules toml-lint -f rules/linux/persistence_setuid_setgid_capability_set.toml         
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

TOML file linting complete
(.venv) 
detection-rules on  issue-4033 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ 

No changes detected

Checklist

• [x] Added a label for the type of pr: bug, enhancement, schema, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• [ ] Added the meta:rapid-merge label if planning to merge within 24 hours
• [ ] Secret and sensitive material has been managed correctly
• [x] Automated testing was updated or added to match the most common scenarios
• [ ] Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[protectionsmachine]

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

• [x] Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
• [ ] Include additional context or details about the problem.
• [ ] Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

• [x] Code follows established design patterns within the repo and avoids duplication.
• [x] Code changes do not introduce new warnings or errors.
• [ ] Variables and functions are well-named and descriptive.
• [ ] Any unnecessary / commented-out code is removed.
• [ ] Ensure that the code is modular and reusable where applicable.
• [ ] Check for proper exception handling and messaging.

Testing

• [ ] New unit tests have been added to cover the bug fix or edge cases.
• [ ] Existing unit tests have been updated to reflect the changes.
• [x] Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
• [ ] Validate that any rules affected by the bug are correctly updated.
• [ ] Ensure that performance is not negatively impacted by the changes.
• [ ] Verify that any release artifacts are properly generated and tested.

Additional Checks

• [x] Ensure that the bug fix does not break existing functionality.
• [ ] Review the bug fix with a peer or team member for additional insights.
• [ ] Verify that the bug fix works across all relevant environments (e.g., different OS versions).
• [ ] Confirm that all dependencies are up-to-date and compatible with the changes.

[Mikaayenson]

@shashank-elastic can you share an example of it working with a rule that has the investigation fields?

[shashank-elastic]

@shashank-elastic can you share an example of it working with a rule that has the investigation fields?

@Mikaayenson Already updated in "How to Test" part of the PR :)

[Mikaayenson]

@shashank-elastic can you share an example of it working with a rule that has the investigation fields?

@Mikaayenson Already updated in "How to Test" part of the PR :)

Looks good. I think we should update the test_toml.json file so the new fields are added to the unit test TestRuleTomlFormatter

[shashank-elastic]

@shashank-elastic can you share an example of it working with a rule that has the investigation fields?

@Mikaayenson Already updated in "How to Test" part of the PR :)

Looks good. I think we should update the test_toml.json file so the new fields are added to the unit test TestRuleTomlFormatter

@Mikaayenson Done

[shashank-elastic]

@Mikaayenson make test-cli execution completed The output is pretty huge and I have truncated for better readability

Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Building integration schemas...
processing endpoint
final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-schemas.json.gz
Detection-rules CLI tests completed!