search

elastic / detection-rules

https://www.elastic.co/guide/en/security/current/detection-engine-overview.html
Other
1.98k stars 505 forks source link

[Discussions] DaC Community Feedback #3298

Open Mikaayenson opened 12 months ago

Mikaayenson commented 12 months ago

Summary

From the beginning of the Elastic detection-rules repo, it not only contained the Elastic prebuilt detection rules files, but also additional tooling for detection rule management like a suite of tests, CLI commands, and automation scripts used by the Elastic Threat Research and Detection Engineering (TRADE) team.

Elastic TRADE team has been following Detections as Code (DaC) practices for years, supporting development and release processes for Elastic prebuilt detection and endpoint rules.

With DaC becoming more mainstream and continuing with our commitment to openness, we are working on making it easier for users to kick-start their own DaC process using Elastic detection-rules repo for their rules management. This DaC expansion will build upon prior detection rules features to provide an end-to-end experience for detection engineers.

As we continue planning and organizing upcoming detections as code sprints, we've started capturing feedback from the community. This issue is designed to organize any subsequent feedback in a common place. Your feedback is important as it will help us prioritize and shape how we refactor our detection-rules DaC experience.

What to Expect

We will be posting updates about work happening on the DaC topic in this issue, so make sure to subscribe to get those updates, try out the changes, and let us know what you think.

How to Contribute

  1. 👋 If you would like to chime in, please feel free to add a comment below!
  2. Check ⬇️ the existing issues and 👍 if they address your need.
  3. If you feel anything is missing, feel free to open an issue and let us know so that we can track appropriately!

Existing Related Issues

leandrojmp commented 8 months ago

Just some feedback as we tried to use the detection rules script to automate the management of our custom rules, but ended up writing our own tool because it didn't work for our use cases.

The main issue was that we do not use index patterns for our data, we use data views which is the recommended way now, and since the index pattern API is deprecated we assumed that using index patterns is also depracted.

But the detection-rules script does not work with data views, it uses index patterns only.

Another big issue is that we have a lot non-ecs fields, which also does not work with detection-rules,

So because of those two main issues, we needed to write a similar tool to automate the deploy of our custom rules.

brokensound77 commented 8 months ago

Thank you @leandrojmp - this is great feedback. We will be updating the rule schema to allow data_view_id shortly, which allows use of index or data views in all rules.

As far as the non-ecs, rules are allowed to used any field values, we just validate stricter for our built in rules based on the corresponding datasource schema. This is what you are likely referring to. We are also in the process of decoupling the validation and schemas of rules to make this simpler.

Today, you can get around this by maintaining the non-ecs-schema, however, with the upcoming changes, user rules will be completely decoupled, allowing more granular control of what validations occur and what schemas are used (in case you want to bring your own)

brokensound77 commented 8 months ago

here is the PR to support data views: elastic/detection-rules#3510

brokensound77 commented 7 months ago

👋 just dropping in to provide an update on progress to supporting DAC more seamlessly.

Updates

In elastic/detection-rules#3466, the kibana module was updated to use the newer APIs for rule management. There were also 2 commands added for simplicity. This merged to main

In elastic/detection-rules#3407, there was a lot introduced, including decoupling significant components of the repo for easier adoptability. This was merged to a feature branch (DAC-feature). We consider this early alpha and ready for users to start testing.

We realize a significant outcome of this is simply conveying our findings and best practices for implementing DAC on these rules, particularly with the built-in components of this repo. So, we started work on the DAC reference. As with any as code, there is tons of variation on viable approaches - this reference aims to enable users to implement DAC with, without, or as a hybrid partially using the repo.

@Mikaayenson and I also recently gave a talk at BSidesOK about leveraging this, where we released the reference. Here are the slides.

Next steps

As stated, we consider this in early alpha and are open to feedback from early adopters. Additionally, we have other smaller snippets of planned work upcoming to further simplify this.

If you have any feedback or questions along the way, the best place to reach us is in the public slack channel for security-rules-dac

ar3diu commented 7 months ago

I think dac-reference is the most detailed documentation on this subject that I have seen so far, which is great because for many teams the concept and implementation methods were not always clear. Thanks for putting this together!

brokensound77 commented 7 months ago

I think dac-reference is the most detailed documentation on this subject that I have seen so far, which is great because for many teams the concept and implementation methods were not always clear. Thanks for putting this together!

Thanks for the awesome feedback! That was our hope all along, as we realized through this process that just understanding is likely more valuable than any code support we could add.

Keep an eye on it as we will continue to expand on it as we build this out

ar3diu commented 6 months ago

Will you consider actions like disable-rule or delete-rule for the kibana command in detection_rules? For ex, when deprecating a rule, usually you will want to at least disable the rule in Kibana (if not to delete it completely).

brokensound77 commented 6 months ago

Will you consider actions like disable-rule or delete-rule for the kibana command in detection_rules? For ex, when deprecating a rule, usually you will want to at least disable the rule in Kibana (if not to delete it completely).

Thanks for the suggestion @ar3diu. This can be achieved by leveraging the bulk action Kibana commands as documented here. You can customize functionality to whatever fits your requirements

botelastic[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

eric-forte-elastic commented 4 months ago

Update 7/21/24

We are preparing to merge our DAC-feature branch to main which will address a number of the issues outlined in the summary. This issue will remain open until we both complete the merge and circle back to any remaining issues that may not have been fully addressed by our merge to main.

Once this issue does close, DaC features will be considered supported. If you have additional features you would like to see, please use the feature request issue template and we will triage them similarly to non DaC feature requests. Thank you!

eric-forte-elastic commented 3 months ago

Update 8/7/24

The DAC-feature branch has now been merged to main :tada: and the community issues from this issue's summary have now been addressed. Check out the blog and our documentation :rocket: .