Closed SHolzhauer closed 1 month ago
I think this could use the
Hey @SHolzhauer I've been talking with @brokensound77 and had some discussions in slack and I think we have an option to sync between this repo to and from Kibana. Here's my proposal for the high level workflow and design:
kibana push
command. Point this command to a folder and it will update or create all rules in Kibana to match. I'm thinking that we can add a --dry-run
flag so you can get a preview of all the changeskibana pull
command. Point this to a folder and it will retrieve all the Kibana rules and create new files or update existing. This would also have --dry-run
kibana prune
command. This would remove rules that aren't in the other location. prune --local
will remove lcoal copies that aren't in Kibana. --remote
does the opposite. --dry-run
makes sense here tookibana sync [--dry-run]
to do a two way sync. The commands will automatically store a JSON file at the root of the specified folder to track the local version of a rule, and what the version of the rule is in Kibana when it was last synced. We can autosync rules that aren't in conflict.How's that plan sound to you?
@rw-access LGTM
Couple of notes:
prune
command, what would happen and/or should it be supported to use both flags? --local
and --remote
. This could be a cleaning step for example.On the --dry-run
flag. Should it maybe be moved up 1 level? So the kibana
command gets it. Just a thought but as it will be shared across multiple commands might be a better place for it.
Great point about the state file. CI/CD solutions generally don't modify files in version control so that's problematic. I think a Kibana saved object might be a more appropriate place to store that state, since it's persistent and outside of VCS.
For Elastic provided rules, we don't have many options because the rules themselves are actually immutable. So we can't change them via API even if we wanted to. We're also working on a more native upgrade mechanism in Kibana, hopefully as simple as an "upgrade rules" button that can happen at any time, without needing a stack update. You can always layer exceptions on top of the rules when you want deviations in the logic but want to remain subscribed to the rule.
Thinkin about it, wouldn't you be able to use the updated_date
field? Shouldn't that one be comparable to the date in kibana?
Hey, @SHolzhauer haven't updated you for a while. This is still on my radar.
We are planning some refactors to the repository internals, and I think that we'll unlock some nice new functionality to make the rules/
directory completely configurable. So you can have your rules live outside the repo, or in a fork or whatever you decide. So then you just point the repository to your configuration file and it'll find everything, can run tests on that, etc.
I think our refactor will span a month or so, and hopefully this becomes a natural solution that emerges easily from there. 🤞. I think this is a really cool use case, and something that I think will be a huge value add for organizations that are very CI/CD leaning for rules management and "detections as code" thinking.
@rw-access thank you for the update, looking forward to it!
Hey @rw-access, I was curious if you had any update regarding this issue?
We are starting to notice more and more bugs with our current work-around/implementation and can really use this feature :smiley:
hey @SHolzhauer, yeah here's some updates for you and your team:
main
, 7.13
, 7.14
, ... branches can all have the same tooling and validation in place, just different rulesgiven all of that, my hope is that you can specify where your rules/
directory lives with an environment variable. like DETECTION_RULES_DIR=../sholzhauer-rules
. and then everything in the repo would use that directory instead of our rules/
directory.
within that directory, you'd put all of the configuration. it would have version.lock.json
, config.toml
, and whatever subfolders you want for your rules. in the config file, you could specify what stack version you want to validate your rules against. that way we can make sure that you're not trying to use rules that require 7.14 features but you only have a 7.13 stack (for example).
and if you want, you could put that directory inside your own git repo or VCS it however you like. then you don't have to worry about forking this repo and dealing with merge conflicts.
Sounds promising, looking forward.
And thank you for the blazing fast response!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.
Just wanted to check if this functionality is still planned?
Based on recent discussions, as we plan to move forward, we will want to support custom rules only (not prebuilt rules for now).
This is now supported for custom rules via https://github.com/elastic/detection-rules/pull/3889. Please feel free to re-open this issue or create a new one if you run into any trouble. Thanks!
Is your feature request related to a problem? Please describe.
Within organizations there is a desire to "version control" the rules/code in use. SOC teams want to organize and make documented, incremental changes to their custom detection-rules and have a "back-up" of their code in case of data loss.
This is an extension of #17 as discussed in the issue.
Describe the solution you'd like
We need a way to
This process should
This FR should not
Possible command could be:
python -m detection_rules kibana git-sync
Additional context
17