elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.8k stars 8.19k forks source link

[Security Solution] Improvements to huge amount of fields generating scripts #162042

Open e40pud opened 1 year ago

e40pud commented 1 year ago

Describe the feature:

In 8.10 we added scripts to allow us to generate huge amount of fields in huge amount of indices with the ability to randomly make some of the fields unmapped. This helps security solution team to test UI (e.g. rule's exceptions page) where we use Field Caps which can be slow in certain cases.

Improvements:

@michaelolo24 suggested a few improvements which will make scripts more handy. Here is the summary of those:

  1. It could be handy to have a "generate data" script that would take that mapping and generate x number of documents that have a value for every single field.
  2. It could be worthwhile combining the mappings:load into the mappings:generate script thereby just auto-loading it after it's created. To replicate the current mappings:generate behavior, we can then provide an arg like --mappings-only or similar to create the mappings without loading it.
  3. Generating docs then can be something like --numDocs to autogenerate docs with the fields from the mappings populated.
elasticmachine commented 1 year ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine commented 1 year ago

Pinging @elastic/security-solution (Team: SecuritySolution)

yctercero commented 1 month ago

@marshallmain did your work utilize these scripts at all? Wondering if these enhancements now exist in your scripts.