Open marshallmain opened 3 years ago
Pinging @elastic/security-solution (Team: SecuritySolution)
Pinging @elastic/apm-ui (Team:apm)
For awareness, proposal 2 is similar to what we implemented in in Fleet recently, which installs a @custom
component template for every index template we install: https://github.com/elastic/kibana/pull/101769
The initial UX is a bit clunky as users will need to go into Stack Management and apply their custom settings/mappings/ILM policies, etc. But the idea is that in the future we can iterate towards a custom UI that is "built in" into Fleet/Integrations that modifies these custom component templates in the background.
Pinging @elastic/security-detections-response (Team:Detections and Resp)
Hi, we are also interested in this issue. It's very frustrating to work with the Alerts overview from the Security Solution and filter out fields, which then result in nothing happening or all alerts being filtered out because the field "does not exist".
@marshallmain Is this issue still valid?
We have had success using run-time mappings to work around this issue. Thank you to @marshallmain for the tip in the slack community.
We should clarify something though. The description above suggests this only impacts custom mappings. Since Elastic has not created all standard ECS mappings in the .alerts index, this leaves it to the customer to do so. Some customers will truly have custom mappings, but in our situation, we are just trying to apply the standard ECS mappings because they are not there natively.
Pinging @elastic/response-ops (Team:ResponseOps)
hi,
We'd also like to make use to custom field in the .siem-signals indices. Our Analists mainly want to use it for dashboarding/ filters in searches.
Advanced users of the Security Solution have a history of wanting additional custom fields mapped in the .siem-signals indices. Even though the Security Solution copies the entire source document into the generated alert, fields that were mapped in the original document may not be mapped in the alert document, preventing users from queries on .siem-signals from using those fields effectively.
Users have sometimes seen this limitation and decided to augment the .siem-signals index template with their own custom fields. However, this causes problems for them when they upgrade Kibana as we often add additional fields to the template - when we add more fields, we overwrite the existing template and in the process wipe out any changes that users made to the template. The guidance we typically give to users in this situation right now is that (1) what they're doing is not supported, so it's inherently risky, and (2) if they still want to continue augmenting fields in a risky way, they may have better success by creating a separate template and using the
order
parameter (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates-v1.html#multiple-templates-v1) to apply their custom fields separately from the built-in mappings shipped by the Security Solution. However, theorder
parameter is going away with index templates v2 and essentially replaced by component templates.To avoid situations where users modify the .alerts index templates directly and are surprised when Kibana upgrades clobber their customizations, it would be nice to decide on and document a way for advanced users to augment the mappings. 2 solutions have been proposed so far:
cc @oatkiller @tsg @MikePaquette @dgieselaar @jasonrhodes @weltenwort