mikecote
commented
3 years ago The @elastic/kibana-alerting-services will eventually have to provide a platform solution while respecting our RBAC model and based itself on the alerts as data story (https://github.com/elastic/kibana/issues/93728). If we were to use saved objects for this approach, below are some mentions I've seen brought up when discussing the schema (https://github.com/elastic/kibana/issues/93728).
- Ability to have an ECS schema (https://github.com/elastic/kibana/issues/93728#issuecomment-799901283)
- If this remains true, we would need to ensure the public contract of this saved object can work with such a structure. (ex: building visualizations on
event.kindinstead ofattributes.event.kind)
- If this remains true, we would need to ensure the public contract of this saved object can work with such a structure. (ex: building visualizations on
- Support for ILM (thread: https://github.com/elastic/kibana/issues/93728#issuecomment-800932963)
- There is an expectation of using ILM to delete old alert documents
- Index per rule type
- I've highly recommended the alert indices are created per rule type (index threshold, Geo threshold, etc.). Each has different parameter mappings that users may want to filter, sort and free-text search on (https://github.com/elastic/kibana/issues/93728#issuecomment-795332265).
- If saved objects support an index per SO type, we would also be able to revisit making rule parameters searchable (https://github.com/elastic/kibana/issues/50213#issuecomment-782130995).
- Migrations (thread: https://github.com/elastic/kibana/issues/93728#issuecomment-801048424)
- We'll need to create a new index for each release version that includes the latest mappings. The current assumption is additive mappings. The discussions around storing workflow fields make me re-think this.
- I'm not sure what the saved object migrations should do with existing documents. The existing documents are left in older indices and queried via index patterns across all these indices in the current implementation. It is assumed these indices will have a large (potentially huge) number of documents.
If we do create an index per saved object type, one thought would be to swap attributes for saved_object and allow custom fields at the root while storing saved object specifics within a nested object.
| Example: | Before | After |
|---|---|---|
| type | saved_object.type | |
| namespcae | saved_object.namespace | |
| rerefences | saved_object.references | |
| attributes.ruleType | ruleType | |
| attributes.params | params |
mappings before
``` { "type": { "type": "keyword" }, "namespace": { "type": "keyword" }, "references": { "type": "nested", "properties": { "type": { "type": "keyword" }, "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, "attributes": { "properties": { "ruleType": { "type": "keyword" }, "params": { "properties": { "foo": { "type": "keyword" } } } } } } ```mappings after
``` { "saved_object": { "type": { "type": "keyword" }, "namespace": { "type": "keyword" }, "references": { "type": "nested", "properties": { "type": { "type": "keyword" }, "id": { "type": "keyword" }, "name": { "type": "keyword" } } }, }, "ruleType": { "type": "keyword" }, "params": { "properties": { "foo": { "type": "keyword" } } } } ```This allows saved object properties to be stored at the root (ex: ECS fields) while still having a place to store saved object specific fields.
elasticmachine
oatkiller
pgayvallet
kobelb
Starting in 8.0 with the formal implementation of system-indices, it will no longer be possible to use applications like Discover, Visualize, Canvas, Maps, Machine Learning, etc. to analyze saved-objects. While it's technically possible to create visualizations, etc. against any of Kibana's system indices prior to 8.0, it is not supported. Kibana's system indices, and the documents that are stored in these indices, are an implementation detail of Kibana. As such, these indices and the documents that are stored in them are subject to breaking changes in minors. Additionally, Kibana enforces its own entity model and role-based access control. Accessing Kibana's system indices using Elasticsearch APIs side-steps all of these protections.
However, this is a rather unfortunate situation because applications like Discover, Visualize, Lens etc. offer incredibly powerful tools for free-form analysis. This has become a recurring sticking point when discussing how to architect certain features because this free-form analysis is so compelling.
While we've found some interesting short-term solutions, like the one that we're using for Alerts as data, it doesn't fulfill all of our needs. An ideal solution would allow us to perform free-form analysis using tools like Discover, Visualize, Lens, etc. against saved-objects while respecting Kibana's entity model and role-based access control.