System-owned resources (policies, templates, jobs, watches, etc)

jaymode commented 3 years ago

As part of the system indices effort, #50251, the topic of "system-owned resources" has come up a few times in relation to items other than indices. An "system-owned resource" (for lack of a better term) is a resource such as a Watch, ILM policy, ML Job, Template, or Ingest Pipeline that is provided by the installed software, either by Elasticsearch itself or a plugin. These resources may be necessary for proper functioning of system features or provided in an attempt to simplify operation for the user.

Examples:

An ILM policy that would be configured for reporting indices but abstracted away by a UI, which keeps the user from needing to know ILM.
Ingest pipeline that a solution or stack component uses to ingest data into a system index.
Watches and ML jobs for stack/solution monitoring.
Reserved users and roles such as the elastic user and superuser role.
System composable templates as described in https://github.com/elastic/elasticsearch/issues/65957

Today, features that wish to provide these types of items manually create these items, which are subject to user modification and deletion. While we are attempting to provide a more resilient system that prevents interaction with data in system indices, modification of these other items could still affect the operation of the system and therefore we should consider whether there is work we want to do to provide protection.

This issue is opened for discussion on how we should handle these types of items moving forward. Some items worth discussing might include:

How do we expose these items to the user? Are all system owned resources hidden by default?
Do we allow these items to be modified?
How do we prevent collisions with existing user resources?
Are there items that should never be system owned?
Should we allow disabling system owned resources completely? (For example, reserved users can be disabled)

elasticmachine commented 3 years ago

Pinging @elastic/es-core-features (Team:Core/Features)

elasticmachine commented 3 years ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

mbudge commented 3 years ago

We are currently using legacy templates to add our custom fields to the SIEM indexes. This is so external systems and query the index + we can display fields to dashboards for analysts and reporting. We do this because dynamic templates are disable on SIEM indexes to restrict the number of fields in the mapping. If legacy templates are discontinued in elasticsearch v8, we will need a way to customise the SIEM index templates using the new index + component template system.

gwbrown commented 3 years ago

Hi @mbudge, that sounds like a separate issue from that described above. The subject of this issue is resources which shouldn't be modified because doing so may cause problems in the operation of a system component - typically that's components built into Elasticsearch, such as the template for Watcher's history indices, which the system expects to always be the same. The SIEM indices don't fall into that category, and you should be able to modify the composable templates for SIEM's indices in a similar way to modifying the legacy templates.

If you require help doing so, there's an active community in the forums that should be able to help get an answer to your question.

elasticsearchmachine commented 10 months ago

Pinging @elastic/es-data-management (Team:Data Management)

elastic / elasticsearch

System-owned resources (policies, templates, jobs, watches, etc) #66413