search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.62k stars 24.64k forks source link

[Discuss] Add wildcard privileges for dot-prefixed Fleet indices to kibana_system and elastic/fleet-server #80332

Open joshdover opened 2 years ago

joshdover commented 2 years ago

As Elastic packages evolve, additional dot-prefixed, hidden indices and data streams are regularly being added that are intended for use exclusively by application code and are not meant to be directly consumed by end-users. Each time one gets added, we have to add additional privileges to the kibana_system role and elastic/fleet-server service account to allow Kibana to install packages and to allow elastic/fleet-server to create valid API keys so that agents can ingest data.

A recent example comes from the Endpoint package which recently added a .logs-endpoint.action.responses-* data stream. This required changes to both kibana_system and elastic/fleet-server to update their privileges:

We could make this less burdensome by granting both of these roles/service accounts the index privileges they need to a broader pattern like .logs-*-* and .metrics-*-* just like we do for the normal non-hidden indices.

Related to https://github.com/elastic/kibana/issues/116715

Additionally, we should probably update all of these non-dot-prefixed index privileges to be logs-*-* and metrics-*-* instead of logs-* and metrics-*. This could be considered as a separate issue

elasticmachine commented 2 years ago

Pinging @elastic/es-security (Team:Security)

joshdover commented 2 years ago

cc @ywangd since you've been helping us out with these changes.

bytebilly commented 2 years ago

Thanks @joshdover, in general that would make sense to me, but we probably need to verify that we are not unintentionally create security issues in the future.

Is there any possible scenario that we can think of where indices in these patterns (e.g. .logs-* and .metrics-*) will be created but they won't need to be accessed by Fleet or Kibana? IIRC these are not system indices, so potentially users can create their own custom indices matching the pattern.

The main difference that I see with logs-*, metrics-*, and traces-* is that these are well-known and documented as "reserved" for Observability use, so it's more unlikely that there may be confusion about their access.

ruflin commented 2 years ago

It seems we kind of accidentally started to support these .logs-*-* indices. Should we make them official supported by also loading the logs template in addition for .logs. Same for metrics etc.?

ruflin commented 2 years ago

@jpountz I'm sure you have an opinion / thoughts on the above.

joshdover commented 2 years ago

Is there any possible scenario that we can think of where indices in these patterns (e.g. .logs-* and .metrics-*) will be created but they won't need to be accessed by Fleet or Kibana? IIRC these are not system indices, so potentially users can create their own custom indices matching the pattern.

None that I know of. And correct, they are not system indices at this time. @ruflin are you aware of any other teams using this pattern? I only know of the Endpoint case.

It seems we kind of accidentally started to support these .logs-*-* indices. Should we make them official supported by also loading the logs template in addition for .logs. Same for metrics etc.?

From my understanding, the base templates are there to cover the case where users just want to start sending data without using Agent. If we don't want customers using these, I don't think we should provide a way to make it easy for them to do so.

ywangd commented 2 years ago

I am generally OK with the proposal if the index expressions all point to hidden indices and do not clash with data stream backing indices (prefixed with .ds-). But in any case, we should make effort to document the namespaces that kibana-system and fleet-server claim. Not sure what would be the best place to document it (ES or Kibana or Fleet or Solution docs?) Do we have something already that can be expanded upon?

The main difference that I see with logs-, metrics-, and traces-* is that these are well-known and documented as "reserved" for Observability use, so it's more unlikely that there may be confusion about their access.

@bytebilly Could you please point me to the relevant docs?

It seems we kind of accidentally started to support these .logs-- indices.

@ruflin Where did we "accidentally started" the support? I don't see it supported in ES yet. Do you mean Fleet and Packages?

Additionally, we should probably update all of these non-dot-prefixed index privileges to be logs-- and metrics-- instead of logs- and metrics-. This could be considered as a separate issue

I think this is a good idea especially since the builtin index templates claim to match the pattern like logs-*-*. There is no index template for traces-* or traces-*-*. I am not sure whether this should follow the same path? Since these index patterns are new in 7.16 for kibana_system, we still have time to make the update for it. But fleet-server have had privileges over the broader index patterns like logs-* since 7.13. If we narrow down the index patterns, can we be sure there won't be any BWC issues?

ruflin commented 2 years ago
bytebilly commented 2 years ago

The main difference that I see with logs-, metrics-, and traces-* is that these are well-known and documented as "reserved" for Observability use, so it's more unlikely that there may be confusion about their access.

@bytebilly Could you please point me to the relevant docs?

@ywangd there is a reference here, but it's not for traces-*: https://www.elastic.co/guide/en/fleet/current/data-streams.html#data-streams-index-templates

It seems we kind of accidentally started to support these .logs-- indices. Should we make them official supported by also loading the logs template in addition for .logs. Same for metrics etc.?

I'm not sure I get all the implications. Given that there may be some conflict, I'm wondering if it's not better to add specific indices rather than having a generic pattern. From a process perspective, adding a new index is not so complex and I guess it's not a very frequent operation. What do you think?