[security] Review APM Server roles and their documentation

[simitt]

Review all features the APM Server supports that are related to either connecting to Kibana or Elasticsearch, and review which privileges are required for these features. E.g. look into sourcemap fetching, agent config management, monitoring, API Key authentication, .. Ensure APM built-in roles have privileges for all features they should support.

Use current docs related to built-in roles and feature roles for the review and ensure they are clear and up to date.

[bmorelli25]

I almost wonder if it's time to stop using some/all of the shared security documentation from Beats, and instead create our own security documentation for APM Server. The current setup works, but it doesn't offer us a lot of flexibility.

[simitt]

Using parts of the shared docs would be optimal; linux seccomp is solely provided by libbeat, copying the docs from there ensures we receive doc bugfixes and updates. Whereas other docs as basic-auth even contain a configuration setup.kibana that APM Server doesn't support any more since 7.0.

[simitt]

Ideally after having a summarized overview of the roles and privileges we reflect the minimum required privileges per version in the Integration testing. This would enforce us to think about newly added features requiring new privileges for every new version.

[simitt]

Open TODOs:

• Figure out the difference between manage_pipeline and manage_ingest_pipeline?
• Do we need the cluster: monitor privilege; where does this come from? In the following proposal I removed the privilege as I didn't need it anywhere.
• ES docs for built-in APM user are outdated; they are pointing to the 7.0 APM Server docs and referencing a deprecated way for configuring monitoring

After testing the minimal required privileges and comparing with current documentation I propose following changes for the 7.7 and 7.8 documentation:

APM Server Setup

Document best practice to create dedicated user for setup and ingestion phase. APM Server setup is run either when calling `apm-server setup` or starting `./apm-server` with default config, where setup is enabled. During the setup phase the APM Server needs privileges to: * set up index templates: `setup.template.enabled: true` * set up pipelines: `register.ingest.pipeline.enabled: true` * set up ILM with aliases, templates and policies: `setup.template.enabled: true` Proposed changes to documentation: * remove documenting `kibana_user` as nothing is loaded to Kibana (index pattern is bundled with Kibana, no official dashboards available) * only document `ingest_admin` once * document for complete setup: * cluster privileges: `manage_ilm` * index privileges: `apm-*`: `manage` * roles: `ingest_admin` * document every configurable setup itself: - setup index templates: * cluster: `manage_index_template` - setup ILM: * cluster: `manage_index_template`,`manage_ilm` * index: `apm-*`: `manage` - setup ingest pipeline: * cluster: `manage_pipeline` (OR `manage_ingest_pipeline`)

APM Server Ingest

During the ingest phase the APM Server needs privileges to * send data to Elasticsearch * upload sourcemaps to Elasticsearch * read sourcemaps from Elasticsearch * read API Keys from Elasticsearch * self-instrumentation Since different elasticsearch users can be configured for the listed features, it makes sense to document them separately. Proposed changes to documentation: * document for full functionality: - index `apm-*`: `create_doc`,`create_index` - index `apm-*sourcemap`: `read` * ingest agent data: - index `apm-*`: `create_doc`,`create_index` (Note: privilege `create_index` is also needed when using ILM, for ingesting the onboarding doc) - index `apm-*sourcemap`: `read` (when sourcemap enabled `apm-server.rum.enabled: true` AND `apm-server.rum.source_mapping.enabled: true`) * uploading sourcemaps: - index `apm-*`: `create_doc`,`create_index` * no extra privileges for fetching API Keys and self-instrumentation

APM Server Monitor (internal)

Creating monitoring data: * Either use `apm_system` user of built-in role `apm_system` * Or create custom role with privileges: - index `.monitoring-beats-*`: `create_index`, `create_doc` Viewing monitoring data: * built-in role `monitoring_user` AND * custom role enabling Kibana space for dashboards (including stack monitoring) see [Kibana privileges](https://www.elastic.co/guide/en/kibana/7.6/kibana-privileges.html#kibana-privileges) * Change the current statement: > Assign the monitoring role, along with the following built-in roles, to users who need to monitor APM Server: to make clear that `apm_system` role is for writing monitoring data, while `monitoring_user` and `kibana_user` are for viewing monitoring data * `kibana_user` is deprecated (replaced by `kibana_admin`) and adds more privileges than required. Better suggest to add Kibana space privileges for reading monitoring data * For cloud setup, change the following note, as there is a UI button available to enable self-monitoring. When clicked, everything get setup up and works out of the box, no need for manual user creation: > Important note for Elastic Cloud users > >Built-in users are not available when running our hosted Elasticsearch Service on Elastic Cloud. To send monitoring data securely, create a monitoring user and grant it the roles described in the following sections.

APM Server Monitor (metricbeat)

Creating monitoring data: Tested with `remote_monitoring_user` configured for `modules.d/beat-xpack.yml` and `metricbeat.yml` (`output.elasticsearch`). Requirement for this to work with the built-in user is to disable system metrics and all the setup. Suggestion: Ensure monitoring is properly documented in `metricbeat` and only link to it. Viewing monitoring data: This section should be the same for both - internal metrics collection and using metricbeat (see suggested changes above)

APM Server agent configuration management

* minimal required privilege (tested): read access to Kibana APM space. There is no built-in role available for this. Describe how to create user with role that only grants read access to [Kibana APM space](https://www.elastic.co/guide/en/kibana/7.6/kibana-privileges.html#kibana-privileges) * check with @sqren if `read` permissions are expected to be enough (instead of read+write permissions). * [kibana username + password](https://www.elastic.co/guide/en/apm/server/master/setup-kibana-endpoint.html#_apm_server_kibana_username): it is not true that the elasticsearch username+password is used

APM Kibana agent configuration management

* Rephrase docs to make it clear these data do not come from APM Server * do not require `admin` role but minimal required permissions: - built-in role `apm_user` - AND granting full Kibana space privileges for APM

APM UI Kibana

Currently the built-in role `kibana_user` and `apm_user` are documented. This is a nice getting-started experience and ensures future features are supported with given roles. For a minimal requirement we could also document following setup: * built-in role `apm_user` AND * granting full Kibana space privileges for APM AND * built in role `machine_learning_admin`

Create API Keys via APM Server

Keep documentation as is; feature is not GA yet.

@bmorelli25 please let me know what you think. Once we agree on the documentation, I can work on adding users and roles accordingly to the apm-integration testing to be in sync with the docs.

[bmorelli25]

This is a fantastic!

A few notes on your todos:

Figure out the difference between manage_pipeline and manage_ingest_pipeline?

They are duplicates. manage_ingest_pipelines is being removed in favor of manage_pipeline. See: https://github.com/elastic/elasticsearch/issues/55640

Do we need the cluster: monitor privilege; where does this come from? In the following proposal I removed the privilege as I didn't need it anywhere.

I chatted with DeDe about this. She says cluster: monitor is required during Beats setup to retrieve cluster details like version info. She assumes we'd need it as well to use the APM Server setup command. I'll give it a test as well.

ES docs for built-in APM user are outdated; they are pointing to the 7.0 APM Server docs and referencing a deprecated way for configuring monitoring

Good find. ~I'll open a PR to fix this today.~ Fixed in https://github.com/elastic/elasticsearch/pull/56537, https://github.com/elastic/elasticsearch/pull/56538, https://github.com/elastic/elasticsearch/pull/56539 .

[bmorelli25]

APM Server agent configuration management minimal required privilege (tested): read access to Kibana APM space. There is no built-in role available for this. Describe how to create user with role that only grants read access to Kibana APM space check with @sqren if read permissions are expected to be enough (instead of read+write permissions).

@sqren, In addition, a question about the Agent config API. After some testing it seems read access to the APM feature privilege is good enough to GET Agent config values, and creating or delete configs requires all access to the APM feature privilege. Does that sound right?

[bmorelli25]

~Still figuring out the Annotation API privileges in https://github.com/elastic/kibana/pull/65963#discussion_r422759066. I think it's something like this:~ Confirmed that annotation API privileges are: Feature: APM --> all Index: observability-annotations. Privileges: create_index, create_doc, manage, and read.

[sorenlouv]

@simitt Thanks for starting this initiative. Much needed as the permission requirements are already very complex to understand - for users and devs alike.

@sqren, In addition, a question about the Agent config API. After some testing it seems read access to the APM feature privilege is good enough to GET Agent config values, and creating or delete configs requires all access to the APM feature privilege. Does that sound right?

That sounds right 👍

check with @sqren if read permissions are expected to be enough (instead of read+write permissions).

Read permissions to apm-* is sufficient to view data in apm ui. Additionally the user must have read privilege to the APM app via Kibana Security.

[simitt]

Ensure APM built-in roles have privileges for all features they should support.

The existing built-in roles for APM are up-to-date.

Also addressing the question if we should add built-in roles for the APM Server setup and data ingestion phases: Whenever the APM Server requires additional privileges for new features, the privileges could be added to the built in role and existing users automatically would get the updated privileges when upgrading the APM Server. We would be providing such a built-in role for configurable user data indices though. Whenever a user changes the apm-* indices to have another prefix or wants to split the indices more fine granular and make use of document level security, the built-in roles could not be used as they would be fixed for apm-* indices.

We decided to not introduce built-in roles for the APM Server at the moment. When providing a fine-granular up-to-date documentation, creating a role and the user vs. creating the user with a built-in role is almost the same experience. The privileges required by the server are not expected to change often (and haven't in the past). To be revisited in the future if flexibility of index creation changes.

[bmorelli25]

@simitt Question about Agent config management. You list the following users:

• APM Server agent configuration management: read access to APM feature.
• Kibana agent configuration management: all access to APM feature and apm_user.

Can you clarify what the difference in use-case is for these users? It sounds like the Kibana user is used to view and update Agent configs from within the APM app. What is the APM Server user used for? Reading central config values?

[sorenlouv]

@bmorelli25 As you say there should be at least two users. In this example there'll be three users:

• APM Server (system user): read access to APM feature
• Joe (real user): all access to APM feature and apm_user role
• Carole (real user, optional): read access to APM feature and apm_user role

APM Server system user APM agents fetch their configs by connecting to Kibana through the publicly available APM Server. So in other words: the agent requests are proxied and authenticated through the APM Server. Therefore the APM Server needs to have read access when calling the agent config search endpoint in Kibana: GET /api/apm/settings/agent-configuration/search. Important: the APM Server does NOT need access to APM indices to make this request.

Joe Can login to Kibana, see APM data (access to apm indicies via apm_user role) and can view/create/update/delete agent configs (via all access)

Carole This user is not strictly necessary but is worth mentioning. This user only has read on APM feature and therefore cannot create/delete/update agent configs.

[bmorelli25]

👋 Quick update: Docs PRs have been opened and are ready for review!

• Beats: https://github.com/elastic/beats/pull/18594
  • Contains changes to the security documentation structure.
  • Removes duplicated content and makes it easier for the APM Server to include customized RBAC content.
• APM Server: https://github.com/elastic/apm-server/pull/3799
  • Updates the security documentation for APM Server
• Kibana: https://github.com/elastic/kibana/pull/67401
  • Adds security documentation for the APM app

Relevant follow-up tasks:

• Monitoring user: We have an open issue to document APM Server monitoring (https://github.com/elastic/kibana/issues/37644). As a part of this issue, I'll look into moving the view monitoring role out of the APM Server reference. I'll also look at removing the Metricbeat monitoring privileges.
• ~Hooking things up:~
  • ~If the Beats PR merges after the APM Server PR, a new PR will need to be opened to ensure the APM Server reference changes are persisted in the Beats repo~ Done.
  • ~If the Kibana PR merges after the APM Server PR, a new PR will need to be opened to link to the new writer content in the Kibana reference.~ Done.

[barkbay]

@sqren, In addition, a question about the Agent config API. After some testing it seems read access to the APM feature privilege is good enough to GET Agent config values [...]

That sounds right 👍

Do you know if it is also true for 7.3 and 7.4 ? Using the built-in role kibana_user to access the agent configuration seems to work only from 7.5

Edit: I just realized that it is a little bit off topic re. the purpose of this issue, sorry about that

[sorenlouv]

Do you know if it is also true for 7.3 and 7.4 ?

@barkbay No, I'm pretty sure that is not the case. The initial security model required the user to have access to the internal index for agent configs (.apm-agent-configuration). We changed that around 7.4 (can't remember the exact version - I can try dig into it if necessary) to the security model we have today that only requires the user to have read or all access to the apm feature.

[simitt]

@bmorelli25 thanks for all the effort you put in, great outcome! Closing this issue now as the PRs are merged.