[Fleet] Support installing transforms without granting `kibana_system` index privileges

[joshdover]

In https://github.com/elastic/kibana/issues/111755 we started using the internal kibana_system user to install integration package assets into Elasticsearch in order to support automatic upgrades of packages without end user intervention. We then leveraged this capability to remove the requirement for end users to have the superuser role to install packages in https://github.com/elastic/kibana/issues/108252 in 8.1.

One downside of this approach is that it requires that kibana_system have the required privileges to install any package assets. This poses a problem for some asset types that require more expansive privileges in Elasticsearch to create:

• transforms requires these privileges:
  • transform_admin built-in role or manage_transform cluster privileges,
  • read and view_index_metadata index privileges on source indices, and
  • create_index, index, manage, and read index privileges on destination indices

Below the current behavior is detailed as well as two proposals for addressing the shortcomings of the current behavior. Progress on either proposal is blocked by completing official support for transforms (making the current behavior available to more packages) in https://github.com/elastic/kibana/issues/134321

Current behavior (for Endpoint only, all packages soon)

The legacy/"illegal" feature (not in the spec, only works for Endpoint package) for installing transforms will work the same from a auth perspective as the official one to be implemented by https://github.com/elastic/kibana/issues/134321.

• No packages are installed with the default Kibana configuration
• Users can add packages to be installed by on Kibana startup via the xpack.fleet.packages field in kibana.yml
• All packages today are installed using the kibana_system user's credentials, regardless of whether it's installed via user interaction in the UI/API or if it's installed on boot
• https://github.com/elastic/kibana/issues/134321 will not change this

sequenceDiagram
  participant U as UI
  participant K as Kibana
  participant ES as Elasticsearch
  alt xpack.fleet.packages is set
    K->>K: Boot
  else is user initiated
    Note over U,K: Authorization: Basic <user credentials>
    U->>K: Install package <x>
  end
  Note over K,ES: Authorization: Bearer <elastic/kibana_system token>
  opt If upgrading existing package
    K->>ES: DELETE /_transform/<id>
  end
  K->>ES: PUT /_transform/<id>
  K->>ES: POST /_transform/<id>/_start

Proposal A: install using user credentials

• The current behavior has the limitation that kibana_system must have index privileges to read/write to the src/dest indices of the transform, respectively.
• We could allow other transforms to be installable without granting kibana_system additional index privileges by leveraging the user's access
• This would require that the user also has access to the manage_transform privilege
• This would not support auto installing or upgrade packages containing transforms on Kibana startup or boot. Packages could still only be installed or upgraded automatically on boot if kibana_system has explicit index privileges

sequenceDiagram
  participant U as UI
  participant K as Kibana
  participant ES as Elasticsearch
  alt xpack.fleet.packages is set
    K->>K: Boot
    Note over K,ES: Authorization: Bearer <elastic/kibana_system token>
    opt If upgrading existing package
      K->>ES: DELETE /_transform/<id>
    end
    K->>ES: PUT /_transform/<id>
    K->>ES: POST /_transform/<id>/_start
  else is user initiated
    Note over U,ES: Authorization: Basic <user credentials>
    U->>K: Install package <x>
    opt If upgrading existing package
      K->>ES: DELETE /_transform/<id>
    end
    K->>ES: PUT /_transform/<id>
    K->>ES: POST /_transform/<id>/_start
  end

Proposal B: install using secondary credentials w/ user API key [⭐ Recommendation]

• This option allows Kibana to automatically install or ugprade packages without requiring explicit index privileges
• Kibana would either create an API key on behalf of the user at install time or require that users specify an API key in the kibana.yml file to be used for initial package installation
• Kibana could then use this API key to install or upgrade packages outside of user interactions
• This option would require that users install packages also have the manage_own_api_key privilege and index privileges for the transform src/dest indices, but they would not need manage_transform

Installing a package from the UI

• During installation of the package initiated from the UI, an API key is created on the user's behalf
• This API key is persisted by Kibana in an encrypted Saved Object (SO) for later use for upgrades
• This API key is then used on the PUT /_transform/<id> API call with the es-secondary-authorization header
• If the user does not have manage_own_api_key privilege, they would not be able to install packages with transforms

sequenceDiagram
  participant U as UI
  participant K as Kibana
  participant ES as Elasticsearch
  Note over U,ES: Authorization: Basic <user credentials>
  U->>K: Install package <x>
  K->>ES: POST /_security/api_key
  ES-->>K: API key
  Note over K,ES: Authorization: Bearer <elastic/kibana_system token>
  K->>ES: Save API key in encrypted SO
  Note over K,ES: es-secondary-authorization: ApiKey <key>
  K->>ES: PUT /_transform/<id>
  K->>ES: POST /_transform/<id>/_start

Installing a package on Kibana startup

• Packages that contain transforms that kibana_system doesn't have index privileges for still need an API key
• In order to enable the use case to install these packages on Kibana boot, the admin would need to manually create and specify the API key in kibana.yml like this:

  xpack.fleet.packages:
  - name: security-host-risk-score
    version: 1.0.0
    api_key: <key> # could also be a reference to a key in the secret storage

• Kibana would persist this key in encrypted SO storage the same way a user-initiated install would
• If a package is specified in kibana.yml that contains a transform but no API key has been specified, Kibana will fail to install the package with an error

sequenceDiagram
  participant K as Kibana
  participant ES as Elasticsearch
  K->>K: Retrieve API key from kibana.yml
  Note over K,ES: Authorization: Bearer <elastic/kibana_system token>
  K->>ES: Save API key in encrypted SO
  Note over K,ES: es-secondary-authorization: ApiKey <key>
  K->>ES: PUT /_transform/<id>
  K->>ES: POST /_transform/<id>/_start

Auto-upgrading a package

• To upgrade a package containing a transform, Kibana first fetches the API key created during installation from the encrypted SO
• This key is then reused on the PUT /_transform/<id> API call with the es-secondary-authorization header

sequenceDiagram
  participant K as Kibana
  participant ES as Elasticsearch
  Note over K,ES: Authorization: Bearer <elastic/kibana_system token>
  K->>ES: Fetch API key from encrypted SO
  ES-->>K: API key
  K->>ES: DELETE /_transform/<id>
  Note over K,ES: es-secondary-authorization: ApiKey <key>
  K->>ES: PUT /_transform/<id>
  K->>ES: POST /_transform/<id>/_start

API key errors during auto-upgrades

There are a couple scenarios where the API key that was stored during installation can't be used to upgrade the package without user interaction:

• The API key has been revoked/invalidated
• The new package version's transforms require new index privileges (new src or dest indices were added)

In either case, the package upgrade should fail gracefully. A user must manually upgrade the package via the UI to generate a new API key for the user's credentials.

Use cases for automatic package upgrades

Today we only support automatically upgrading packgages for a few key 1st party packages: apm, endpoint, synthetics, fleet_server, elastic_agent.

The reason these packages need to be automatically upgraded is to ensure that the UIs that depend on the backing indices of these pacakges can rely on specific mappings being in place. For example, this helps avoid bugs where the UI may try to filter on a specific field which was only added in a recent release of the package. This would fail if the package wasn't upgraded because the underlying data streams or transform destination indices do not have the correct mappings.

If we expect additional packages containing transforms to require automatic upgrades in a similar manner, we should consider moving forward with proposal B which would allow users to install these packages without the user needing explicit manage_transform privileges and also without requring kibana_system to be granted index privileges.

We do not currently install any packages by default due to user feedback that this was confusing when users weren't using these packages. The only expception today is the APM package on Elastic Cloud, which is required to boot the APM Server successfully. I don't recommend that we explore installing any more packages by default and instead should rely on user interaction with the Solution to install new packages. Proposal B can then be leveraged to automatically upgade these packages if needed to avoid bugs like the ones mentioned above.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[szeitlin]

It seems like Proposal B is your preferred method, and we expect to want to be able to install and automatically upgrade without the user needing manage_transform access, so I think that makes sense based on what I know right now.

Some questions that occurred to me while reading this:

1. What are the scenarios when we might expect the user does not have manage_own_api_key privilege to be a problem? It seems like we're assuming that's a non-issue, based on who we're assuming would be allowed to install packages. Is that correct?

2. How do we link the API key to the index privileges? Does that happen in the package? What if we wanted to run the same transform on a different index/more indices? Does that all get handled by the user's existing index privileges, or would they need to generate new API key(s)?

[joshdover]

1. What are the scenarios when we might expect the user does not have manage_own_api_key privilege to be a problem? It seems like we're assuming that's a non-issue, based on who we're assuming would be allowed to install packages. Is that correct?

Yes that's about right. Today, the user needs "All" access to the Integrations app in order to install packages. This would add a new requirement for users to also need manage_own_api_key to install packages containing transforms. This design makes the assumption that users with this level of access probably have this privilege. If they don't the UI should notify the user which privilege they're missing.

2. How do we link the API key to the index privileges? Does that happen in the package? What if we wanted to run the same transform on a different index/more indices? Does that all get handled by the user's existing index privileges, or would they need to generate new API key(s)?

Great question. In this design, you are right that the system wouldn't be able to automatically upgrade a package containing a transform if the indices required for the transform(s) requires read or write access to new indices (though it would work if the new version of the package requires fewer indices).

I've updated the proposal to call this out explicitly.

[joshdover]

This needs to be evaluated within the context of other asset types, like ILM. We should strive for a common solution so we don't have to build separate mechanisms and UI flows for different asset types.

[droberts195]

The key question is, is it acceptable to mandate that the user who installs a Fleet integration package has the required permissions on all the indices it references? I think it would be safest to assume not, as it turned out to be unacceptable to require superuser to install packages. It seems like there is a requirement that one class of user is responsible for installing a package while another class of user later uses the things it contains.

We came up with this as an outline plan for how things could work for transforms:

• Transform runs with narrow privileges
  • Create API key as intersection of required privileges and current user privileges so that we don’t end up storing superuser privileges if superuser happens to install a package
  • Supply API key as secondary credentials on transform creation, i.e. kibana_system creates it but tells it to store the API key for future use
  • Store API key in encrypted saved object for later use in automatic upgrades
• Transform can be installed by user who only has Fleet privileges
  • Don’t check credentials on unattended transform creation
  • Might mean transform doesn’t work for a while, as it could be running with an API key that doesn’t allow all required access to source and destination
  • Ability to automatically upgrade
  • Reuses stored API key retrieved from the encrypted saved object
• In-app banner to authorize the transform if necessary
  • Used in case of initial creator not having sufficient privileges
  • Somebody who has the required privileges needs to give the go-ahead for the transform to run if the initial installer didn’t have those privileges by clicking a link in the banner
  • Same banner used to reauthorize if API key gets invalidated or accidentally deleted
  • Would be displayed by most appropriate app in Kibana, e.g. Security
• Unattended transforms will also surface failures due to lack of permissions in health API (eventually)
• Allow transform UI to “create API key and run as this”
  • API key is stored in encrypted saved object replacing any that was created on package install
  • Credential change can be achieved by a no-op update with the new API key supplied as secondary credentials

The same approach could be used for enrich processors and ILM policies if they supported secondary credentials (and that is not a particularly difficult feature to add).

@joshdover shall we schedule a meeting for early next week to talk it through with @dakrone?

[szeitlin]

Seems like this has kind of stalled. Were there any further discussions/decisions made since October?

[joshdover]

We have a meeting today to discuss next steps.

[droberts195]

In the meeting we came up with a similar but slightly simpler plan:

• We will still use an API key supplied via secondary credentials to authorize transforms
• Transforms created via the package installer will check if the credentials supplied at install time are adequate, and if not will set a flag to indicate that their credentials are inadequate to run
  • We need to decide if the trigger for this functionality is that the transform is an unattended transform, or if there'll be a separate flag on the create transform endpoint
  • The flag indicating inadequate credentials could be surfaced through transform health information in the transform stats
• Transforms with inadequate credentials won't attempt to run, but will wait for an update to supply usable credentials
• A page in the Fleet UI will list the entities that are awaiting a sufficiently empowered user to authorize them
  • The page could be titled "Deferred installation", although the components listed on it will already be installed, just not yet operational
  • A toast/banner will direct users to that page if any entities require authorization

This revised plan avoids the need to store API keys in encrypted saved objects. Potentially that functionality can be added at a later time.

There's nothing to stop some packages being permitted to run as kibana_system user if we want to keep adding permissions to that user for some use cases, but allowing a different user to authorize transforms after install adds the flexibility to have packages containing transforms that kibana_system cannot run.

It sounds like enrich policies might not have the same issues as transforms, as they access the enrich index as an internal user when they run in an ingest pipeline.

ILM policies are still problematic as they need functionality to report what permissions they will need to operate many days in the future, and adding that functionality is difficult.

So it sounds like we should just focus on transforms initially.

[jamiehynds]

@droberts195 our integration development partner, Crest, is currently working on an integration with Rapid7 Threat Command. As you can see in the PR review we realised a user has to follow several steps to manually load the transforms with the integration. Passing along in case it helps to have a real world example on the need for transforms within an integration.

[droberts195]

On talking through the detailed sequence of API calls with @qn895 I realised there is a major flaw in the idea of using a minimal API key. To create an API key with minimal permissions for the transform requires that a role exists with those minimal permissions. You cannot just create an API key with a list of permissions. In general there won't be a pre-existing role that has the required permissions for some arbitrary transform, so for the Fleet installer to set up an API key for a transform to use would require that the Fleet installer can create a suitable role. The kibana_system user does not have permission to create roles, so using API keys to authorize transforms would just shift the permissions problem from transforms to roles.

We can still use an approach that's similar to the previous plan, but using the credentials of the user directly instead of using an API key. Alternatively, we could still use an API key, but supplying an empty roles list, which is a point-in-time snapshot of the privileges of the current user.

@qn895 also highlighted another problem, which is that when 2 transforms in a package are chained it will be hard to create the second one, because its source index is the destination index of the first one. We'll have to create the second with defer_validation=true to avoid problems there. However, defer_validation=true also skips the check on whether the user creating the transform has the necessary permissions on the source and destination indices. We'll need to change that so that permissions are still checked when defer_validation=true, but lack of permissions is just recorded in the config rather than throwing an error and preventing the transform being created at all.

In some cases this approach is going to mean that aliases cannot be adjusted on the transform destination indices by the Fleet installer. We'll have to add functionality to the transforms themselves to be able to adjust aliases on the destination index (if the required permissions are available).

cc @przemekwitek

[droberts195]

I have opened https://github.com/elastic/elasticsearch/issues/93259 to track the work to enhance transforms to know whether they have sufficient credentials or not. While doing this two things occurred to me:

1. We'll be using the _has_privileges API to work out whether a transform has sufficient credentials. This API does not work for cross cluster search. For any transforms in Fleet packages that do cross cluster search we'll have to document that the user who installs the Fleet package needs to have all the necessary credentials that the transform needs to run, and that they are responsible for ensuring this. The documentation for that particular Fleet package should state this pre-requisite. However, I am assuming that many (most?) transforms in Fleet packages will not be configured to do cross cluster search, so the functionality to check, record and update inadequate credentials will still be useful most of the time.
2. We should probably keep the option to install transforms as kibana_system. For some internal transforms that are very hidden away it may be better that an internal user runs them, to avoid confusing users about why they need permissions for deeply internal parts of the system. However, this should be an option that we use in selected cases, and not the default behaviour. All Fleet packages that provide higher level functionality and those created by 3rd parties will need transforms to run as a user with appropriate privileges.