[Fleet] Include space ID in `namespace` piece of integration data stream names

[kpollich]

In order to support data segregation as part of Fleet's space awareness project, Fleet needs to begin including the space ID to which a given policy belongs in documents generated by integrations on that policy.

To accomplish this, Fleet should allow users to include the space ID as part of the namespace value in their integrations' data streams. Generally, the namespace part of a data stream's name is intended as a "user configurable arbitrary grouping" (see docs), so in order to preserve this intent, the space ID should be included as a prefix, and a use should still be able to provide a custom namespace.

For example, if a user configures a policy in space123 with a custom namespace of my_namespace, the resulting data streams for their integrations should follow the form {type}-{dataset}-space123.

Importantly, this behavior can break existing usages of the namespace value. e.g. if a user has ILM policies or processors that rely on a certain namespace value we would break these components by changing the namespace once a policy moves to a new space. Hence, this functionality should be opt-in.

Implementation

• [ ] Add a checkbox to opt-in to space-aware namespaces in Fleet settings
• [ ] Include the space ID as a prefix for the namespace value generated in compiled agent policies
• [ ] Ensure the generated namespace prefix cannot contain invalid characters (e.g. -)
• [ ] Add documentation about this functionality

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

@kpollich @nimarezainia Trying to think that one to be future proof, if we move to agent policies belonging to multiple space, how this will work? we will only restrict the namespace in the one of the namespace.

For example with the spaces: security_global, tenant1, tenant2 how this will work? In my opinion if I create a policy in tenant1 or tenant2 I expect namespace to be restricted to tenant1* tenant2* but for the security_global space?

[kpollich]

I think we'd have to allow for custom prefixes if we move to a multi-space policy model and rely on the user setting up their own data segregation model.

[nimarezainia]

yes I agree with @kpollich , as an extension to what you suggested @nchaulet , let's set the namespace to the space the Policy was created in but also allow for namespace to be modified.

[nchaulet]

I think we'd have to allow for custom prefixes if we move to a multi-space policy model and rely on the user setting up their own data segregation model.

@kpollich @nimarezainia Do you have any idea where that data segregation model should live? We could introduce a per space settings saved object, this will mean an user with the settings:All permissions will have to switch to that space go to the settings tabs and here in a new section list the prefix it want to allow (the default behaviour will be no restrictions), it something that will work here?

[nimarezainia]

I think we'd have to allow for custom prefixes if we move to a multi-space policy model and rely on the user setting up their own data segregation model.

@kpollich @nimarezainia Do you have any idea where that data segregation model should live? We could introduce a per space settings saved object, this will mean an user with the settings:All permissions will have to switch to that space go to the settings tabs and here in a new section list the prefix it want to allow (the default behavior will be no restrictions), it something that will work here?

do you mean where in the UI? reference is to the namespace that is defined in Agent Policy correct? so only users that have access to modify the agent policy in that space can modify the settings and modify the namespace.

[nchaulet]

do you mean where in the UI? reference is to the namespace that is defined in Agent Policy correct? so only users that have access to modify the agent policy in that space can modify the settings and modify the namespace.

Yes where in the UI, are we going to limit the authorized namespace for datastreams. If we want to limit agent policy in space_a to write to logs-space_a*, metrics-space_a*, ... where the config for that limitation should live/and who can modify it.

It is more clear?

[nchaulet]

What I propose to do here is to introduce a new API in fleet (and the equivalent in kibana.config)

Acessible to someone with the settings:all permissions

POST kbn:/s/{spaceId}/api/fleet/space_settings
{
 "allowed_prefix_namespaces": ["test1", "test2"]
}

Default will be and will not restrict namespaces prefix

{
 "allowed_prefix_namespaces":  null
}

@kpollich does it make sense to you?