[Fleet] Add `<type>@custom` component template to each integration index template's `composed_of` array

[kpollich]

To better support user customization across all documents of a given type, Fleet should add a @custom component template to each integration data stream's respective index template. For example, the index template for the logs-system.auth-default data stream should include logs@custom in its composed_of array:

logs@settings
logs-system.auth@package
logs@custom <---- Global customizations to all documents of type `logs` go here
logs-system.auth@custom
ecs@mappings
.fleet_globals-1
.fleet_agent_id_verification-1

Implementation checklist

• [ ] @custom component templates are included for all types: logs, metrics, traces, synthetics, and profiling
• [ ] Documentation is added to Fleet's data streams docs regarding the new component templates

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[felixbarny]

A note on the ordering: in order for the <type>@custom template to be able to override field mappings from the package definition, it should go after @package, but before the <type>-<package>@custom template that's more specific to the package. For example:

logs@settings
logs-system.auth@package
logs@custom <---- Global customizations to all documents of type `logs` go here
logs-system.auth@custom
ecs@mappings
.fleet_globals-1
.fleet_agent_id_verification-1

[mbudge]

A note on the ordering: in order for the <type>@custom template to be able to override field mappings from the package definition, it should go after @package, but before the <type>-<package>@custom template that's more specific to the package. For example:

logs@settings
logs-system.auth@package
logs@custom <---- Global customizations to all documents of type `logs` go here
logs-system.auth@custom
ecs@mappings
.fleet_globals-1
.fleet_agent_id_verification-1

Are you sure?

Doesn't it need to go above "logs-system.auth@package" to override the package settings?

I've been wanting to add the lowercase normaliser to fields like host.name, user.name and user.target.name since we moved to Elastic-Agent. This is because users are constantly missing logs or struggling to search for hostnames where most search interfaces/KQL is case-sensitive. We want to add these lowercase normalisers globally to all log data, overriding the elastic integration mappings.

[felixbarny]

Yes, later component templates have a higher precedence.

See also https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html#multiple-component-templates

When multiple component templates are specified in the composed_of field for an index template, they are merged in the order specified, meaning that later component templates override earlier component templates.