Add subobjects definition (at the data stream level)

zmoog commented 5 months ago

What does this PR do?

Add support for subobjects: false at the data stream level.

Here is an example:

# From /packages/good_v3/data_stream/subobjects/manifest.yml
title: my-data-stream
type: logs
elasticsearch:
  index_template:
    mappings:
      subobjects: true

Why is it important?

Give integration developers (per data stream) access to the subobjects option in the integration's index template mappings.

Since we added the subobjects option in stack version 8.3, users could customize how Elasticsearch handles fields that contain dots in their names from true (expanded, current default) to false (not expanded). However, integration developers could not set this up in the integrations.

Note on per filed option: the subobjects option has been available at the field level since package-spec 3.1.0. However, to make this happen at the data stream level, we needed https://github.com/elastic/elasticsearch/issues/99860 to land in Elasticsearch.

Checklist

[x] I have added test packages to test/packages that prove my change is effective.
[x] I have added an entry in spec/changelog.yml.

Related issues

felixbarny commented 5 months ago

I validated that it works as expected when this kind of configuration hits the install logic in the fleet kibana plugin and from what I can see it works fine

Does that mean that no code changes are required in Fleet to pick up the subobjects flag?

flash1293 commented 5 months ago

@felixbarny

Does that mean that no code changes are required in Fleet to pick up the subobjects flag?

I will validate once that's in, but by my understanding this is the case, yes. The index template mappings object is spread into the "mappings" of the package component template in the fleet code: https://github.com/elastic/kibana/blob/b2add71278992d838fcf5955ff9d6b3e4470274c/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts#L380

elasticmachine commented 5 months ago

:green_heart: Build Succeeded

Buildkite Build
Commit: bfbf87e9e6f21f3f7dc8d66c81287986ad081f40

History

:broken_heart: Build #597 failed 03547fc463199a176573bbaee49b46e77c234b2e
:green_heart: Build #594 succeeded 55bb155e698d3397e97e0e9edbbaf6f79e1802b2
:green_heart: Build #593 succeeded df5eabf8e91a7241ba1ef8f9c371671991faf5f9
:broken_heart: Build #592 failed 8353884278df16c27618fe0d5de42051726278cd
:green_heart: Build #589 succeeded 9647d64e646ef80f34ad2389467b6e91c0b69839
:green_heart: Build #588 succeeded 017d54cdbff30aba6e48eb250e58cd7741a1dd7c

cc @zmoog

elastic / package-spec