search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.82k stars 8.21k forks source link

[Dataset Quality] APM is not being marked as an integration #197445

Open yngrdyn opened 2 days ago

yngrdyn commented 2 days ago

Before https://github.com/elastic/kibana/issues/190330 this issue was pure visual problem, but now we have mitigation flows based on whether the data stream belongs to an integration or not.

Image

The principal problem now is that we are able to offer an automagic solution, in the fix it flows, for dataset belonging to integrations while we offer "manual" support for the rest (see example bellow)

https://github.com/user-attachments/assets/98b59e64-bb8f-45aa-8ea7-6fdfcd8c9399

The problem is generated by the introduction of x-pack-apm plugin which basically installs index templates, component templates, and ingest pipelines for Elastic APM instead of relying in fleet packages.

We rely on dataStream metadata to infer the integration, e.g.

{
  "package": {
    "name": "endpoint"
  },
  ...
}

but with the introduction of the aforementioned plugin this metadata is not present for APM dataStreams.

Being APM part of elastic we should offer the best possible experience for users.

elasticmachine commented 2 days ago

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

achyutjhunjhunwala commented 2 days ago

Picking up from the discussions which happened on Slack, @felixbarny @flash1293

Is the below understanding correct -

Definition of Integration for Dataset Quality purposes -

Instead of - checking metadata available on the data stream API

We will - Call the index template API extracting the index template name from the data stream and see if its present or not. If its present, we will check if any of the composes_of component templates with @custom exist or not. If that happens, we will consider that DataStream to be part of an Integration.

This will solve the problem with APM Integration. I just hope we are not missing anything here as we will do change for all integration check in Dataset Quality and will not this specific for APM