Misleading overview for granular integrations in Integrations UI

[akshay-saraswat]

Kibana version: 7.16

Elasticsearch version: 7.16

Server OS version: N/A

Browser version: N/A

Browser OS version: N/A

Original install method (e.g. download page, yum, from source, etc.): N/A

Describe the bug: Kibana is not looking at the data stream level manifest. There are some granular integrations that are actually data streams of a package. Those integrations appear as separate tiles in the integrations UI. Overview/Description of such integrations is sometimes misleading. For example, AWS Billing.

Steps to reproduce:

1. Click on "Add integrations" button
2. Click on "AWS" category in the integrations UI
3. Click on "AWS Billing" integration tile

Expected behavior:

• This integration has not been GA but "Beta" flag is missing
• Features mentioned for the integration are "metrics, logs" when this integration supports only metrics

Screenshots (if relevant):

Errors in browser console (if relevant): N/A

Provide logs and/or server output (if relevant): N/A

Any additional context: N/A

[kaiyan-sheng]

@mtojek Is it a problem to fix in the package registry? I don't see release displayed at the policy template level from /search?package=aws endpoint:

[
  {
    "name": "aws",
    "title": "AWS",
    "version": "1.4.6",
    "release": "ga",
    "description": "Collect logs and metrics from Amazon Web Services with Elastic Agent.",
    "type": "integration",
    "download": "/epr/aws/aws-1.4.6.zip",
    "path": "/package/aws/1.4.6",
    "icons": [
      {
        "src": "/img/logo_aws.svg",
        "path": "/package/aws/1.4.6/img/logo_aws.svg",
        "title": "logo aws",
        "size": "32x32",
        "type": "image/svg+xml"
      }
    ],
    "policy_templates": [
      {
        "name": "billing",
        "title": "AWS Billing Metrics",
        "description": "Collect billing metrics from Amazon Web Services with Elastic Agent",
        "icons": [
          {
            "src": "/img/logo_billing.svg",
            "path": "/package/aws/1.4.6/img/logo_billing.svg",
            "title": "AWS Billing logo",
            "size": "32x32",
            "type": "image/svg+xml"
          }
        ]
      },
...
]

[mtojek]

It's a design issue as we haven't assumed that policy templates should have "release" tags. They are virtual groupings of data streams, which means that theoretically, you can group beta and GA data streams.

[mostlyjason]

@mtojek @akshay-saraswat isn't the same true for packages as well? A package may be GA, but still contain beta or experimental data streams. An example of this is the Nginx package, in which the third party REST API input and streams are experimental. A single experimental data stream does not mean the whole package is experimental. It seems like we need a way to communicate the release status at a lower level of granularity than the package. The policy template seems like a good place to start since we already have a design for communicating the release status of integrations, and users will expect to see it there. Its just a matter of populating the information.

Could we treat data-stream level granularity as a separate issue? Its a little trickier since it overlaps with https://github.com/elastic/kibana/issues/102816.

[mtojek]

The policy template seems like a good place to start since we already have a design for communicating the release status of integrations, and users will expect to see it there. Its just a matter of populating the information.

The policy template aggregates different data streams, so correct, for labelling purposes Kibana can pull this information from data streams. We don't need to introduce another property for policy templates as it would be redundant.

[mostlyjason]

@mtojek What is the aggregation rule you're proposing? Its possible for an integration to be GA but contain beta or experimental data streams. An aggregation rule like min(data stream release levels) would not account for this use case. In the case of Nginx, the package authors decided to release this integration as GA, and label the input (and data streams) as experimental separately.