[Change Proposal] Allow configuration of "featured integrations" on a per-category basis

[kpollich]

Goals

Allow Kibana to render a dynamic set of "featured integrations" in the Integrations UI based on the currently selected category.

Currently, these featured categories are hardcoded in Kibana, which results in potentially featuring irrelevant integrations based on specific onboarding flows. For example, if a user goes through a guided onboarding process and indicates that they're interesting in logging custom data, we'd like to be able to land them on this page with the custom category pre-selected and a curated set of featured integrations.

The data around what integrations are featured in a given category should be present in the API endpoint here: https://epr.elastic.co/categories.

Proposed Implementation

Support for a top-level categories.yml file that allows for defining featured integrations for a given category, e.g.

categories:
  - custom:
      featured_integrations:
        - log
        - httpjson
        - aws_logs

When the data for the /categories API is generated, we should check the categories.yml file for a corresponding category after deriving the list of categories from all packages. For each category in the resulting JSON object, we should also include an array of strings containing its featured_integrations value if one exists.

Integrations UI will only render the first three integrations featured here, and if there are less than three then Integrations UI should still handle presenting the featured integrations in a reasonable design. Integrations UI will also handle cases where a featured integration does not actually exist, etc. We don't need advanced linting here limit the length of the featured list or expect only valid package names, but it could be nice to have.

[jen-huang]

Another path to consider is introducing a ranking system in the package manifest that Fleet can use to sort the integration tiles (currently they are alphabetical based on package name).

This is potentially more scalable for future needs, as we are not limited to 3 "featured" integrations but can instead customize the sorting of all integrations based on product needs.

[jsoriano]

Support for a top-level categories.yml file

Where would this file be? Would this file be a configuration file for the package registry?

Another path to consider is introducing a ranking system in the package manifest that Fleet can use to sort the integration tiles (currently they are alphabetical based on package name).

With this approach each package developer could chose their ranking, as they maintain their manifests, potentially self-promoting their own packages.

This is potentially more scalable for future needs, as we are not limited to 3 "featured" integrations but can instead customize the sorting of all integrations based on product needs.

Approach proposed by Kyle wouldn't be limited to 3 integrations, right? The list of featured_integrations could have any length.

[jen-huang]

Approach proposed by Kyle wouldn't be limited to 3 integrations, right? The list of featured_integrations could have any length.

Yes, that's true. I was thinking it was more scalable in that we would not have maintain a ranked list, instead Fleet could just determine sort order based on the ranking parameter across all packages. But this does have the drawback you mention of self-promotion.

@jsoriano Do you have any alternate suggestions to achieve the end goal of selective sorting?

[jsoriano]

Do you have any alternate suggestions to achieve the end goal of selective sorting?

Kyle's approach looks good in principle, but we have to think about where this categories.yml would live, and how it would be updated.

[mtojek]

Don't forget that package-storage repository will be archived soon (in 8.5?). I guess that you want to keep this file somewhere around published packages.

I would still consider the option proposed by @jen-huang to calculate the ranking. It might be beneficial that for a different customer, we may want to promote different integrations.

[kpollich]

I was thinking to keep the categories.yml file in https://github.com/elastic/package-registry since the categorization structure is an API concern as I see it. It could also simply be a hardcoded struct in the Go code or something like that. As long as it can be updated via PR's reasonably by any engineer I think we'd be happy here.

My main concern here is making it relatively easy for any team to update the featured integrations(via GitHub PR ideally) for a given category without deep technical knowledge of the package spec or EPR infrastructure. As long as we can meet that need, I am happy to be flexible about where the actual config file lives.

[jsoriano]

If categories.yml is part of the package registry, changing it for users will require to merge the PR, but also to deploy a new package-registry. With the storage v2 we have now all the packages state and its metadata there, it could be an option to have this information there, with some kind of API (or jenkins job) to update it.

It might be beneficial that for a different customer, we may want to promote different integrations.

Is this needed? Personalized promotion would be almost another level of complexity, we don't have anything like this now.

[kpollich]

It might be beneficial that for a different customer, we may want to promote different integrations.

Is this needed? Personalized promotion would be almost another level of complexity, we don't have anything like this now.

I would say this is not in scope for what we're proposing here. Maybe a further future enhancement, but probably something that would need to be handled in Kibana based on the current logged in user, the set of installed integrations, etc.

[mwtyang]

cc @gavinwye @MikePaquette from the Security Solution Team who will also use this feature for the Security Onboarding cc @dandreaj @alisonelizabeth @yuliacech @osmanis @LL6688 @bojanasan

[jen-huang]

@jsoriano, how can we move this forward? We want to have this in place for work that is planned for 8.6.

[jsoriano]

@jsoriano, how can we move this forward? We want to have this in place for work that is planned for 8.6.

I guess we need to prepare a proposal on where this information may live and how it is updated and published. We can maybe start with this for 8.6.

[LL6688]

@jen-huang Hi, are you using this GH for what needs to be done in the next sprint? If not, can you tag the new GH you have? thanks!

[jen-huang]

@LL6688 I added this GH issue to the onboarding tracker: https://github.com/elastic/kibana/issues/141823

This one can be removed from the tracker (maybe I already did?) as it is to think about the long-term solution. It is deprioritized from our end.

[LL6688]

thanks. @alisonelizabeth fyi

[LL6688]

Hi All, From Security solution team, we’d like to pin:

Defend (formerly Endpoint and Cloud Security) Kubernetes Security Posture Management Network Packet Capture Thanks, Mike P.