[OneDiscover][Extension] Extension Point for Recommended Fields in Discover

[ninoslavmiskovic]

WHY:

In Discover, users often encounter difficulties in finding relevant fields. Fields are currently grouped generically, but most users—particularly new or junior users—struggle with understanding which fields to use, such as whether to choose host.name or host.hostname. This can lead to a frustrating user experience, as users must often scroll through an alphabetically sorted list of mostly irrelevant fields.

The Recommended Fields extension point offers a more opinionated, curated list of fields that Elastic believes are most useful for that particular use case, taking away the guesswork and improving the overall UX, especially for newer users. This would allow users to focus on the right fields immediately, reducing time spent searching and ensuring more effective data exploration.

WHAT

Introduce an extension point to define a Recommended Fields section that presents a curated set of fields based on the contextual profile defined and powered by the contextual architecture. This extension point will:

• Surface fields that users always use (e.g., event.dataset, service.name, host.name, etc.), functioning as a more opinionated version of the “Popular Fields” list.
• Provide a far better UX for new and junior users by automatically showing the most relevant fields upfront, taking away the guesswork involved in choosing between similar fields.
• Allow the list to dynamically adapt based on the contextual profile defined and become more contextually aware. For instance, if a user filters for event.dataset:"kubernetes.container_logs", the system can bubble up more relevant Kubernetes-specific fields, providing a more contextual, task-driven experience. A similar experience we already provide with existing extension points for log exploration.

BENEFITS:

• Reduced Complexity: Users no longer have to scroll through irrelevant fields or guess which field is appropriate. They can rely on Elastic’s recommended fields to guide them.
• Improved UX for New Users: The opinionated field list dramatically enhances the experience for new or junior users, giving them a curated set of fields without overwhelming them.
• Contextual Adaptation: As the system evolves, it will get smarter, dynamically surfacing more relevant fields based on user filters and contexts (e.g., showing Kubernetes-specific fields when dealing with Kubernetes logs).

EXAMPLES:

An example of recommended Fields for Observability could be :

• event.dataset
• service.name
• host.name
• message
• log.level

An example of recommended Fields for Security could be :

• message
• event.category
• event.action
• host.name
• source.ip
• destination.ip
• user.name

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

I'm interested in e.g. Severity, would this be a group of severity related fields, or do you mean it would allow to filter by predefined/prequeried severities?

[ninoslavmiskovic]

It was just an example. But good question. It could either be the fields for severity like log.level or the actual values like "error" 🤔

It depends on the consumer.

Perhaps the real question is if a category could be a field ?!

[timductive]

The examples listed here are confusing to me. The only one that looks like a category to me is "Recommended fields" which maybe should just be one of our generic categories and the recommendations themself would be the "contextual" component. All of the other examples here look like pre-configured fields to me which should be a different task, I would argue that this could be done via recommended queries.

[ninoslavmiskovic]

@timductive This is an extension point, so the examples were just examples of how the consumers can create categories or more precisely sections in the field list. You are right in terms of them not being field categories but rather relevant fields that are relevant. This is why I have used the name "Section".

For adding the contextual part underneath the "Recommended fields", could be a place to add the pre-configured fields based on the contextual profile. E.g. if I am exploring logs*-* or we know that you are in O11y area of Kibana or serverless project, then show these contextual fields under "Recommended Fields"

[gbamparop]

Another example of categorisation could be based on ECS when relevant, e.g. a cloud section with appropriate fields, this would be more of a hierarchical view though based on prefixes and a similar outcome could be achieved through field search.

[LucaWintergerst]

The initial use for Observability would be a set of recommended fields that is applicable to all sources of observability data. Some fields that a user always uses. In a way this is a more opinionated list of "popular fields". Fields that Elastic thinks should be popular.

In addition this also takes some of the guesswork away from users. Most users have no idea if they should use host.name, host.hostname for example. And likewise, if they want to look for a particular service, they may not know that in ECS the field is called service.name. So in addition to them not having to scroll/search for the field anymore, more junior users also have a far superior UX since we show a curated set of fields right away.

Recommended Fields:

event.dataset
service.name 
host.name
message
log.level

Later on we'd like add additional groups to this as we get smarter in using this. For example, if a user were to filter for event.dataset:"kubernetes.container_logs", then we could bubble up some of the more relevant kubernetes fields to the user Today, the UX for this is pretty bad, as I just get a alphabetically sorted list with mostly irrelevant fields

We would use this extension point to be more opinionated with the field selection, offering the alphabetical list as a fallback. So if a user looks for kubernetes data we'd eventually show them more relevant kubernetes fields under a separate section at the top, so they don't need to scroll for ages to find what they might be looking for.

[ninoslavmiskovic]

@timductive + @LucaWintergerst

I’ve updated the GH issue to focus on Recommended Fields driven by a contextual profile and architecture.

For example, @LucaWintergerst, we already have a K8 contextual profile that pre-selects default columns. We could extend this to showcase “Recommended” fields.

@paulewing – I added some Security examples for you to review, based on those you already use in Timeline. These are just examples for now.

The goal is to deliver the extension point and control recommended fields based on the contextual profile.

[flash1293]

event.dataset

Actually, there are two main "flavors" of this - event.dataset and data_stream.dataset. They should be the same, but the data_stream.dataset is special in the sense that it uses a constant keyword field, so it can exclude indices based on the mapping alone without having to actually look at the data.

If possible, filtering should be done on the data_stream.dataset because of this. There might be setups where event.dataset is set, but data_stream.dataset won't be - not sure what to do about those.

Mostly a tangential to the extension point itself.

[davismcphee]

Another example of categorisation could be based on ECS when relevant, e.g. a cloud section with appropriate fields, this would be more of a hierarchical view though based on prefixes and a similar outcome could be achieved through field search.

@gbamparop This sounds similar to an ON Week PoC I worked on to add field list sub-groupings to Discover profiles. Not that we'll necessarily go this direction, but worth sharing for reference: