[Logs Explorer] Support for logs backed data views

[ruflin]

In Logs Explorer, currently if a user clicks on a data view, the users is redirected to Discover view. But there is a list of data views where we know these contain logs data. These should be able to stay in Logs Explorer and view the data. In the Data View drop down, the data views which are supported by Logs Explorer should be visually separate from the other data views. For example if a user clicks on metrics-*, the user should still be redirected to Discover.

The initial list of data views which should be supported by Logs Explorer are:

• filebeat-*
• logstash-*
• auditbeat-*
• Any data view pattern that matches logs-*-* like logs-foo-*

If the user clicks on any of these data views, these are shown in Logs Explorer by default. The basic assumption is made, that in most cases the above data is in data streams. In case a relevant log field is missing / not in ECS, the UI must handle it gracefully.

The above helps with the long term goal of bringing users to logs-*-*. It allows us to offer migration options for users using filebeat-* as an example. In addition, we can support these users migrating to ECS/semconv. The scope of this issue itself is limited to support the list of data views above.

Note: This partially conflicts with the description in https://github.com/elastic/kibana/issues/169964 and we need to coordinate with the Discover team

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[mwtyang]

This makes sense in our current stateful offering, where separate Discover and Log Explorer apps are under main and sub left-navigations. Once users find their way to the Data View tab in the current stateful Log Explorer, allowing users to stay in the Log Explorer makes sense.

Serverless and future stateful offerings with unified left navigation and a single Discover app with two tabs - Data View & Log Explorer; I'm not clear on the suitable UX for this. Users will see top-level Data View and a sub-tab Data View under Log Explorer.

cc @ryankeairns @ninoslavmiskovic @timductive

[alvarolobato]

Thanks Nicolas for creating this. I agree that when a user is in Logs Explorer and click a logs specific data view they should be able to stay in discover instead of jumping to discover directly and keep them within the logs specific experience. They are always free to jump into discover when they want but we shouldn't force them. cc. @evin-elastic @abhiksingh

[ruflin]

Users will see top-level Data View and a sub-tab Data View under Log Explorer.

The final PR (https://github.com/elastic/kibana/pull/171467) that was merged for https://github.com/elastic/kibana/issues/169964 calls the tab "Discover" and not "Data View". This should solve the UX issue.

[tonyghiani]

Based on an offline conversation with @ruflin, we spoke about some additional specs.

✔️ Acceptance Criteria

• The following data views, when selected, should keep the user on the Log Explorer:
  • Anything that starts with logs-: e.g. logs-foo-bar.
  • Anything that starts with filebeat-, winlogbeat- or auditbeat-. We can later add also logstash-.
  • Any CCS prefix to one of the above patterns should be supported.
• Any data view matching the above criteria...
  • should be sorted on top of the data views list in the selector.
  • should load the related additional fields for the stored data view.
• Any data view NOT matching the above criteria...
  • should redirect to Discover as currently does when selected.
  • should have a visual element that tells the intent of that entry (go to discover).
• When a user refreshes the page, it should correctly restore the selected data view.

🎨 Design

@isaclfreire can we look into a design option to mark each Non-Logs data view to tell the user it will navigate to Discover? We currently have the generic badge shown in the description's screenshot, but that's only a presentational element and applies to all the data views.

💡 Implementation hints

• We can treat it as an allowed list of patterns (strings or regexp) to match the target data views we want to browse in LogExplorer. There is already a state machine to control the data views retrieval and sorting, we could probably apply this new piece of logic here.
• As an initial approach, we will derive a new ad-hoc data view starting from the persisted one. This means we'll need to add a lookup step to retrieve the data view by its ID when we should restore it on refresh/navigation. The log explorer controller state machine is the place to look for this step.

[tonyghiani]

@ruflin while talking with @weltenwort, he raised an interesting point about data views that partially match the criteria defined to stay on the Log Explorer.

For example, say there is a data view with a pattern which is logs-*,metrics-*. Although it's not the typical use case, users can still create data views like this, so the question is if we should keep redirecting them to Discover, or extract the part related to logs and keep them in Log Explorer.

My opinion is that we should keep them on Log Explorer only when the selected data view exclusively matches a log-related pattern from those defined earlier, do you have any thoughts about this?

[ninoslavmiskovic]

Small comment here - we are working on enhancing the field list component in the near future, so we need to coordinate around that area.

Also it is very common that users are having several tailored index patterns to a dataviews that are not necessarily isolated to a specific type of data e.g logs .

[ruflin]

For example, say there is a data view with a pattern which is logs-,metrics-. Although it's not the typical use case, users can still create data views like this, so the question is if we should keep redirecting them to Discover, or extract the part related to logs and keep them in Log Explorer.

Good find. In case of doubt, let's always link to Discover. Each index pattern must match our criteria. So for your example logs-*,metrics-* we send the user to Discover. Seems like we are all aligned.

@ninoslavmiskovic We definitely need to coordinate on the field list but it is not clear to me how it affects this change. Can you elaboarte?

[ninoslavmiskovic]

@ruflin

I was commenting on this piece :

"should load the related additional fields for the stored data view."

If we want to something else here then what we do today 👍

[ruflin]

Thanks @ninoslavmiskovic I expect the behaviour we do is exactly what data views do today and if it changes for data views, we will change it accordingly.

[isaclfreire]

@isaclfreire can we look into a design option to mark each Non-Logs data view to tell the user it will navigate to Discover?

I can for the short term, but I don't think that's the design/UX solution we should be looking long term here - I would prefer to put our efforts in a more meaningful discussion.

Being Logs Explorer a tab of Discover now, in a certain way it becomes part of Discover to the user. It's not a separate app anymore (even if in the background for us it is). We should address the navigation between different ways of viewing logs data inside what we call "Discover" regardless the tab and how it works back and forth.

[isaclfreire]

I have been giving a thought to this issue and started to put down the user flows for short and long term scenarios in this Figma file.

Short-term solution My criteria and assumptions for this proposal are:

• Regardless if the user arrives from the 'Logs Explorer' link within the Observability menu of their stateful deployment, or from the 'Discover' link of their o11y project in Serverless, they should land in the 'Logs' tab (the default/preferred experience for o11y users).
• Regardless if the user is in the Discover tab or in the Logs tab, they should be able to navigate to both Data views or Integrations. It's their data after all, so we should show everything. We can discuss if there should be a default one when the selector is active (e.g. Data views for the Discover tab, and Integrations for the Logs tab).
• At the moment, both data selectors (Discover and Logs tabs) are different components, but I think we should try to make them as closer as possible, as they share the same objective. So one day, they can become one component.
• If users are in the Discover tab data selector, and click on a logs-only data view I don't think it's enough to only label where this data view will be opened. The difference between the apps Discover/Logs is a concept we've never presented to the users, and are familiar with only internally. When they make their selection, we should provide them with a message/modal communicating we have a new dedicated experience for Logs they can try on (and opt in as preferred experience if they wish), or they can decide to still rely on Discover flavour, as they might be used to it. This allow us to track how many make the switch or not.
• Users should have the possibility to change back to the original Discover flavour from the Logs tab at any time.
• If the above proposal comes through, my only question is: for the short-term solution, should we only show logs-only data views to users inside the Logs tab data selector, or are we able to provide a comprehensive list all the way from the start?

Long-term solution This is the direction that seems the most logical to me to pursue in the future, but I'd like to back this up with user feedback, testing and research. Regardless from where the user comes from (o11y serverless project, or stateful deployment), the data selector is one component that provides access to all data available to the user. Depending on what the user clicks on (data views, integrations, etc), we act in the background to provide the best experience/flavour for them to consume that data. But it's all Discover.

Wdyt? Feel free to leave comments on Figma as well, if you see any discrepancy.

[weltenwort]

@isaclfreire that vision makes sense to me when we assume that vanilla Discover and the Logs Explorer will converge more over time. From the past year of work on this I haven't gained any confidence that this is actually going to happen. So if we move down a path based on that assumption we should have explicit and well-informed consent on all sides about this and its implications.

One way to look at it in the long term could be to answer questions like these:

• Will Discover be the storefront for all signal-type-based exploration?
• If so, how?
• Or will it be a generic fall-back?

[ninoslavmiskovic]

@weltenwort

Those are all good questions and something that needs to be explored since it is for the long term.

Something that the O11y and Discover team need to work together on together.

[isaclfreire]

I've put together a prototype to explain what the short-term navigation could look like. You can find them here:

1- Stateful use case 2- Serverless use case

I think it'd be interesting to review these together sometime soon. I have not address a couple of things in these:

• ESQL use case
• Planned UI changes to the unified search bar

cc @MichaelMarcialis @andreadelrio

[mwtyang]

@isaclfreire do we need both Integration and Dataset subtabs? Isn't Integration a representation of Dataset?

[isaclfreire]

Hi @mwtyang thanks for raising the question. I would say that Integration is a way of categorising datasets by what kind of 'package' originated them, rather than its representation. The main character is still the dataset. The mainly assumption is that users might have different browsing preferences: (A) start from picking the integration and then the dataset that belongs to it, or (B) just searching for a specific dataset name. Also considering that some datasets might not belong to any integration at all, so the list is more comprehensive.

@ruflin please correct me if I missed something

[ruflin]

Dataset vs integration: We need both but I'm challenging it it warrants 2 tabs.

[ruflin]

I understand how this issue triggered a wider discussion and we need to find answers for the open questions. But lets continue this discussion in a separate issue and get back to what this issue was about initially: Support logs backed data views. I believe we can have a simple solution for this problem and in parallel solve the overall issue.

[isaclfreire]

Makes sense. Just to keep us on track, here's the latest designs on this issue's topic:

I've put together a prototype to explain what the short-term navigation could look like. You can find them here:

1- Stateful use case 2- Serverless use case

[tonyghiani]

Adding final design ready for implementation:

🎨 Logs-only data views design

https://github.com/elastic/kibana/assets/34506779/ab3e1337-e21f-45df-9872-147a387d4c4c

[tonyghiani]

The research phase on this is completed and we are ready to move this to its implementation step, which we are tracking on https://github.com/elastic/kibana/issues/175767.