[Obs UX] Allow passing of selectable options in Lens charts

[roshan-elastic]

Problem

Currently, when a wants to start exploring visualisations to further their investigation within the Observability UI (e.g. Hosts), they can only do so effectively if they know ECS and elastic intimately because of the need to understand field names and how they are used. This makes customisation daunting for the new user at best.

Example workflow

Solution Hypothesis

"By allowing engineers who in leverage lens to pass in sensible default options into the lens vis' they embed, users will be able to explore their problems and save customisations quicker by leveraging this context. We expect to see an increase in the usage of lens visualisations in the O11y UI"

We would like to be able to pass in some kind of sensible defaults that users can options within Lens visualisations to allow users to customise selections without needing to learn ECS.

Example of how default disabled options could be presented to users

[roshan-elastic]

@smith - What do you think of this request? It wouldn't have to be this solution specifically but I think the ability for us to pass in more options (but disabled) into lens visualisation, users who do edit them will have some guard rails (vs throwing them in the deep-end with needing to know ECS)

[roshan-elastic]

@teresaalvarezsoler @smith - wondering if you'd seen this? It's pretty amazing!

Editing lens-powered visualisations but with O11y guardrails

https://github.com/elastic/kibana/assets/117740680/a83f0313-8922-4d68-9810-88f662aef9ec

(cc @drewpost)

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[andrewvc]

I really like this @roshan-elastic , I'm guessing the example here is for a 'Host' Vis?

[gbamparop]

@tsullivan I've seen you tagged the Logs UX team. I think this is meant for @elastic/kibana-visualizations ?

[dej611]

I think we brainstormed this kind of idea multiple time, but haven't consolidated a specific roadmap for it. I'll raise the concern in the next team weekly sync. cc @timductive @ninoslavmiskovic

[drewdaemon]

The specific disabled dimension idea is described in https://github.com/elastic/kibana/issues/134315.

The general proposal to guide users WRT solution use-cases feels like suggestions v2 material.

[roshan-elastic]

I really like this @roshan-elastic , I'm guessing the example here is for a 'Host' Vis?

Hey @andrewvc, that's right. The initial example I show is for Hosts and the nice thing is in synthetics.

My thought here is that we don't necessarily need an all-singing dancing Lens vis that would be capable of understanding all fields etc - just something that we could pick up and use from lens where we can easily provide configurable user-friendly 'edit' views (vs jumping off the 20m diving board :) )

(Not hating on Lens here, just looking to provide our O11y users with some armbands to help them boss it)

[stratoula]

Talking about UX I would also suggest to consider opening the in-app editing flyout instead of navigating to the editor. It was introduced in 8.11 https://github.com/elastic/kibana/pull/166169 and you can see how it works on dashboards.

I see that you are opening the Lens editor into a new tab. Maybe is a better UX to open the flyout in the Obs apps and have this disabled dimensions there. So the users don't lose context, they can enable and disable dimensions and see the change immediately in the obs UX.

[ninoslavmiskovic]

@stratoula I agree . The in-app editor for Lens embedded charts is on the roadmap 👍

[stratoula]

A clarification here :) You can still do it but we want to investigate if we can make the api easier for the consumers. Ping me for more technical details

[timductive]

hey @roshan-elastic thanks for starting the discussion. To me this rolls up under a growing theme of making the tools "context-aware". Aware of the schema, notable/popular fields, defaults for dimensions, etc.. In general, this is something that we should collaborate closely on and we have a few related topics: Suggestions v2, schema-aware field list, and this as smart or configurable metrics.

From both a product and technical implementation perspective my opinion would be: We should bring Lens to you, open Lens configuration directly in o11y via the inline-edit mode as @stratoula mentioned. Then, we give you interfaces/hooks to build more opinionated/custom options into the inline-editor (or separately and then passed in). The idea being that this would be a more tightly integrated system like Metrics Explorer but where o11y still owns the implementation.

[roshan-elastic]

Hey @timductive, thanks for the feedback.

To play this back, it sounds like what we'd want to aim for is something like the following:

• User is in O11y
• They see a chart and want to start exploring/editing the data in it
• They click edit
• The lens chart can be edited in-line
• O11y can curate this to pass in some helpful options (e.g. some metric options like CPU or memory or dimensions like 'host name) - effectively allowing users to manipulate the chart without needing to understand any Elastic fields if they don't want to.

Something like that would be perfect!

[timductive]

Yes @roshan-elastic in my mind this would be a good starting point and we would have to figure out an implementation plan to provide that interface for custom options. It also could be that the custom options are too complicated to be housed in the inline-edit flyout (like metrics explorer), in this case maybe Lens just needs a way to pass in the metric and O11y owns the full interface... Is this what you had in mind @stratoula and @ninoslavmiskovic ?

[stratoula]

@timductive @roshan-elastic I might be confused but I see 2 different topics here:

• the one is that we want to have some dimensions/fields preconfigured but disabled and the user opens the flyout and can easily enable/disable. I think this is something we want to do in the Lens editor and is this issue https://github.com/elastic/kibana/issues/134315
• give the consumers the ability to have their own custom options panel which will be part of the flyout but not owned by us. So the flyout should allow passing an interface with custom options but this will be manipulated and owned by the consumers.

Personally I think that both are valid and we should do them.

(e.g. some metric options like CPU or memory or dimensions like 'host name)

This example here given from @roshan-elastic I think should be handled from the first solution. It is bad UX imho to have preconfigured dimensions elsewhere than the lens layer configuration.

[roshan-elastic]

This example here given from @roshan-elastic I think should be handled from the first solution. It is bad UX imho to have preconfigured dimensions elsewhere than the lens layer configuration.

Hey @stratoula (@timductive), yeah - this is exactly what I was thinking.

give the consumers the ability to have their own custom options panel which will be part of the flyout but not owned by us. So the flyout should allow passing an interface with custom options but this will be manipulated and owned by the consumers.

Yeah, this feels like enhanced (more risky but v powerful) functionality that would benefit customers who are using their own field names (a lot of existing customers).

Taking a step back... I just wanted to say that nothing has been prioritised on the O11y side yet so I'm really happy to be talking about this but I don't want to use up your time if we need to have resource coordinated on our side to think about this (@smith / @crespocarlos WDYT?)

[crespocarlos]

Hey @roshan-elastic, SGTM.

I see a couple of things here though

1 - Usage of the in-line edit feature: This is nice. but users can't really edit the charts in the Host view, because it's not a real dashboard. All they can do is inspect the formula and maybe create a duplicate version of it to be used in a custom dashboard. But regardless of that, it's better to let users open the flyout instead of redirecting them to a new page. It's just that none of the changes made will persist - which can be confusing.

2 - One thing that seems to be clear in this thread is the fact that we'd like to provide a better UX around formula creation, this ticket https://github.com/elastic/kibana/issues/167089 covers that. We could try to contribute to the discussion there?

[stratoula]

It's just that none of the changes made will persist - which can be confusing

I see, I got it wrong. I had understood that we are keeping the changes the users make. So yes if we don't this will be possibly confusing 🤔

One thing that seems to be clear in this thread is the fact that we'd like to provide a better UX around formula creation, this ticket https://github.com/elastic/kibana/issues/167089 covers that. We could try to contribute to the discussion there?

Yes can you describe your use case on the suggestions v2 ticket?

[roshan-elastic]

Hey @crespocarlos @stratoula,

1 - Usage of the in-line edit feature: This is nice. but users can't really edit the charts in the Host view, because it's not a real dashboard. All they can do is inspect the formula and maybe create a duplicate version of it to be used in a custom dashboard. But regardless of that, it's better to let users open the flyout instead of redirecting them to a new page. It's just that none of the changes made will persist - which can be confusing.

I see a couple of scenarios here - making changes to the current view and saving vs saving it somewhere else. Here are a couple of examples to illustrate what I'm thinking:

1. User wants to change the columns in the table and save them

• User edits table
• User sees a few unused fields (e.g. Disk Max - I can imagine us only allowing some presets here)
• User changes the selections and saves it as a new 'view' (maybe they set that new view as the 'default')

This might not need any lens editing at all...e.g. ddog handle this fairly simply but restrictively here:

2. User wants to explore the chart in more detail and save it in a dashboard

This is what Carlos was referring to. Editing in-line might be confusing as you say.

3. (unsure of whether we would need this right now) User wants to edit some of the charts and save in the current view I can imagine a user wanting to be able to change and saving a chart by editing in in-line but I'm not sure about this...1 and 2 I believe are the primary things right now...

Note : I can imagine users perhaps just wanting to select particular metrics to show in the table AND the charts (i.e. just configure the metrics you're interested in and the whole UI updates...rather than having to edit each one). Not sure though...I'd imagine this would be handled by us rather than any kind of lens editing

[stratoula]

@roshan-elastic this makes sense.

1. If this is handled by you, is there something that you will need from us?
2. This is already possible, you don't need anything from us

[roshan-elastic]

Hey @stratoula,

This is already possible, you don't need anything from us

This is the main gap as I see it. Once you start editing a lens chart, the user doesn't have any kind of 'guardrails' to help them carry on their investigation (as per the problem)

For example, they'll see all of the elastic fields, they'll see no metrics which make sense (it'll be elastic fields again) and you generally need to be an expert in Elastic to do anything with it.

I'm not sure how to solve this one idea was in the solution hypothesis.

I'd imagine it's probably best for @smith 's team to think about what we want to request but I'd imagine a collaboration between O11y and platform would figure out the best solution?

[stratoula]

I see. I am bit confused with the UX.

So if I understand correctly we want the users to start from the Obs UI but end at a dashboard, correct? If they decide to edit the chart they cant save the changes in the Obs UI. I just wonder loudly if this is the best UX :)

Anyway I get the request now, we are happy to help. As I said personally I think that pre configured dimensions or more personalized suggestions make perfect sense to me and I think is the path forward.

[roshan-elastic]

I just wonder loudly if this is the best UX :)

Yeah, this is the question...from a user POV I think it's fairly clear that they should see other dimensions and metrics related to whatever they're looking at.

For example, if they're looking at a trend of the top 10 most CPU bound hosts...they should be able to select other metrics (or dimensions) related to those hosts...but it depends on the chart.

At the current state is that as soon as they edit the chart, they need to understand exactly what fields they want need to show the metrics/dimensions that they want to see (e.g. memory usage, disk usage, availability zone) so it's trying to make it so we can give them a curated view which makes it easy for them to see what they're looking for (vs a mass of Elastic fields).

If that makes sense?