Dashboard level index patterns

[stacey-gammon]

Based off the numerous feedback given in https://github.com/elastic/kibana/issues/3668, this issue is to dig further into a specific implementation.

What problem are we trying to solve?

A common use case is having many dashboards with the same visualizations that differ only by index pattern. This creates a scaling problem. If you have X dashboards, with N visualizations on each, a user has to create and update X * N visualizations. With a dashboard level index pattern, they would only need to maintain X dashboards which all use the same visualizations.

Current available workaround

You can achieve the same goal by using index patterns with matching prefixes. For example, if you have index patterns called animals-*, animals-cats-*, animals-dogs-*, you can create visualizations based off the shared prefix, animals-*, and use a filter on each dashboard to segment your data.

Where does this workaround fall short

I'd like to hear more from the community on this front, but one way is discoverability. It is not an obvious solution.

How could we improve on the workaround?

We could allow users to specify an index pattern at the dashboard level that will override any index patterns specified on the individual visualizations.

UI

The simplest approach would be an extra option:

Selecting an index pattern would cause it to be applied to all visualizations, instead of the index pattern chosen with it:

Challenges

This might be unintuitive or confusing since this flow still has users saving index patterns with their visualizations. I can't think of a good way to get around this since knowing the available fields is crucial when creating a visualization.

It's also possible that a visualization has fields that don't match the index pattern chosen at the dashboard level so error handling for this situation is important.

It's also possible for visualizations to use multiple index patterns. Do we replace all the index patterns with the dashboard level one or should we allow for finer grained control, like deciding which patterns to replace with which other patterns?

What's lacking

This solves the issue with needing to create and maintain duplicate visualizations, but keeping the dashboards in sync would still require duplicate actions (e.g. adding or removing a visualization for each dashboard).

This secondary problem I believe could be solved by nested dashboards.

Even if nested dashboards doesn't solve this secondary issue and we want to introduce the concept of dashboard templates in a different manner, we'll still need the ability to set dashboard level index patterns, so this is a pre-requisite step no matter what (even if the UI changes).

cc @elastic/kibana-sharing @alexfrancoeur @AlonaNadler

[ReanimationXP]

Splunk handles this problem by allowing user variables (with defaults) within dashboards, along with end-user-facing controls to populate them. Visualizations can then optionally use/substitute those "global" dashboard variables within their assigned search string. There isn't a concept of limiting or assigning a visualization to one index - each visualization has a search string associated with it containing one or more indexes that can be populated by dashboard "global" variables. This also inherently can make creating a visualization from a search a one-click operation, optionally adding in a dashboard variable if desired.

[stacey-gammon]

from @ppisljar:

Currently every visualization is based on specific index pattern (expressions don't really matter at this point). So setting index pattern on dashboard level might have really unpredictable effects. Your visualization index pattern might have completely different field than dashboard index pattern selected. Having all visualizations on dashboard use same index pattern would solve that issue, but i guess its fairly common to have dashboards where that's not the case ? We might have something like default index pattern we set for dashboard, and then only visualizations using that same pattern would be updated to use the new selection. We would certainly need a way to limit this, and also limit what index patterns can you switch between (only the ones with the same fields ?)

With expressions we bring in another level of complexity, where one visualization could be using multiple index patterns (retrieving your final data by doing multiple es request). Still by defining the default index pattern for dashboard and limiting switching to only compatibly index patterns this should still work same way as if we would be using one index pattern only.

a hacky implementation for visualizations shouldn't be too hard (something like have dashboard pass in 'defaultIndexPattern' and 'indexPattern' then in visualize_loader check if (vis.indexPattern === defaultIndexPattern && indexPattern !== defaultIndexPattern) vis.indexPattern = indexPattern; long term we could make this work with variables in expressions which are currently planned for as soon as we are done with NP migration.

[tomryanx]

From my thought bubble....

Have an option in viz configuration to allow index-pattern overrides (with a link to the relevant docs or a helpful mouseover). Then in dashboards, expose an index-patterns option if and only if the dash has visualisations with the option enabled.

Ideally the dashboards index-pattern UI would indicate the present value/s, and possibly allow either dashboard-wide override or per-present-value granularity. Ideally those visualizations not effected by the override would be desaturated/dimmed while the dashboards index-pattern UI is active.

[andrew-goldstein]

Synchronization Issues With Kibana Index Patterns Referenced by Embeddables

In summary, when embeddables, including Lens, and other visualizations created in the Visualizations app, (i.e. Line charts, Maps, and Tag Clouds), reference Kibana Index patterns, the title property of a Kibana Index pattern (e.g. auditbeat-*, endgame-*, filebeat-*, packetbeat-*, winlogbeat-*), stored in the Kibana Index Pattern saved object, will become out of sync with apps, like the SIEM app, that don't use Kibana Index patterns, per the screenshot below:

Details

The SIEM team is planning on rendering Kibana dashboards in the SIEM app via the Embeddables Dashboard. Ideally, (for our use case), the Embeddables Dashboard component would reference the saved object id of a Kibana dashboard to allow (direct) reads and writes (e.g. layout mutations) to the saved Kibana dashboards from within the SIEM app.

Every Kibana space where the SIEM app will be used must be seeded with the following saved objects, to utilize realize the above using embeddables:

• Kibana dashboards, that in-turn reference (by saved object ID), multiple Kibana visualizations
• Kibana visualizations, including Lens, and other visualizations, e.g. charts, maps, and (TBD) SIEM-specific emeddables. Today Lens, and all of the existing visualizations described above require a reference (by saved object ID) Kibana Index Patterns
• Kibana Index Patterns are required by Lens and the other visualizations described above to both render data, and provide an interactive editing UX for the visualzation itself. Note: The Kibana > Index Patterns Management UI does not provide an affordance for users to edit the title property of the UI. Per a chat with Spencer, it should be possible to update the title property of a Kibana Index via the Saved Object API, however the Fields in the index pattern must be programmaticly refreshed after the tilte is updated.

A (Not So Great) Workaround

It should be possible to programmatically update the title property of a Kibana Index Pattern via the Saved Object API, however the Fields in the index pattern must be programmaticly refreshed after the tilte is updated.

Ideally it would be possible for the SIEM app to pass (at runtime, perhaps through props) it's configured index pattern to the Embeddable Dashboard, so it could-in turn pass the pattern down to the vizualizations it rendered. If this mechansim was made availble as a platform feature, the platform feature would also need to address the edit-time UX of existing visualztions, which today require a reference to a Kibana Index Pattern.

[richievos]

@stacey-gammon where did that quoted text come from? I'm trying to find more on "variables in expressions" but I'm not seeing any hits.

fwiw @ReanimationXP's approach and experience with Splunk is exactly the solution I want. Let the dashboard be dynamic on everything, then tings like dashboard level index patterns are just an implementation detail of how someone decides to build their dashboard.

[ppisljar]

https://github.com/elastic/kibana/issues/46908 this is the relevant issue which is still missing the details, but the idea is to add variable support to expressions, so when running your expression you can pass this variables in, and inside your expression you can use either special functions or special syntax to provide variable arguments to functions.

something like

esaggs indexPattern=@indexPattern aggs={....}

[rayafratkina]

Quick update here: initial support for variables in expressions was added in https://github.com/elastic/kibana/pull/54788

[monfera]

Regarding the original problem @stacey-gammon described:

A common use case is having many dashboards with the same visualizations that differ only by index pattern. This creates a scaling problem. If you have X dashboards, with N visualizations on each, a user has to create and update X * N visualizations. With a dashboard level index pattern, they would only need to maintain X dashboards which all use the same visualizations.

Data Views (née index patterns) could be composable. It can be a more powerful, yet simpler option than the creation of some special mechanism, as the user wouldn't need to learn new concepts. Such composition was one of the motivations for reimagining Data Views (search for compos... here and see this comment by @ruflin and this comment)

I see references to nesting dashboards; feels like the motive is similar, though composing with Data Views is more expressive than nesting dashboards (which would represent too much coupling, and doesn't really exist in analytics anyway). Eg. can be used outside Dashboards 😄 There's no need to turn dashboards into de facto Data Views. Requires some entity relationship modeling to do it well. There are a couple of ways for including variables in Data Views. It looks sufficient.

In short, a Data View can rely on, and augment, another Data View. Example:

• a bunch of dashboards, presentations or presentation pages share some indices, field sets, formatters, calculated fields, etc.: create a Data View for it
• within this, individual dashboards may each have a specific Data View, if needed
• charts within a dashboard can rely on these dashboard specific Data Views, or some chart group may even require yet another Data View

There's no need to make Data Views exclusive to the dashboard level. For example, a bunch of charts shared on diverse dashboards may have enough commonality to warrant a common Data View.

The role of Data View doesn't necessarily start at the dashboard level. The Data View is an abstraction over the physical ES indices, and

• can let users use friendly field names; hide fields unnecessary for given tasks (making Lens field picks easier)
• the physical ES indices may change underneath, and it's enough to update one, or few Data Views (the dependent Data Views, dashboards, chart configs need no update)

[grayblack]

Just a heads up that this can also easily be done with kuery: '_index:animals-dog*'

This allows you to construct an embedded generic dashboard by following: Custom Kibana URLs

for example in the URL include: query:(language:kuery,query:'_index:animals-dog*')

[ThomThomson]

Closing this issue in favor of https://github.com/elastic/kibana/issues/134706 as adding variable support more broadly to Embeddables would cover this use case and more!