Configure APM, Infra/Logs and Uptime index patterns centrally instead of in each app

[roncohen]

Configuring index patterns for our apps happens in a lot of different ways today.

• APM uses a setting in Kibana.yml
• Logs/Infra uses a flyout config
• Uptime is hardcoded to heartbeat-{MAJOR_VERSION} and asks people to alias it
• SIEM uses an advanced setting:

I suggest all the solution UIs move to doing using an "Advanced setting" like the SIEM it does and that we try to align on the language.

cc @weltenwort @jasonrhodes @sqren @justinkambic @tsg

[graphaelli]

APM is tracking this in functionality #39741 (not the centralization aspect). Also related: #40704 - if possible, we should consider including the sample data indices in the defaults for UIs that don't currently include them, like APM.

[weltenwort]

The Infra and Logs UI saves the settings in separate space-scoped saved objects. Not sure how well we could revert to the advanced settings storage model. How would per-space read/write permissions be given to subsets of the advanced settings?

[weltenwort]

Another aspect is the ability to provide custom editing controls in the saved objects. Can we insert something like the drag & drop column configuration in that UI?

[jasonrhodes]

@roncohen there are at least two reasons for doing this that I can think of, please correct me/add any I've missed:

1. Consistent user experience for users across our apps
2. Easy ability for each app to read this information in from the other apps if we want to be able to blend the experiences together more easily

For the second one, if this isn't easy to do, we could solve that with a solution similar to our "linking function" solution where we define some sort of "data access" function to return the right index pattern or something along those lines (someday maybe embeddable UI components is an even better solution?).

For the first, I think that's sort of a big problem right now in a lot of ways and needs more thought. We could for example lean on the advanced settings team to support space-aware settings and custom UI elements as @weltenwort outlined (this could be good for Kibana as a whole), or if necessary, we could potentially build some kind of Observability Settings system that at least we could share amongst our groups.

tl;dr is that I'm not sure we can just jump into advanced settings with the systems we already have. Even if we did some kind of default values in advanced settings, we'd likely just make things more confusing with multiple layers of configuration applied on top of each other.

[sorenlouv]

I suggest all the solution UIs move to doing using an "Advanced setting" like the SIEM it does and that we try to align on the language.

Is this backed by settings in kibana.yml? I can imagine that not all customers are interested in ui-only settings so it would make sense to make all options configurable via kibana.yml, and then overwritable and space aware through a UI.

[roncohen]

For the first, I think that's sort of a big problem right now in a lot of ways and needs more thought

Make sense. @tsg do you have plans to move to a settings-per-space model in SIEM?

[roncohen]

Is this backed by settings in kibana.yml? I can imagine that not all customers are interested in ui-only settings so it would make sense to make all options configurable via kibana.yml, and then overwritable and space aware through a UI.

Agreed. Currently APM and Infra use kibana.yml. Infra uses the values there as defaults, allowing users to overwrite them from the UI flyout. APM doesn't give the option to overwrite from the UI yet.

[roncohen]

I didn't find anything in the Kibana docs for SIEM or Uptime in relation to configuring through kibana.yml

[tsg]

For SIEM, it's currently only an Advanced Setting, it's not also in kibana.yml. It was also not Space aware, for us the indices to query is a fairly an administrative setting, for now, so making it global feels better than per space.

I should say that we did this under time pressure before the 7.2 FF and figured that Advanced Settings are better than hardcoding and slightly better than kibana.yml (because it doesn't require a Kibana restart and works better in Cloud). We didn't really consider Spaces at the time but we did say that in the future we'd like to bring this setting in the App itself for better discoverability. We're happy to consider alternatives for making it more consistent with the other apps.

[sorenlouv]

Advanced Settings are better than hardcoding and slightly better than kibana.yml (because it doesn't require a Kibana restart and works better in Cloud)

How do people setup kibana environments via configuration with the desired settings if it's not in kibana.yml? Can they start kibana with some flag somehow?

I completely understand why this approach was taken due to time constraints. I think the solution we should strive for (if time permits) is that users can apply settings without going through the UI. On top of that they should be able to overwrite the settings via the UI (which shouldn't require a restart and is easier for cloud users).

[tsg]

@sqren I'm not aware of such flag. I guess they could use the Kibana API to change the setting or even the Elasticserach API to modify the saved object directly. I see how editing the Kibana config file would be easier in this case, though.

[justinkambic]

I didn't find anything in the Kibana docs for SIEM or Uptime in relation to configuring through kibana.yml

Regarding Uptime - as of today we don't provide any configurability for index patterns and advise users to create aliases for our index if needed. I'll continue to monitor the discussion here and try to make sure we conform to best practices that come out of it.

I think the solution we should strive for (if time permits) is that users can apply settings without going through the UI. On top of that they should be able to overwrite the settings via the UI (which shouldn't require a restart and is easier for cloud users)

++

[sorenlouv]

@roncohen APM has an issue open to allow kibana.yml settings to be overwritten via the UI (no restart required): https://github.com/elastic/kibana/issues/39741 Should we move forward with that or close it in favor of what you are proposing here?

In my opinion it might be easier to get started if each plugin has their own settings page, as long as we align on the language, design and how it works. We could even share the same UI components if that makes sense.

[roncohen]

OK. In addition to what's already been brought up, there are a couple of things to factor in:

1. will each solution need an in-app settings page anyway, maybe for lack of options with regards to UI components available in the Kibana Advanced Settings pages? Sounds like that might be the case. It's probably better to have an "advanced" section on an in-app settings page, than to split the settings across the two options
2. for each solution, how rare do we think it is for a user to change the index pattern? Is it something that we even want to support? this goes to a bigger question around how accommodating solutions will be to non-standard data etc. For example, we recently decided to move towards hard coding the infra/logs UIs to rely on ECS and remove the options to override the standard field names. For APM, we've discussing allowing users to only customize a postfix, while forcing 'apm-' prefixes. In a sense, hiding the index pattern options in an "advanced" area is a good start towards removing the option to completely customize them, if thats where we're going.
3. Finally, since we'll soon have alignment on the top bar (https://github.com/elastic/observability-dev/issues/150) it might be a good option to go for in-app settings pages across solutions. The work then becomes ensuring that the design of those pages are aligned cc @elastic/observability-design

tldr; Let's work towards standardizing on a design/components for in-app settings pages instead of relying on the Kibana Advanced settings pages.

[roncohen]

Closing this issue to instead start the discussion on aligning on design for in-app settings pages across solutions.