KQL autocomplete should show existing fields and not all fields

[ruflin]

The current Kibana autocomplete feature shows all fields which are defined in the index pattern. As Beats creates the complete index pattern with all fields on setup this means the query auto complete feature in Kibana shows all fields when starting typing even though most fields do not exist. My expected behaviour would be that by default only the fields that exist in this certain time frame would be show. Kibana seems to already have the knowledge which fields exist in the selected time frame based on the field list on the left.

My assumption is that most of the time users want to filter on fields that exist in the time frame. Otherwise they would see empty results. There could an option to turn on all fields or have Show more ... on the bottom of the list if needed.

As an example see the two screenshots below. When starting to type be I would expect only to the beats fields to show up but all show up even though there is not data for kubernetes or elasticsearch.

Reproduce

• Run Kibana Master
• Run ./metricbeat setup
• Run ./metricbeat
• Go to Kibana discovery tab

[ruflin]

@tsg @monicasarbu @tbragin @alexfrancoeur FYI

[alexfrancoeur]

@AlonaNadler @Stacey-Gammon any thoughts on this? I imagine there are some better ways we can handle this in the UI. With input controls we have some options that give the functionality Nicolas is looking for. Maybe we can borrow some ideas from there

On Mon, Oct 29, 2018, 3:36 AM Nicolas Ruflin notifications@github.com wrote:

@tsg https://github.com/tsg @monicasarbu https://github.com/monicasarbu @tbragin https://github.com/tbragin @alexfrancoeur https://github.com/alexfrancoeur FYI

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/elastic/kibana/issues/24709#issuecomment-433813662, or mute the thread https://github.com/notifications/unsubscribe-auth/AYA-f3k5A7sbtqKuBZm9VhxluPhEeLhRks5upq_ygaJpZM4X-nNr .

[stacey-gammon]

Im assuming it's matching on the be inside kubernetes. what happens if you write bea?

[stacey-gammon]

actually nm that doesn't account for all the fields that are showing up.

@Bargs - this is a bug, not expected, right?

[Bargs]

As far as I can tell all of the fields showing have a "be" in them. I believe @ruflin is referring to the fact that fields are being suggested even if those fields don't exist in any of the documents within the given time range.

This is working as intended. We've had this debate many times before actually, dating back to when suggestions were first added to the filter editor. The counter argument is that just because a field has no results does not mean you don't want to query on it. The fact that it has no results may be surprising to you and a useful piece of information to learn. It's easier to convey that information with a 0 results screen than it is to mysteriously remove it from the suggestions, which the user may just interpret as kibana being broken.

[stacey-gammon]

Oh good call, numBEr, memBEr. missed those.

[ruflin]

I don't think it has be one or the other. If for example I start filtering on fields and don't see the field I'm looking for, I could click on "more ..." or "all ..", perhaps make it a config option.

If we have 20 fields and I only see 18, agree this could be confusing. But in Beats we are talking about 2000 fields and most fields are not used because the module is not enabled. Even if it's only a config option, from a Beats perspective this would be a big win.

[stacey-gammon]

in Beats we are talking about 2000 fields and most fields are not used because the module is not enabled.

Another way to tackle this is to be able to only import/update fields in the index mapping from modules that are turned on, which has other benefits, like those fields not showing up when creating visualizations. I think this is something we want to do right? I can't remember if I had this conversation in Dublin or somewhere else. Do you know @ruflin, if there is an issue filed for this?

[Bargs]

I'm not fundamentally opposed to a flag for turning it on and off, but we've talked about it before and failed to come up with a good way to present it to the user in a clear manner. We could put the flag at different levels (global, per index pattern, per user, per search) but I think ultimately each individual user needs to be able to control it for themselves otherwise behavior could be confusing and unexpected. So we would need a design that makes it obvious to the user which mode they're currently in, what that mode means for the suggestions they're currently seeing, and how to change it if they want.

Another possible point of difficulty is that some users have expressed a desire to have the filters in the filter bar apply to the suggestion results as well. @ruflin would you want that as well, or do you only want the time filter to apply? If only the time filter, then we're looking at trying to squeeze in three different types of suggestions.

[ruflin]

@Stacey-Gammon This is something we are actively looking into to split up modules per indices. It could result in multiple index patterns in Kibana but this causes other issues. For example often a user wants to see data from the system module and the mysql module (as an example). There is no public issue for this yet as we are still figuring out the right approach on our end.

@Bargs I would also expect the filtering on the filtered results. Let me make an example which I hope also explains what an index pattern per module is not going to solve it either fully.

The user wants to see all data which comes from the system module for the cpu metricset. The filter he will apply is metricset.module: system AND metricset.name: cpu. Now when he starts typing, he see all the fields that exist in this time frame and are sent in one of the cpu documents. He will not see the memory fields which are also in the system module because he filtered on for CPU. He will also not see ticks fields for example if he didn't enable these in the Metricbeat configs and are in the actual events. The suggested values should also only be the ones available for all the filters applied.

What about packing it under the Syntax options? It would also be nice to at least have it as a feature available on the Solution side, meaning if we prefer in the solution to use the above behaviour by default, we could do that and still use the query bar but instantiate it with a different flag.

[tbragin]

@AlonaNadler @rayafratkina @Stacey-Gammon @bargs Any chance the team could discuss this during the Kibana App planning week this week and let us know where something like this this fits with priorities? As @ruflin mentioned, having this way of using KQL in the InfraOps and Logs app could help quite a bit because of the way Beats populate index patterns today.

[Bargs]

The filter he will apply is metricset.module: system AND metricset.name: cpu. Now when he starts typing, he see all the fields that exist in this time frame and are sent in one of the cpu documents.

@ruflin in this example, is metricset.module: system AND metricset.name: cpu in the filter bar or the query bar? Or could it be either?

It would also be nice to at least have it as a feature available on the Solution side, meaning if we prefer in the solution to use the above behaviour by default, we could do that and still use the query bar but instantiate it with a different flag.

This is already doable thanks to work the APM team did. Pass in any ES query as the boolFilter param to the autocomplete provider and it'll filter the suggestion results. We could of course implement a flag to make this particular use case easier, just want to point out that it is possible today without any changes to the autocomplete code.

[ruflin]

I normally always use the query bar. In the logging UI for example that is the only one we have. I would not expect the behaviour to be different but I might miss something here.

@Bargs Very interesting about the provider. Seems like a lot of logic already exists.

[rayafratkina]

Quick follow up on the timing: @tbragin we discussed in the planning but unfortunately we are way overbooked at the moment. We are going to park it until next time and reconsider then.

[tbragin]

@makwarth FYI - this is the KQL auto-complete enhancement I mentioned to you in our chat last week.

[science695]

Don't mean to bump an old issue, but is this something where, instead of hiding empty fields, prioritizing fields with data above those without?

Like showing all results, but those with active data show above a line / divider, and the others show below.

[wavded]

In my case it seems even if I "exclude" fields in the index, I still see fields excluded as well in autocomplete, even though they don't show up elsewhere in the UI. Is that expected?

[ella-strandberg]

It would be nice if the exclusion of fields in the index pattern was also applied to the autocomplete as well as the suggested fields for the filters.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesUx)

[ruflin]

Can someone comment on what the current behaviour around this is? I think a lot of changes have happened around Data Views / Index pattern since this was last discussed. Maybe @mattkime can chime in?

How does that also work in combination with integration packages? @kpollich I just installed the k8s integration but don't see the fields show up in the data view (which is good and bad). Maybe this partially explains https://github.com/elastic/kibana/issues/148361

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)