[Discover][ES|QL] Improve changing of index pattern performance

[kertal]

When using ES|QL, there's a different index pattern entered, behind the scenes there's a request for all fields started. This request needs to be completed for the actual data fetching to be started. This step is necessary to get information if there's a @timestamp field available and of type date. Here's the code:

https://github.com/elastic/kibana/blob/5d6812966aac7dd9893a1393112ba027dfa4f32e/src/plugins/discover/public/application/main/utils/get_data_view_by_text_based_query_lang.ts#L35-L42

What can be improved here is to just request what's needed, which is the information of the availability of the field. When DataViewLazy is available (https://github.com/elastic/kibana/issues/167750), this can be improved to just fetch the one field of interest. This can significantly boost performance, when the field list takes a long time to load.

Testing kibana_sample_data_logs in DevTools

• the request for the single field took 0.4s instead of 0.4-0.6s
• It mostly beneficial that less payload has to be transfered

Testing a large logs index pattern in DevTools

• the request for the single field took 0.3-0.7 instead of 1.5 - 2.5s

Testing a large CCS metrics index pattern in DevTools:

• the request for the single field took 3-4s instead of 15s-21s
• excluding frozen tiers on top 1-3s So the time to result would be 14-17s faster

Testing a large CCS logs index pattern in DevTools:

• the request for the single field took 0.7 - 1.1s instead of 4s-9s
• excluding frozen tiers on top and the request took just 0.5-0.6s So the time to result would be 3.5-8.4s faster

What's more, depending on the implementation of the cache the field cache https://github.com/elastic/kibana/issues/169622 we could also use the fields cache the get the information, if the index pattern fields have already been requested, before sending an request to ES.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[mattkime]

@kertal I think this need some more detail before its ready to be worked on. Its easy to get a single field but this call is being made in the context of code that expects the full field list to be available. So while I could load a single field its possible that other code would quickly load the whole field list anyway. I see two ways of thinking about this - one in terms of state loading - can we be certain that using DataViewLazy in a certain code path would delay loading the field list until after some other event? Or do we want to introduce DataViewLazy to various layers of discover until its widespread enough that it hits the field list loading bottlenecks?

[kertal]

Thx for the research @mattkime. It looks like this is tightly coupled to a general switch to DataGridLazy consumption in Discover, which is a bigger task. During the research in #179120 mentioned, we can improvement but showing a loading indicator when an ES|QL query takes care of the data view initialization before the query is sent to ES I'm closing this in favor of #179703