[ES|QL] Prioritize fields based on ecs information on the editor

[stratoula]

Describe the feature:

Right now, we are suggesting fields based on the alphabetical order. In case of ECS we want to suggest first the fields that are defined in the schema.

I think the logic of retrieving this ECS schema logic should live outside the editor, possibly on a package.

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)

[drewdaemon]

I think the logic of retrieving this ECS schema logic should live outside the editor, possibly on a package.

@kertal this need dovetails nicely with your ECS field-descriptions PoC

[drewdaemon]

Right now, ECS info is available as an in-memory browser bundle. However, the @elastic/kibana-data-discovery team has discussed making a server-side API for retrieving ECS data in more efficient, specific way.

[stratoula]

yes I would love to use this here

[kertal]

Yes, this is something we should discuss, so what I did was doing a POC for a browser side solution (problem: async bundle size additions, and a push back on making it a shared dependency) https://github.com/elastic/kibana/pull/180230 And a follow up to figure out, what happens when moving it to server side, no bundle size issues (but also not mergable) https://github.com/elastic/kibana/pull/180543

[kertal]

FYI, another intiative related to this topic: https://github.com/elastic/observability-dev/issues/3331

[ryankeairns]

UX consideration

Not certain how 'expensive' this would be, but ECS field descriptions could be useful. An example for the host.name field:

Place that description here in the 'Read more' panel, like so:

[stratoula]

Yes this is not a bad idea, I guess we should take advantage of the ECS fields information as much as possible

[tonyghiani]

As @kertal linked on the related issue, the exposed services for client/server consumption of ECS and Integrations fields might be useful here, as it's also possible to improve the service to programmatically get the fields metadata for specific integration datasets once they are selected in the FROM clause.

Happy to chat more about this if there are any questions 👌

[stratoula]

We can use the almost ready to land fields_metadata api to retrieve the information we want https://github.com/elastic/kibana/pull/183806

[tonyghiani]

We can use the almost ready to land fields_metadata api to retrieve the information we want #183806

I am happy to share that the PR is now merged and the fields metadata service is available for development, in the README.md for the service there are initial guidelines for its usage and you can see a full demo here:

🎥 Fields Metadata service full demo

[stratoula]

Thanx Marco, most possibly we will deal with it in 8.16. We will ping you if we have more questions but great job 👏