[ES|QL] [Monaco] Investigate if we can render a different UI component for the from command

[stratoula]

Describe the feature:

At the moment when we are suggesting indices we use the suggestions UI of the monaco api.

We want to make this component a bit more complicated. We don't want to suggest only indices but also integrations, datastreams etc.

We need somehow to make this distinction in the UI which means that the current UI of the monaco api needs to change.

Below is an idea:

The ideal scenario would be to be able to render our own UI component for the from command suggestions.

If this is not possible we could possibly define different icons to the different options and possibly different styling (?)

The purpose of this issue is to investigate the various options in order to make a decision forward.

Next steps:

• Fetch the integrations from the fleet api
• Integrate the api to the getSources api of the editor

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)

[drewdaemon]

I understand the need but not why Monaco's default doesn't cover it.

We can already label each option on the right and change the icon on the left...

[stratoula]

@drewdaemon yes, but what if we want a different more compolicated UI? This is the investigation we want to do. What capabilities do we have? Can I have something fancier?

[drewdaemon]

@stratoula sounds good. I just read

We don't want to suggest only indices but also integrations, datastreams etc.

and looked at the wireframe and I think the default can cover these cases.

Still, I see value in this investigation as a way to stake out the boundaries of the possible 👍

[stratoula]

One quite hacky PoC from Joe https://github.com/elastic/kibana/pull/181112

[dej611]

Another hack could be to combine the suggestion api with the commands API:

• show a list of options for a menu as autocomplete auggestions
• on click it will trigger a command with some parameters
• the command will render a custom overlay widget

[stratoula]

So I investigated the different options we have with the monaco editor:

1. Use the commands API and create a new option in the sources suggestions. This option can then open a modal or a flyout. This option is not hacky at all, we have used something similar in the enrich command. One demonstration below:

2. Use the commands api and create a new option in the sources suggestion. This options can open a new popover:

This is using on one hand the commands api to create naturally a new option in the autocomplete but unfortunately the popover should be created by hand. Also:

• we need to define how the keyboard is going to work by overwriting the monaco key events
• we need to define when the popover will be hidden
• it adds a lot of customization from our side
• on an integration selection we need to update the query which won't look as natural as it looks with the native suggestion popup

3. Do what Joe proposes above. This is not very different from the previous example implementation wise but it has an extra hack / drawback. We need to detect when the from command has been typed and open the popover.

Note: I also investigated the overlay widget idea from Marco but widgets are not what we want. They don't act as popovers, they are overlays inside the editor so I could not make them work as we want.

4. Another idea we had with @timductive is to have a special character or a combination of special chacters (which is not going to be used by ES|QL - might be tricky) that the user can type and this is going to display a menu:

• We are still using the commands api
• We need to calculate the position that the popover will appear
• It must be a combination of characters e.g. SHIFT + / to be certain that we are not interfering with user typing a query
• Can be quite useful to display with this way actions to the users to help them with the query authoring.

The 1 and 2 and 4 ideas can be seen here. They are not working prototypes, they are just the result of my research. The second approach will need additional work to make it work smoothly and bug free. The 4th is a bit cleaner and I personally like it as an idea. I also like 1 too. The 2nd and 3rd are very dirty and we need to interfere with monaco events so they are def more prune to bugs and need a lot of love.

[flash1293]

The options above are mostly exploring how to open the custom popover. This is part of the problem, the other part is how the users is interacting with the popover, which is probably also worth some thinking.

Some considerations:

• Option 1 means the user leaves the editor completely - we could auto-focus a search input or something in the modal and do a completely separate thing. If the user does something in the modal, the question is where to add something in the query editor, as there is no active cursor anymore. I guess as long as we just edit the "from" part and don't have joins it will be fairly simple, but it probably falls short of the "notion-like" experience. Or maybe we can store the old cursor position and put it back once the user is done in the modal
• Option 2-4 all "stay" in the editor, which leaves the option to allow the user to keep typing into the editor and react to it in the popover. I explored this a bit in my POC by suppressing the arrow up/down keys and update the state of the popover, while other commands still work as before. While possible, this is hard to manage properly, as monaco is still in the loop and its internal state can diverge with the custom popover state if we aren't careful.
• I really like that option 1 is so clean - we do not abuse any API and individual components don't fight each other. Also, we could build it in a way the user could continue controlling the whole thing with the keyboard.
• Option 4 is relatively clean, but as it won't show "on its own" (you need to use the special shortcut), IMHO it feels too much of a "separate" feature - also, I expect the interaction with the popover to be the hardest part, and that applies to option 2-4 equally

Another angle to the whole problem - can we enumerate all the features we want/need? As @drewdaemon is mentioning, you can do things like custom icons, maybe we can get by with that already?

[drewdaemon]

Great discussion and research going on here.

As @drewdaemon is mentioning, you can do things like custom icons, maybe we can get by with that already?

Unfortunately, I have to throw a bit of water on my own party. I took a second look, and I don't think Monaco facilitates custom icons. Instead, it chooses from a set a predefined icons based on the kind property (doc). Not sure if it could be hacked... (doesn't look like they're going to change this: https://github.com/microsoft/monaco-editor/issues/2654).

However, we can definitely write custom labels and descriptions for each entry which could associate datasets with integrations for example.

https://github.com/elastic/kibana/assets/315764/f8d43b2d-ddc0-484b-990b-ef500e269ede

The little documentation popover accepts markdown, so we can put a lot of formatted information in there.

[stratoula]

Thanx Drew, it is ok. It doesnt seem we are going that way so this limitation is not so important atm

[stratoula]

I am closing this as we investigated the things we wanted to work on with @ryankeairns and now we can move forward to the design and actual implementation. Here is the PR which all the experiments https://github.com/elastic/kibana/pull/183377