Nested field support in visualizations

[waynehamadi]

Is your feature request related to a problem? Please describe.

As you know, nested support is not available in OpenSearch Dashboards Visualizations. The community craves for it because not everyone is working with unstructured data.

A lot of users work with a relational database as a source of truth they connect to Opensearch. So the data is relational by nature, and unless you can easily denormalize your index, nested objects are the recommended way to model this relationship along with joins (in some cases joins are better to easily update a relationship, or when the number of children for a given parent is really high).

So there is a strong need for nested field support in OpenSearch Dashboard in order to visualize this relational data.

Describe the solution you'd like

Being able to build nested queries and nested aggregations in the visualization dashboard.

A clear and concise description of what you want to happen.

We might want to learn from others that have tried to build the feature in the past, like ppadovani, filirom1 or SpaceManiac. ppadovani suggested a design then implemented something like this as a plugin along with a language called  KNQL.

This is a huge feature, so we could do it step by step instead of looking for the perfect solution :

• make ppadovani's plugin compatible with openSearch dashboard. Then identify the elements that make it hard to integrate it to the visualizations and try to find a fix for it.
• get feedback. To get even more feedback and make the plugin easily usable by all users, we could add a small checkbox(really hidden) on the settings to allow for it to be displayed in the visualizations, without having to do the steps to install the plugin. But users would need to be aware that it can break their dashboards if they use it.

I know it's not a really clear trajectory, but I can't really say where the feedbacks of the community are going to lead us to. It looks like there are 3 options to tackle this problem. The interface can convert user interactions (clicks, drag and drop, typing) into : 1- an extended version of KQL. Currently, KQL is only used to filter data, and has no role in aggregating it 2- a separate language similar to KNQL. This language would be sent to the node web server which converts it back to a JSON sent to Elasticsearch. 3- json query directly, without using any language as a proxy.

Option 1 seems the ideal solution, because we don't create a separate language just for nested support. But the more pragmatic way of doing it might just be to have 2 different languages and THEN find a way to unify them into KQL.

Describe alternatives you've considered

I don't think there are alternatives to supporting nested fields natively on open search dashboard, but to cope waiting for it, vega visualizations allow to build pretty much anything, if you are willing to spend the time and effort.

Additional context This is the history of that feature request in Kibana : Request in 2014 : PR 1 by filirom1 : PR 2 by ppadovani PR 3 by ppadovani [Plugin] (https://ppadovani.github.io/knql_plugin/overview/) by ppadovani This request is considered in Kibana as high hanging fruit (I agree) and low impact (I disagree).

I will leave you with this screenshot which summarizes what you can get with the plugin right now :

[joaopfcruz]

+1 here.

Any updates on this?

[RawanS]

Same here, please let us know of any updates on this or if there is some workaround. Thanks.

[suraj-mathew]

It has been 2.5 years that everyone is asking for a much needed enhancement. But sadly no direct answer so far. It will be good if any of the following replies can be made

• Yes, it will be implemented
• No, it will not be implemented
• Still working on for a solution

so that people will get a notification whether to wait or go-ahead with some other solution (if any).

[ahopp]

I think it's safe to say that @merwanehamadi is no longer working on this issue - we can all see there hasn't been much activity on it recently and there are quite a few unresolved inquires about its current status. It's also flagged as 'help wanted' which typically means there are no current assignees working on the dev side. In the meantime, we might want to mark it as part of issues tagged 'backlog'? Thoughts - @joshuarrrr @ananzh @kavilla @ashwin-pc

[ahopp]

Also, if anyone is interested in helping out with this issue or want to contribute to the resolution of this issue, please feel free. Everyone in the community is welcome to make contributions. And I don't think @seanneumann is the right assignee - I don't thinks he's driving from the dev effort on the AWS side.

[nkalpakis21]

i see this has been open for awhile. is there trouble getting a dev to fix this issue? what is blocking this for so long