[Feature] Show raw unaggregated data in Data Table

[pettax]

Is your feature request related to a problem? Please describe. It is not possible to display raw, unaggregated data in table form in a dashboard.

Describe the solution you'd like I suggest either extending the Data Table visualization to support showing data without aggregations or introducing a new visualization type that focuses only on raw data. All other features of the Data Table visualization should remain intact.

Describe alternatives you've considered Unaggregated data can be shown in the Discovery section. However, this view lets the user reveal all underlying data in the index (by showing the expanded document), which may not be desirable. The Data Table visualization, on the other hand, shows only the columns that you have configured, which is what I am after.

Another option considered is aggregating the data in such a way that it shows all unique rows. This approach is very resource-intensive with larger datasets though.

[ashwin-pc]

@pettax I think what you are referring to is a "Saved Search" and yes you can using the Discover app. You can navigate to that app and save a raw search that you want to add to the Dashboard and display it. As for limiting the columns to view, that can also be done in the Discover app. Do this before you save the search and only those columns will be visible when the datatable is rendered on the Dashboard. Does this solve your problem or am i missing something?

[pettax]

@pettax I think what you are referring to is a "Saved Search" and yes you can using the Discover app. You can navigate to that app and save a raw search that you want to add to the Dashboard and display it. As for limiting the columns to view, that can also be done in the Discover app. Do this before you save the search and only those columns will be visible when the datatable is rendered on the Dashboard. Does this solve your problem or am i missing something?

Thank you for your reply, @ashwin-pc!

I do follow you with the "Saved Search" in the Discovery app. The step that I do not understand is how to use/visualize that Saved Search in a Data Table placed in a Dashboard. If I edit my dashboard and select New Visualization I get the screen below.

If I add the Data Table Visualization from here I get the UI for defining my aggregations. I cannot find a way to populate this table with a Saved Search showing raw data.

Is there another way to visualize a Saved Search in a Table in the Dashboard app that I am missing? Can you guide me?

[ashwin-pc]

Yes, this is not as intuitive as creating a visualization from the Dashboard, but what you need to do is:

1. go to disciver and create a new saved search
2. Go to the dashboard you want to add it to and click edit
3. Then click "Add" which opens a plyout with all your saved panels.
4. Find the saved search here and add it to the dashboard.

This is because technically Saved searches are not visualizations but Dashboards can contain things that arent strictly visualizations.

@kgcreative what do you think about adding a link from the create new visualization panel to Discover to create saved searches?

[kgcreative]

@ashwin-pc -- I really like this. I think from Discover, when you go to "Save search", we should include an action to also be able to add it to a dashboard of your choice:

We have a similar fow today that only surfaces when you duplicate a visualization when coming from a dashboard. We could extend and improve that.

[pettax]

Thanks again @ashwin-pc for engaging in this!

Then click "Add" which opens a plyout with all your saved panels.

OMG. I mixed saved queries with saved searches. I had saved queries but not saved searches. If I do have a saved search it is selectable as you say.

However, the added panel has all the drawbacks of the one in the Discovery app. I listed this under the heading Describe alternatives you've considered in my original description. It allows the end user to reveal all underlying data in the index (by showing the expanded document), which may not be desirable.

What I am after here is a visualization that looks the same as the Data Table and has that export functionality but contains unaggregated data just as the saved search does.

[ashwin-pc]

Hmm good point. I'm a little torn here because showing the raw unaggregated data isnt really a visualization and also seems like duplicative functionality. I can see two paths forward here:

1. Add functionality to Saved Searches to allow more customizations such as allowing for the rows to expand(and in Discover 2.0, open a doc view flyout)
2. Adding a new visualization type or modify the datatable visualization to allow the user to create tables with the raw unaggregated query.

I am leaning towards the first since its a much easier lift and does not duplicate the existing saved search feature. I'd love to hear your thoughts on which approach you think makes more sense since you have a need for this feature.

[pettax]

@ashwin-pc

I'm a little torn here because showing the raw unaggregated data isnt really a visualization

Why, in your opinion, does not unaggregated data classify as a visualization?

I'd love to hear your thoughts on which approach you think makes more sense

Knowing nothing about implementation details such as what is easier to achieve or could cause duplicated functionality I'd like to see option 2. Visually, option 2 is exactly what we need. I assume here that you are referring to using the Saved Query created in the Discovery app in the Data Table somehow. I just understood that there is a difference between a Saved Search and a Saved Query.

The use case we have is to prepare ready-to-run dashboards for our end users. In my head, the end users should use the Dashboard app. The discovery app seems to me more like a tool to explore the data to help produce the dashboards and not a place where end users (in our case) should go.

If option 1 would allow the panel to be customized in such a way that it mimics the Data Table then this ofc also an option.

[kgcreative]

@ashwin-pc + @pettax -- Reading this, I think there may be a misunderstanding of a saved search vs a saved query -- and perhaps there should be something we do here to help solve that.

A saved query might be a bit of a misnomer, and i've been considering whether it would make sense for us to update the terminology. Perhaps saved filters would make more sense. Specifically, a saved query saves simply the state of the search/filter bar, and can be used by Discover, Dashboard and some Visualize visualizations.

A saved search saves the state of your discover search, including selected columns. (for example, this),metadata:(indexPattern:d3d7af60-4c81-11e8-b3d7-01146121b73d,view:discover))&_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:now-24h,to:now))&_q=(filters:!(),query:(language:kuery,query:''))) saved search: is also used to populate the table at the bottom of this sample data dashboard&_a=(description:'Analyze%20mock%20flight%20data%20for%20OpenSearch-Air,%20Logstash%20Airways,%20OpenSearch%20Dashboards%20Airlines%20and%20BeatsWest',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(),gridData:(h:7,i:'1',w:32,x:0,y:0),id:aeb212e0-4c84-11e8-b3d7-01146121b73d,panelIndex:'1',type:visualization,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:12,i:'3',w:23,x:17,y:7),id:c8fc3d30-4c87-11e8-b3d7-01146121b73d,panelIndex:'3',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:15,i:'4',w:48,x:0,y:85),id:'571aaf70-4c88-11e8-b3d7-01146121b73d',panelIndex:'4',type:search,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:12,i:'5',w:17,x:0,y:7),id:'8f4d0c00-4c86-11e8-b3d7-01146121b73d',panelIndex:'5',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:14,i:'6',w:24,x:24,y:33),id:f8290060-4c88-11e8-b3d7-01146121b73d,panelIndex:'6',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:14,i:'7',w:24,x:24,y:19),id:bcb63b50-4c89-11e8-b3d7-01146121b73d,panelIndex:'7',type:visualization,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:12,i:'10',w:24,x:0,y:35),id:'9886b410-4c8b-11e8-b3d7-01146121b73d',panelIndex:'10',type:visualization,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:8,i:'13',w:14,x:10,y:19),id:'76e3c090-4c8c-11e8-b3d7-01146121b73d',panelIndex:'13',type:visualization,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:8,i:'14',w:14,x:10,y:27),id:'707665a0-4c8c-11e8-b3d7-01146121b73d',panelIndex:'14',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:8,i:'21',w:48,x:0,y:62),id:'293b5a30-4c8f-11e8-b3d7-01146121b73d',panelIndex:'21',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:7,i:'22',w:16,x:32,y:0),id:'129be430-4c93-11e8-b3d7-01146121b73d',panelIndex:'22',type:visualization,version:'2.10.0'),(embeddableConfig:(mapCenter:!(42.19556096274418,9.536742995308601e-7),mapZoom:1),gridData:(h:15,i:'23',w:48,x:0,y:70),id:'334084f0-52fd-11e8-a160-89cc2ad9e8e2',panelIndex:'23',type:visualization,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:8,i:'25',w:10,x:0,y:19),id:f8283bf0-52fd-11e8-a160-89cc2ad9e8e2,panelIndex:'25',type:visualization,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:8,i:'27',w:10,x:0,y:27),id:'08884800-52fe-11e8-a160-89cc2ad9e8e2',panelIndex:'27',type:visualization,version:'2.10.0'),(embeddableConfig:(vis:(legendOpen:!f)),gridData:(h:15,i:'28',w:24,x:0,y:47),id:e6944e50-52fe-11e8-a160-89cc2ad9e8e2,panelIndex:'28',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:6,i:'29',w:8,x:40,y:7),id:'01c413e0-5395-11e8-99bf-1ba7b1bdaa61',panelIndex:'29',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:6,i:'30',w:8,x:40,y:13),id:'2edf78b0-5395-11e8-99bf-1ba7b1bdaa61',panelIndex:'30',type:visualization,version:'2.10.0'),(embeddableConfig:(),gridData:(h:15,i:'31',w:24,x:24,y:47),id:ed78a660-53a0-11e8-acbd-0be0ad9d822b,panelIndex:'31',type:visualization,version:'2.10.0')),query:(language:kuery,query:''),timeRestore:!t,title:'%5BFlights%5D%20Global%20Flight%20Dashboard',viewMode:edit))

With that said, I do see the saved table embeddable does have the expandable row functionality from the old Discover. @ashwin-pc, let's add an item here to follow up on the embeddable to use the same flyout that discover uses for the "inspect" panel, and perhaps we should add a setting to the embeddable to allow users to supress the "document details" column? -- the other option would be to give users an easy way to "Create table from search" where we simply pre-populate the table component with the selected fields from the saved search

[ashwin-pc]

Yep, the latest version of Discover does use the flyout, that was just part of the migration. As for the ability to suppress document details, this should be possible technically, how do you suggest we add this control though?

[pslusarz]

I use Open Search dashboard for regression analysis on general datasets. This may not be typical use case these days, but was really popular and intuitive way to drill down to your data back in the early days of Kibana. Somehow, neither the Data Table, nor "saved search" captures this use case well when it comes to displaying detailed records.

• Saved search cannot aggregate (in the enclosed example, there may be many categories per review, so to only display one review, it needs to be aggregated)
• Saved search cannot rename columns
• Saved search exposes entire record (after expansion) to the user, which may be undesirable
• Saved search has no way to remove Time column
• Data Table forces you to have a some kind of aggregation statistic displayed, which is meaningless in my use case (I labeled it "Category count" below in the "review-detail" visualization below, but I'd rather not have it at all)
• Data Table has bugs where not all records matching the dashboards current search criteria are being displayed (see attached screenshot, where counts indicate there should be 50 detailed records available to view, but only 8 are being displayed in the detail view, in other cases, I have nothing showing up, at times).

Finally, this type of dashboard is really difficult to configure nowadays, where IMO this should be the default to facilitate the adoption. That's what initially sold me on Kibana, and I suspect many users would jump on it, if they only saw how intuitive this interface becomes once configured.

[ashwin-pc]

cc: @kgcreative @wbeckler

[wbeckler]

@pslusarz would you be willing to create a separate issue for "not all records matching the dashboards current search criteria are being displayed" ?

It might be related to bad UI around which filters apply to which visualizations in a dashboard, or might be an actual bug.

[wbeckler]

It sounds like table visualizations needs more features rather than an embeddable saved search.

[aver1cal]

@ashwin-pc @kgcreative Could this issue be solved by a plugin adding a new visualization type? Similar to DataTable visualization however without setting the required "metrics" and "buckets" aggregations, in this case it would not require any setup and simply populate a table with search results from visualization source like a Saved Search. Discover page table does just that and it would be great to be able to have this in dashboards. Does this seem like possible approach?

[ashwin-pc]

@aver1cal yes this could definitely be solved with a plugin but honestly, id rather we address each of these requests in the saved search itself. From what i read there are a few additional features that either a Saved Search or Table Vis needs to add that helps achieve what you want and that would be useful to other users as well. Wuld you be willing to contribute these features to OSD? I can help you bring this feature into Discover if you are interested. Let me know if that is something you're interested in.

[aver1cal]

@ashwin-pc I am not sure whether this would achieve the desired result? The table visualization would still require to set metrics and buckets aggregations unless we decided to remove this or make it optional. The goal is to have a table in dashboard that does not set any additional aggregations on top and just displays result from source search. I'd definitely be willing to contribute if this was the outcome.

[kgcreative]

We do have the option to display the results of a saved search directly in Dashboards, and I think what Ashwin is pointing out, is that we might be able to modify the preferences of how that gets embedded to better match your needs: