[Reporting] Full data table content must be contained in the pdf generated

[elasticmachine]

Original comment by @tbragin:

Right now only the preview of the data table shows up in the PDF report, because we essentially take a screenshot of the first page of the table. We should consider how to improve this.

Possibilities to discuss:

• Option to export data tables (not saved search tables) to CSV (to accompany the PDF)
• Option to export a full data table as a formatted PDF table (limited to N results, IBM Qradar does max of 500)

This is similar to the problem we have with saved search export LINK REDACTED

[elasticmachine]

Original comment by @tbragin:

cc: @kobelb

[elasticmachine]

Original comment by @tbragin:

We discussed this live and decided on the following approach in the immediate future:

• Export a full data table as a formatted PDF table, limited to N results.
• 500 seems like a reasonable default for N.
• For now, do not offer the option of making N configurable.
• Put these data tables at the end of the PDF, after the charts.

Note, this is what Splunk does as well

Reason for not going with the other option:

• Adding an option to export CSV along with PDF adds the complication of dealing with multiple files (we can zip them up, of course)

[mbarretta]

also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.

[stacey-gammon]

We discussed this a bit in the Sharing team planning week. Here is some of the notes from that session:

Saved Search and Data Tables: treat them the same.

• Problems:
  • Panel is cut off so page 1 might not show all the rows on the first page.
  • Table might have 10 pages, but only first page. Even if panel shows all rows of the first page, users want to have all 10 pages of data.
• Possible Solutions
  • Show just page 1 table in the pdf, but bundle data table information as .csv
  • Show just page 1 table in the pdf but add tables at the bottom of the pdf, maybe as extra pages.
  • [NO] auto expand rows so all rows show in a screenshot.
  • We don’t want this, it could be way too much information
  • [Best option] Offer an option in report PDF:
  • [Yes/No]: Include zip of csvs, one for each table. Tablename.csv
  • Add dashboard level option, export raw data as csv. rawDataForVis1.csv

[tsullivan]

@mbarretta

also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.

For saved searches, there is a CSV export option in the "Share" menu when viewing the search in Discover.

There's a gap right now on being able to export CSV from a saved search when viewing it in the Dashboard context though, and we need to fix it.

Providing more convenient access to exporting CSV, for saved search and data table, is a general theme where we could see a lot of wins.

1. Have a POST "Generation URL" to allow exporting data table (aggregated) visualization data to CSV
2. Be able to export CSV from a saved search when viewing it in the Dashboard context

[mbarretta]

@tsullivan I appreciate that the CSV export allows saved search records to be exported, but not exporting everything seen in a dashboard breaks the "promise".

I've seen many dashboards that are essentially text-only: the preference is for raw data over graphics so that they can see everything in one place and avoid the extra click required to go from a visualization to the underlying data. In those cases, the dashboard is really more like a few Discover views glued together.

So yes, users could export the various data, import them into something like Excel, and then export to PDF from there, but I think it's a fair expectation for them to assume if there are 20 rows of records on a dashboard, they will be visible when that dashboard is exported.

[Eniqmatic]

+1 is there any more progress on this or when we will likely see it? We have a dashboard that is data tables with split tables based on how many security attacks have occured. But our PDF reports only show 1 page. Is there any temporary workarounds for this?

[cmagdalena-racowireless]

+1

[tsullivan]

@Eniqmatic The App team is actively working on doing a lot of heavy lifting around how the visualizations get their data, and the fruit of that effort will be that we can get data from a saved visualization on the server-side. That will lead into options on how to implement both of the items from the description:

• Option to export data tables (not saved search tables) to CSV (to accompany the PDF)
• Option to export a full data table as a formatted PDF table

There is a workaround to export CSV from an aggregated table visualization. It's from the UI, so unfortunately it would be hard to automate: you can click the Inspector for the visualization to see the underlying data. On that screen are export controls. See: https://www.elastic.co/guide/en/kibana/6.6/vis-inspector.html

[Eniqmatic]

@tsullivan Excellent sounds like what I am looking for. Hopefully the guys make good progress with that! Thanks for your response.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[mikeh688]

Enterprise users of Elastic need to be able to share pdf reports. The current limitation of only generating a pdf image for the first page of tabular data massively undermines the value of Elastic and KIbana in the eyes of these customers. It's fantastic that Kibana will do all the really funky new stuff (Lens etc) but, ultimately, if it cant deliver the basics, enterprise users wont adopt it because it doesnt meet their needs (mainstream adopters vs early adopters).

[fbaligand]

I fully agree with @mikeh688 Then, I have several users that ask me to have alerting with a Kibana table visualization as CSV attachment. And today, I can't address this need, that is very useful to be able to open data in Excel, and even, simply copy the text of the data table. Formatted data is necessary for me. Particularly to use number format locale. Microsoft Excel expects number format locale from current country. This is important for decimal numbers to us the good decimal separator. Same thing for dates.

[reighnman]

Same as above comments..

I ran into this issue with table data on a dashboard. I then tried just a pdf report at the table visualization, same issue.

I then saved the data table filters as a saved search to try a standalone csv report, but cannot do things like GROUP BY/aggregation/etc at the discovery layer that are necessary for this data.

Overall reporting still feels like a third-party addon or external cog, having to setup things like email configuration directly in the config. I would hope things are going in the same direction as alerting is going, with use of "connectors" managed internally and available to use with scheduled reports as well (in addition to fixing the issues with PDF generation on data tables).

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[tsullivan]

With SearchSource from @elastic/kibana-app-services available on the server side, it is now possible to re-create the query and tabify the search results using the reporting framework. The formatting of the data can easily(?) be the same in a report as it is seen in the browser. Once Reporting is able to access the raw table data, it could support CSV or PDF exports for the table data.

I think that is what has been needed all along. It has taken a long time to get here because it depended on having SearchSource available on the server-side.

[mikeh688]

SearchSource sounds like something internal. So, your post sounds like great news and what does it mean in terms of something being available in release form that enables paged tables to be distributed in pdf reports? thanks Mike

[tsullivan]

@mikeh688 Hi, thanks for asking!

Feature:Reporting is owned by a larger team of devs that also maintain "SearchSource" which is a configuration scheme for searches that is becoming more and more ubiquitous in Kibana. The larger pool of resources and expertise means this can be worked on in the short term.

I have added this issue to a project board for App Services, and it will be discussed in a planning session so that an owner can take it on.

[mikeh688]

Tim - thanks and great news that this is heading towards resolution.

[fbaligand]

As you say @tsullivan, most of my Kibana clients ask to be able to have a Data Table content sent by mail every day, in CSV format, as an attachment.

[tornis]

+1

[tsullivan]

This can be started after https://github.com/elastic/kibana/pull/88303

[JamesB797]

It looks like this can be started now, as #88303 has been merged.

Being able to email daily reports from a lens visualization via watcher and a POST URL would solve an immediate business need for my company.

[mikeh688]

Hi guys - great to see this moving forwards (and thanks). Is there any idea of a release ETA? thanks

[tsullivan]

Looks like there is a greater need for exporting CSV from table-based visualizations in an automation-friendly way. Therefore, we can close this issue in favor of https://github.com/elastic/kibana/issues/30982

[mikeh688]

Can i challenge this being closed please? Customers who create dashboards and want to share (for example, management reports), require data table content to be contained in the pdf generated. The issue at the moment is that only the first page of the table is contained in the pdf output and this is very limiting.

[tsullivan]

Hi @mikeh688 I agree we still need an issue to provide what you wrote. This issue was a little vague because the description didn't put it as succinctly. I am going to re-open and edit the description to focus on "Customers who create dashboards... require data table content to be contained in the pdf generated"

The other issue is for allowing full CSV export from data table visualizations in an automation-friendly way, which solves the "full export" problem, but it isn't in PDF context.

[mikeh688]

thanks

[petrklapka]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.

[ThomasEscobar]

Hi @petrklapka, I stand with @mikeh688 in this case and would also oppose this being closed.

This is definitely still a requirement for us, and something we would love to see coming in the future. People are always interested in the dashboard export, and then think "ah..." when we tell them the limitations, especially for the data in their tables.

[fbaligand]

I’m also very interested by this feature!

[mikeh688]

Q: when is a reporting tool not a reporting tool? A: when you cant share the data

Its quite embarrassing promoting KIbana as a reporting tool with lots of great features until a customer asks "can we share pdfs of the reports" and we have to answer "yes, but only the first page of any tabular data". It undermines the great features.

Please reopen the ticket

[mbarretta]

This is still an unmet requirement, per the last reopening. Reopening again.

[petrklapka]

Thanks for the feedback all! This is unfortunately the only way to dig out of our cruft of almost 900 issues. @mbarretta looks like you reopened it for me, so thanks. The squeaky wheels will get the kicks. Only a few hundred more to go!

[mikeh688]

no problem and many thanks

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[tsullivan]

I chatted with @vadimkibana a bit about this today. Ultimately, the dashboard application will need to render itself to support a feature where tables are fully visible when a screenshot is taken, or when we get the PDF from Chromium. There needs to be two enhancements:

1. In "preserve layout" mode (default for generating a PDF in dashboard), we can include additional pages at the end of the PDF for each table panel on the dashboard. Multiple pages would result in the PDF by having the Dashboard app provide multiple "locator params" - one for each page in the PDF. The concept of using multiple locator params in a report job is already used in Kibana for the Canvas application.
2. In "print layout" mode, the custom CSS for screenshot layout, which is under the Dashboard domain, should have a selector for panels with tables inside of them, and ensure the height of those panels is not constrained and the panel doesn't show a scrollbar.

This outline is just to give context on why Vadim and I felt we need to tag the Presentation team on this issue

[ThomThomson]

@tsullivan I'm not sure it's feasible for us to build functionality into the "preserve layout" mode that appends pages onto the report for the contents of each table. When printing a dashboard, that mode is really more of a WYSIWYG sort of experience, where the layout of the dashboard grid is reproduced. I'm not convinced that we should introduce print specific changes to it.

The idea of ensuring the entirety of each panel is shown with no scroll bars in print mode is appealing, and something we could attempt.

[scottf9]

Highly highly request this feature! So many times I get requests to schedule/build pdf reports and this severely limits what we can do.

[mrin9san]

Hi, we are platinum customers of Elastic and request this feature. Its not practical to simply open up elastic account in presentations and show reports. Anyways a report with screenshot of the first page is a useless feature.

[gord-lyr]

+1 on this enhancement. Moving from a competing SIEM product, and can't provide the same level of reporting to our customers.

[petrklapka]

Looping @osmanis, @kevinsweet and @rayafratkina - This feature ask has percolated up several times now here.

[mikeh688]

we'd still like to see this feature; the requirement hasn't gone away (see earlier responses).

[tsullivan]

The feature added in https://github.com/elastic/kibana/pull/174511 (8.13) makes it possible to create a CSV report using data from an ES|QL query. It doesn't entirely satisfy these requirements, but it does for the first time make it possible to get a report of an aggregated search.

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[tsullivan]

@elastic/kibana-presentation Hi, this issue pertains to the layout engine of the dashboard, so wouldn't really fall under the SharedUX team, which maintains the service of reporting. What I'd like to do next is add this to a new meta issue that collects other issues involving reporting where dashboard layout is factor of making the feature work better.

Update: created https://github.com/elastic/kibana/issues/191532

[ThomThomson]

This is a really interesting one. It's definitely the layout of Dashboard, but it's also the pagination of the table, etc. For instance, even if we limit to 500 rows - how do we do this? Are we calculating the exact pixel size of each row, rendering all rows in one page and spitting out a massive PDF of the Dashboard?

To be honest, I think this issue could be closed. Firing off a CSV export is likely a much better way to get all of the data from a table.

[tsullivan]

To be honest, I think this issue could be closed. Firing off a CSV export is likely a much better way to get all of the data from a table.

I agree with you, @ThomThomson. There is a ton of complexity under this and many open questions about how this could be implemented. If we close this we could instead focus on https://github.com/elastic/kibana/issues/30982, which about capturing aggregated table data (not a saved search, which is currently supported) in a CSV report.