[Reporting] Scheduled Reports

[joelgriffith]

The Feature

We want to give our users the ability to get Reports from Kibana on a regular interval. In maturity this will encompass CSVs, PDFs and more of our user's content in Kibana. For initial release we'll likely scope this down to just PDFs

Some core use-cases include:

• Users can schedule a report, and generate/send from external sources (email, slack, etc.).
• Users can view historical information about these reports, including the ability to download them again from Kibana.
• Users can update and delete these reports.
• Users can get these reports on a daily/weekly/monthly interval.

In order to achieve this, we'll be leveraging TaskManger internally to take care of the scheduling and running of these jobs. Our plugin will handle the user and licensing permissions checks, loading pages, gathering data, and all the connector/saving logic of the binaries into ES.

What we'll do to achieve this

There's some core modifications that'll need to be done in our reporting plugin in order to achieve this. Our current stack is sufficient for most of the work needed for report scheduling, but there's a few things that we'll need to alter in order to achieve this:

• Removal of ES-Queue in favor of TaskManager. Our current implementation doesn't have the ability to schedule (in the future) reports, or at regular intervals.
• Remove our static reporting-user config check in favor of API Keys privileges checks. Since authorization token's aren't long lived, but API keys are, we'll need users to get an API key for the purposes of scheduled reports.
• Since we anticipate an onslaught of users looking to use this new feature, a Diagnostic tool of some kind would be helpful in understanding what's going on in their stack(s).

Why we're doing this

There are a number of table stakes features users of enterprise software products expect. Granular security and alerting capabilities to name a few, are ones that Elastic currently support and continue to evolve today. Reporting also falls under one of these features. While this functionality is being used, it is not easily configured. Because of this, our users are significantly under utilizing one of the more important features the Elastic Stack offers today. Whether you are using the stack for Observability, Security, Enterprise Search or one of the seemingly infinite use cases we support - reporting is an expected feature and one that truly makes our products useful.

Once we have a more robust reporting system in place, we can unlock more powerful workflows within Kibana. While it's not something we've committed to at this time, being able to have an alert trigger a report is something that we can envision happening in the future.

Open Questions

• Concurrency of reports per Kibana, and how do we limit these with TM?
• Should failed reports continue to retry? At what point do they "give up" or do they?
• Some care should be considered with regards to sensitive information potentially being blasted to a wider audience on accident.

Diagrams and more to come

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[eeijlar]

Can the following use case be considered:

• I want to run a performance test in an empty Kubernetes cluster. I deploy an application, and run the performance test. During the test, the relevant performance metrics from the application are sent to an ELK instance running in the Kubernetes cluster. At the end of the test I want to generate a report (pdf or png) from an imported Kibana dashboard, or download all data from ELK as CSV. I want to be able to do this from a shell script so that it can be automated. At the end of the test, the application and the ELK cluster are torn down. I don't want to have to visit the Kibana UI at any point during this cycle.

The Post URL approach which is used in the current Kibana implementation does not cover this use case, as the URL is not portable across repeated installs of ELK.

[tsullivan]

@eeijlar this sounds like more of an Alerting use case. We do want to enable reporting as an Action of an Alert. Could your use case look like something this:

1. Import the dashboard into the stack
2. Create an alert in the stack that triggers when testing is done.
3. Run the performance test and send the metrics to the Elastic stack.
4. When tests are done, the alert fires (maybe based on some flag in the data).
5. The alert has an action which creates the report of your dashboard, and sends it to you.

I think that is a cleaner solution, as you wouldn't need your own scripting to figure out a reporting POST URL.

[tsullivan]

Edit: deleted this quote, since the detailed design for starting work is in an RFC: https://github.com/elastic/kibana/issues/114167