search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.81k stars 8.2k forks source link

[Reporting] Schedule Reports with Task Manager #53900

Closed tsullivan closed 3 years ago

tsullivan commented 4 years ago

Stakeholder teams in bold

  1. Remove ESQueue from the Reporting libs [Reporting]
  2. Create TM tasks for the various kinds of scheduled reports, (starting with Canvas PDF) [Reporting]
  3. The Reporting screenshot observable needs to be in scope of the task, have an easy interface to run a job, and get run when the task fires [Reporting]
  4. Add UI controls to the visualization container to schedule regular reports [App Teams]

Making the screenshot observable more visible should have side benefits of improving testability.

elasticmachine commented 4 years ago

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

kobelb commented 4 years ago

@elastic/kibana-alerting-services Does the task manager have any type of "per task type" concurrency control? We've been focusing on making task-manager capable of being more concurrent to support Alerting's needs, but if we spawn multiple Reporting jobs per instance of Kibana, we're likely to run out of memory.

mikecote commented 4 years ago

@kobelb Not at this time. There was a way around this with numWorker until it got removed in https://github.com/elastic/kibana/pull/45504. It was possible to set numWorker on a task definition to equal the number of max workers and would ensure the task only runs while nothing else is running.

We have an issue for alerting / task manager to properly limit concurrency (https://github.com/elastic/kibana/issues/54916) but no efforts started yet.

tsullivan commented 4 years ago

In ESQueue currently, we avoid claiming more than 1 job at a time, and I think we hold off on polling for new jobs while a current job is executing. We might need to find a way to retain that kind of behavior by handling for it in our task run function.

The plan is for Reporting to register only 1 task for all of its reporting needs.

mikecote commented 4 years ago

I believe #54916 is the solution you're looking for. It would allow alerts to still run concurrently while a reporting task definition can limit to only one reporting task run at a time per Kibana instance.

Is such feature what you're looking for? If so, what release is scheduled reports targeting?

alexfrancoeur commented 4 years ago

I don't want to speak for @tsullivan and @joelgriffith but I was hoping we'd be able to deliver scheduled reports (on task manager, and simplified UI creation, management UI's) in 7.x 😄 . We haven't fully defined MVP yet though.

elasticmachine commented 3 years ago

Pinging @elastic/kibana-app-services (Team:AppServices)

tsullivan commented 3 years ago

Closed via https://github.com/elastic/kibana/pull/64853