[Reporting] Document steps of monitoring Reporting

[tsullivan]

Report jobs are queued in an Elasticsearch so they are executed one at a time. If there are other reports in the queue when a new report is requested, it will have to wait until the other jobs process.

What to monitor? What logs do we need?

• Monitor search latency from Elasticsearch in Stack Monitoring
  • If ES search is slow, reports will take longer to generate as well
• Check the "queue time" of the report in the report job details in Stack Management > Reporting.
  • The reports may have had to wait a long time in the queue, if there are many reports going on.
  • This can be solved by adding more Kibana instances to the deployment, to load-balance the report queue.

Monitoring Reporting:

1. Use Metricbeat to view the memory consumption of all the processes on the server.

2. Enable verbose logs for Reporting to understand the steps that happen from requesting a report and completing the report, and how much time each of those steps takes:

   logging:
    loggers:
       - name: plugins.reporting
         appenders: [console]
         level: debug
       - name: plugins.screenshotting
         appenders: [console]
         level: debug

3. Using Filebeat, you can ingest the logs from above and find ECS-compliant logs with data about the Reporting queue and Screenshotting performance.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[tsullivan]

This should depend on https://github.com/elastic/kibana/issues/29281. The documentation should give examples of bad performance vs good performance. The examples will be illustrated with results that the user can find in the new Monitoring of Reporting area

[tsullivan]

Closing this, no one is asking for it

[tsullivan]

Reopening to help with https://github.com/elastic/kibana/issues/147914

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[tsullivan]

I have rewritten this issue to make it less broad, and focus on how to monitor reporting.

Previous text was:

Report jobs are queued in an Elasticsearch so they are executed one at a time. If there are other reports in the queue when a new report is requested, it will have to wait until the other jobs process.

What can we do to improve report generation time?

• Make sure you do not have the xpack.screenshotting.hostName setting, and if you have a "basePath," make sure you have server.rewriteBasePath: true
  • This avoids connecting the headless browser to the Kibana webserver via a proxy
• If you have NODE_OPTIONS set for Kibana, make sure you aren't allocating 100% of the system memory to Kibana
  • Chromium is a child process of Kibana, and uses its own memory space. There should be at least 1GB of system memory available to Chromium when it needs it.
• If possible, limit the dimensions of the Canvas workpad or Dashboard. Smaller PDF/PNG reports will generate faster
• Along with making sure there is enough system RAM, make sure the system has a decent CPU for Chromium.
  • Kibana only needs a single CPU core for its main thread, but Chromium is able to run multiple threads for rendering.

What to monitor? What logs do we need?

• Monitor search latency from Elasticsearch in Stack Monitoring
  • If ES search is slow, reports will take longer to generate as well
• Check the "queue time" of the report in the report job details in Stack Management > Reporting.
  • The reports may have had to wait a long time in the queue, if there are many reports going on.
  • This can be solved by adding more Kibana instances to the deployment, to load-balance the report queue.

More advice:

1. Use Metricbeat to view the memory consumption of all the processes on the server.

2. Enable verbose logs for Reporting to understand the steps that happen from requesting a report and completing the report, and how much time each of those steps takes:

   logging:
    loggers:
       - name: plugins.reporting
         appenders: [console]
         level: debug
       - name: plugins.screenshotting
         appenders: [console]
         level: debug

3. Using Filebeat, you can ingest the logs from above and find ECS-compliant logs with data about the Reporting queue and Screenshotting performance.