In 8.5-8.7, the Logstash team introduced and then expanded Flow Metrics in the Logstash node stats API, which means they are already available as part of capture from this tool. Flow metrics give us current, recent, and lifetime flow rates for many meaningful metrics, allowing us to compare a pipeline or process's current behavior to itself over time.
Currently, the support diagnostics Analyzer does not yet incorporate Flow Metrics and still requires two-spaced captures to provide meaningful rate-of-flow insights. Additionally, the as-presented rate-of-flow metrics in the analyzer tool are not standardized or cohesive (mixing units or mixing wall-clock vs CPU clock)
We want to create a new diagnostic template, embedded in this support-diagnostics tool that includes process-, pipeline-, and plugin-level Flow Metrics so that it can provide guidance and insights from a single capture.
These changes should only affect the Logstash remote/local reports
Added the Apache freemarker dependency. It might look like an overkill solution for the given template replacements, but the plan is to also migrate a few existing reports generated by other tools, which will leverage the template system (more info about the project can be found here: https://github.com/elastic/logstash/issues/15144).
Added a new resource/config folder named logstash-diagnostic-templates, which initially contains a single file flow_metrics.html.ftlh. The new GenerateLogstashDiagnostics command reads all the template files (.ftlh) inside that folder and writes them to the output artifact naming the output files as logstash_diagnostic_<template-name> (removing the .ftlh extension)
The flow_metrics.html.ftlh is a single-page application that shows the pipeline and plugin flow metrics.
Checklist
[x] I have verified that the APIs in this pull request do not return sensitive data
Description
In 8.5-8.7, the Logstash team introduced and then expanded Flow Metrics in the Logstash node stats API, which means they are already available as part of capture from this tool. Flow metrics give us current, recent, and lifetime flow rates for many meaningful metrics, allowing us to compare a pipeline or process's current behavior to itself over time.
Currently, the support diagnostics Analyzer does not yet incorporate Flow Metrics and still requires two-spaced captures to provide meaningful rate-of-flow insights. Additionally, the as-presented rate-of-flow metrics in the analyzer tool are not standardized or cohesive (mixing units or mixing wall-clock vs CPU clock)
We want to create a new diagnostic template, embedded in this support-diagnostics tool that includes process-, pipeline-, and plugin-level Flow Metrics so that it can provide guidance and insights from a single capture.
More details can be found on the meta issue: https://github.com/elastic/logstash/issues/15145
Changes
These changes should only affect the Logstash remote/local reports
freemarkerdependency. It might look like an overkill solution for the given template replacements, but the plan is to also migrate a few existing reports generated by other tools, which will leverage the template system (more info about the project can be found here: https://github.com/elastic/logstash/issues/15144).logstash-diagnostic-templates, which initially contains a single fileflow_metrics.html.ftlh. The newGenerateLogstashDiagnosticscommand reads all the template files (.ftlh) inside that folder and writes them to the output artifact naming the output files aslogstash_diagnostic_<template-name>(removing the.ftlhextension)flow_metrics.html.ftlhis a single-page application that shows the pipeline and plugin flow metrics.Checklist
Closes https://github.com/elastic/logstash/issues/15145