Dashboard for metrics

[tchiotludo]

following this one https://github.com/kestra-io/kestra/issues/1155, the next steps is to be able to have a complete dashboard that will gather multiple metric per flow or per namespace.

Some features asked by @yuri1969 on general dashboard that could help on the design:

• Execution success rate of the last 1h/15min
• Execution fail rate of the last 1h/15min
• Histogram of count of successful executions in the past 1h
• Histogram of count of failed executions in the past 1h
• Histogram of count of started executions in the last week (with possible trends)
• Table/pie of top 10 started executions in the past 1h containing label env:prod grouped by label x and flowId => Which PROD flows of a "type" specified by the given label x are most frequent?
• Table/pie of top 10 failing executions in the past 1h containing label env:prod grouped by label x and flowId => Which PROD flows of a "type" specified by the given label x are most troublesome?
• Table/pie of top 10 failing task types in the past 24h
• Table/pie of top 10 failing taskId in the past 24h
• Table/pie of top 10 tasks types with attempt count > 1 in the past 24h (to be possibly filtered by labels foo:bar or env:prod)
• Table/pie of top 10 tasksId with attempt count > 1 in the past 24h
• Histogram of count of task runs with attempt count > 1 in the past 24h (to be possibly filtered by labels foo:bar or env:prod)

Related https://github.com/kestra-io/kestra/issues/3421

[anna-geller]

Adding notes from today's discussion:

1. METRICS FROM PLUGINS - We want to start with a dashboard for metrics emitted by plugins (the real "Metrics" concept in kestra rather than metrics in the sense of e.g. "the count of successful executions in the past 1h")
2. The metrics visualization will be included in the Flow and Namespace Dashboard — the initial scope will be to focus ONLY on the dashboard displayed in the Overview tab on the Flow's page and later we can extend this to the Namespace and to the main Dashboard
3. For each metric displayed on the Flow's page, add a button "Add to my Dashboard" - this will prompt the user to give a title for the visualization. Then, the metric visualization will be displayed on the Flow's page. Later we may give user the option to choose to which Dashboard they want to add a specific metric viz — Flow, Namespace or Main Dashboard

The Flows page is the initial scope:

Scope for later:

Requirements:

• we start with Flow-level dashboard
• user should be able to define their dashboard as code. While it's natural to click through the UI to configure the aggregation type etc but in the end, this should generate YAML to keep the promise of everything as code and everything from the UI.
• Ideally, we should give the user the flexibility to configure where a given chart should be placed on the Flow Overview page—TBD, depending on how difficult this would be.

Use cases:

• track nr of rows processed in Query tasks
• track duration of some processes within script tasks
• track bytes processed for cost management e.g. in BigQuery