[8.15](backport #41570) Metricbeat: add configurable failure threshold before reporting streams as degraded

mergify[bot] commented 1 day ago

Proposed commit message

Add configurable failure threshold before reporting streams as degraded

With this change it is possible to configure a threshold for the number of consecutive errors that may happen while fetching metrics for a given stream before the stream gets marked as DEGRADED. To configure such threshold, add a "failure_threshold": <n> to a module configuration block. Depending on the value of <n> the threshold will be configured in different ways:

n == 0: status reporting for the stream has been disabled, the stream will never become DEGRADED no matter how many errors are encountered while fetching metrics
n==1 or failure_threshold not specified: backward compatible behavior, the stream will become DEGRADED at the first error encountered
n > 1: stream will become DEGRADED after at least n consecutive errors have been encountered

When a fetch operation completes without errors the consecutive errors counter is reset and the stream is set to HEALTHY.

Checklist

[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
~~[ ] I have made corresponding change to the default configuration files~~
[x] I have added tests that prove my fix is effective or that my feature works
~~[ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

Disruptive User Impact

No disruptive user impact since not specifying the new configuration key maintains the previous behavior

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates https://github.com/elastic/elastic-agent/issues/5332

Use cases

Screenshots

Logs

This is an automatic backport of pull request #41570 done by Mergify.

mergify[bot] commented 1 day ago

Cherry-pick of f84c05b5774a83fbf3d8caa03758352ee2293f80 has failed:

On branch mergify/bp/8.15/pr-41570
Your branch is up to date with 'origin/8.15'.

You are currently cherry-picking commit f84c05b577.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
    new file:   metricbeat/mb/module/wrapper_internal_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
    both modified:   metricbeat/mb/module/wrapper.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

botelastic[bot] commented 1 day ago

This pull request doesn't have a Team:<team> label.

elastic / beats