New metric since logstash 8.13: worker_utilization

[VCCPlindsten]

Feature Description

There is a new metric on pipeline level since 8.13: https://www.elastic.co/guide/en/logstash/current/logstash-8-13-0.html#notable-8.13.0

Please collect and include it.

_node/stats sample (logstash 8.15.0)

``` "main": { "events": { "duration_in_millis": 1544425, "filtered": 4510626, "out": 4510626, "in": 4510877, "queue_push_duration_in_millis": 1269044 }, "flow": { "input_throughput": { "current": 1765, "last_1_minute": 2083, "last_5_minutes": 2111, "last_15_minutes": 1754, "lifetime": 1487 }, "worker_concurrency": { "current": 0.5364, "last_1_minute": 0.6848, "last_5_minutes": 0.6716, "last_15_minutes": 0.5716, "lifetime": 0.5093 }, "filter_throughput": { "current": 1765, "last_1_minute": 2079, "last_5_minutes": 2111, "last_15_minutes": 1754, "lifetime": 1487 }, "output_throughput": { "current": 1765, "last_1_minute": 2079, "last_5_minutes": 2111, "last_15_minutes": 1754, "lifetime": 1487 }, "worker_utilization": { "current": 53.64, "last_1_minute": 68.48, "last_5_minutes": 67.16, "last_15_minutes": 57.16, "lifetime": 50.93 }, "queue_backpressure": { "current": 0.4336, "last_1_minute": 0.5846, "last_5_minutes": 0.5729, "last_15_minutes": 0.4807, "lifetime": 0.4185 } }, ```

Version of logstash-exporter this feature request applies to

None

Motivation and Goals

This is a useful metric to determine how saturated logstash is since it uses the pipeline-level configured worker count, 100% means its saturated, no matter how many workers are currently configured.

Alternatives Considered

Using a different way to collect logstash metrics - not a good option.

Additional Context

There doesn't seem to be another way to get at this number - because the amount of workers is a per-pipeline attribute, which doesn't seem to be exposed by logstash. logstash_info_pipeline_workers only collects the default number.

[kuskoman]

@VCCPlindsten are you using v1, or v2? I create a PR for v1: #369 but v2 is in rather unstable state on the branches that i am working on RN and i wonder whether i have to rebase them, or just fix that later on

[kuskoman]

also, i forgot to ask. could you do the review on the PR i just posted?if it is allright i can merge it soon and create a PR

[VCCPlindsten]

Looking a little bit closer at the _node api, it seems per-pipeline workers is exposed, under the "pipelines" key. Not sure if that is a new thing as well - I've not noticed it before.

Reviewing shortly.

[kuskoman]

i am currently working and should not be doing this task, but it seems like a small one :D the PR was created really quick, so there may be these mistakes in it

Are you using v1 version of the exporter? I will replicate the PR to v2 later on

[VCCPlindsten]

I am using v1, yes - since v2 is not stable yet.

[VCCPlindsten]

Working in release 1.8.2