search

jenkinsci / datadog-plugin

A Jenkins plugin used to forward metrics, events, and service checks to an account at Datadog, automatically.
https://plugins.jenkins.io/datadog/
MIT License
30 stars 48 forks source link

jenkins.executor.free metric includes executors on hosts that are offline #434

Closed magnusbaeck closed 1 month ago

magnusbaeck commented 1 month ago

Describe the bug The executors of a host that's offline are still counted in the jenkins.executor.free metric, which is misleading since those executors aren't actually free. It's also inconsistent with how Jenkins itself counts executors in e.g. the Load Statistics page (/label/$LABEL/load-statistics).

The jenkins.executor.count metric has the same problem, but perhaps it's okay that it counts all executors. Then you'd also be able to compute the number of unavailable executors by subtracting jenkins.executor.free and jenkins.executor.in_use from jenkins.executor.count.

To Reproduce Steps to reproduce the behavior:

  1. Mark a Jenkins build node with at least one executor as temporarily offline.
  2. Visit /label/$LABEL/load-statistics for a label assigned to the node and take note that the Available executors data series drops.
  3. In the metrics explorer in Datadog, look at the jenkins.executor.in_use metric and include tags for the host and label in question. Note how the metric value did not drop after the node was marked offline.

Expected behavior I expect the jenkins.executor.free metric to give me the number of actually free executors. As it stands, the metric is barely useful and doesn't match Jenkins's own behavior.

Screenshots In the following screenshot, taken from /label/$LABEL/load-statistics, note how the available executors drop the zero around 10:10 and how there's a short-lived queue. At 10:20, one node with 15 executors that was previously offline was taken online again, increasing the number of online executors. image

In Datadog, the number of free executors never drops below 15: image

Environment and Versions (please complete the following information): Jenkins 2.426.3 running 6.0.2 of the Datadog plugin.

Additional context N/A

nikita-tkachenko-datadog commented 1 month ago

Hi @magnusbaeck, thank you for reporting this! The problem should be fixed in the latest plugin release v7.1.1