Prometheus metrics "fluentbit_output_upstream_busy_connections" shows negative value during connection timed out

[ashishmodi7]

Bug Report

Describe the bug Prometheus metrics "fluentbit_output_upstream_busy_connections" showing negative value during connection timed out

To Reproduce Steps to reproduce the problem:

1. Deploy Fluent Bit in Kubernetes (https://docs.fluentbit.io/manual/installation/kubernetes#installing-with-helm-chart)
2. Configure Port forwarding to view the Prometheus metrics using below command: export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=fluent-bit,app.kubernetes.io/instance=fluent-bit" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $POD_NAME 2020:2020
3. Configure Fluent Bit Output to Elastic Search or Splunk Server
4. When the Elastic Search or Splunk Server is not reachable, it will give connection timed out error.
5. Check the Prometheus Metrics "fluentbit_output_upstream_busy_connections" showing negative value. curl -s http://127.0.0.1:2020/api/v2/metrics/prometheus|grep conn

Expected behavior Prometheus Metrics "fluentbit_output_upstream_busy_connections" should show 0 or positive value.

Screenshots

Your Environment

• Version used: Fluent Bit 3.0.5
• Configuration: Deafult configuration
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes Client Version: v1.30.1 , Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 , Server Version: v1.30.0
• Server type and version: Linux
• Operating System and version: Rocky Linux 8.9
• Filters and plugins: Elastic Search / Splunk

Additional context Monitoring graphs are not showing correct values

[drbugfinder-work]

Verified on my end. I can also see negative values here. fluentbit_output_upstream_busy_connections{name="forward"} -899

[douglasawh]

@drbugfinder-work Is there any additional information we can provide to help get this resolved?

[drbugfinder-work]

Just as a side note This is where the calculation is done (without mutex): https://github.com/fluent/fluent-bit/blob/8aee285464c30d1af03fdfbf1dcbdf784b5ace33/src/flb_upstream.c#L1157-L1214

Called here: https://github.com/fluent/fluent-bit/blob/8aee285464c30d1af03fdfbf1dcbdf784b5ace33/src/flb_upstream.c#L801-L807

(First guess: Is access to the metrics thread-safe? cc @leonardo-albertovich)