search

SteeltoeOSS / Steeltoe

.NET Components for Externalized Configuration, Database Connectors, Service Discovery, Logging and Distributed Tracing, Application Management, Security, and more.
https://steeltoe.io
Apache License 2.0
998 stars 162 forks source link

Prometheus metrics reset every 5s #800

Closed merusso-regions closed 2 years ago

merusso-regions commented 2 years ago

Describe the bug

Prometheus metrics are being reset every ~5 seconds, preventing Prometheus server from collecting accurate metrics.

Steps to reproduce

Steps to reproduce the behavior:

  1. Generate Steeltoe app: https://start.steeltoe.io/#!name=Sample&namespace=Sample&description=Sample%20project&steeltoeVersion=3.1.1&dotNetFramework=netcoreapp3.1&dotNetTemplate=&language=csharp&packaging=zip&dependencies=management-endpoints
  2. Expose /actuator/prometheus endpoint
  3. Hit /actuator/prometheus endpoint in browser

Expected behavior

Counters like http_server_requests_count should continually increment. Instead, these counters are resetting after ~5 seconds, discarding these counter metrics. This results in incorrect metrics data in Prometheus server.

Environment (please complete the following information):

Screenshots

Example output from /actuator/prometheus:

# HELP http_server_requests_seconds Steeltoehttp.server.requests.seconds
# TYPE http_server_requests_seconds summary
http_server_requests_seconds{exception="None",method="GET",status="200",uri="/actuator/prometheus"} 0
# HELP http_server_requests_count Steeltoehttp.server.requests.count
# TYPE http_server_requests_count counter
http_server_requests_count{exception="None"} 0
http_server_requests_count{method="GET"} 0
http_server_requests_count{status="200"} 0
http_server_requests_count{uri="/actuator/prometheus"} 0
# HELP Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadWait_RetiredWorkerThreadCount SteeltoeMicrosoft-Windows-DotNETRuntime.ThreadPoolWorkerThreadWait.RetiredWorkerThreadCount
# TYPE Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadWait_RetiredWorkerThreadCount summary
Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadWait_RetiredWorkerThreadCount{kind="worker"} 0
# HELP clr_threadpool_available Steeltoeclr.threadpool.available
# TYPE clr_threadpool_available summary
clr_threadpool_available{kind="worker"} 32763
# HELP Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStart_RetiredWorkerThreadCount SteeltoeMicrosoft-Windows-DotNETRuntime.ThreadPoolWorkerThreadStart.RetiredWorkerThreadCount
# TYPE Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStart_RetiredWorkerThreadCount summary
Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStart_RetiredWorkerThreadCount{kind="worker"} 0
# HELP Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStop_ActiveWorkerThreadCount SteeltoeMicrosoft-Windows-DotNETRuntime.ThreadPoolWorkerThreadStop.ActiveWorkerThreadCount
# TYPE Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStop_ActiveWorkerThreadCount summary
Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStop_ActiveWorkerThreadCount{kind="worker"} 0
# HELP Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadWait_ActiveWorkerThreadCount SteeltoeMicrosoft-Windows-DotNETRuntime.ThreadPoolWorkerThreadWait.ActiveWorkerThreadCount
# TYPE Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadWait_ActiveWorkerThreadCount summary
Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadWait_ActiveWorkerThreadCount{kind="worker"} 4
# HELP Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStop_RetiredWorkerThreadCount SteeltoeMicrosoft-Windows-DotNETRuntime.ThreadPoolWorkerThreadStop.RetiredWorkerThreadCount
# TYPE Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStop_RetiredWorkerThreadCount summary
Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStop_RetiredWorkerThreadCount{kind="worker"} 0
# HELP Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStart_ActiveWorkerThreadCount SteeltoeMicrosoft-Windows-DotNETRuntime.ThreadPoolWorkerThreadStart.ActiveWorkerThreadCount
# TYPE Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStart_ActiveWorkerThreadCount summary
Microsoft_Windows_DotNETRuntime_ThreadPoolWorkerThreadStart_ActiveWorkerThreadCount{kind="worker"} 0
hananiel commented 2 years ago

@merusso-regions the metrics are collected and aggregated over an interval in your case 5 seconds. So in the last 5 seconds if there are no new http requests, it will reset. So its up to the backend system downstream from prometheus to treat this information appropriately.

hananiel commented 2 years ago

will be addressed in #299

hananiel commented 2 years ago

Resolved by https://github.com/SteeltoeOSS/Steeltoe/pull/828