Prometheus - Error on ingesting samples with different value but same timestamp

reefland commented 3 months ago

Have the following being reported:

ts=2024-06-13T19:35:41.501Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/home-assistant/frigate/0 target=http://10.42.3.40:9100/metrics msg="Duplicate sample for timestamp" series="frigate_gpu_usage_percent{gpu_name=\"amd-vaapi\"}"
ts=2024-06-13T19:35:41.501Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/home-assistant/frigate/0 target=http://10.42.3.40:9100/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=1

Hardware is a simple Minisform UM560:

$ lspci -k | grep -A 3 -E "(VGA|3D)"

04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c2)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0123
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu

When reviewing what is being scraped from Frigate-Exporter, I do see two entries:

# wget -O- http://10.42.3.40:9100/metrics | grep -i amd

Connecting to 10.42.3.40:9100 (10.42.3.40:9100)
writing to stdout
-                    100% |********************************| 31594  0:00:00 ETA
written to stdout
frigate_gpu_usage_percent{gpu_name="amd-vaapi"} 1.67
frigate_gpu_usage_percent{gpu_name="amd-vaapi"} 48.6

# wget -O- http://10.42.3.40:9100/metrics | grep -i gpu

Connecting to 10.42.3.40:9100 (10.42.3.40:9100)
writing to stdout
-                    100% |********************************| 31606  0:00:00 ETA
written to stdout
# HELP frigate_gpu_usage_percent GPU utilisation %
# TYPE frigate_gpu_usage_percent gauge
frigate_gpu_usage_percent{gpu_name="amd-vaapi"} 1.67
frigate_gpu_usage_percent{gpu_name="amd-vaapi"} 48.6
# HELP frigate_gpu_mem_usage_percent GPU memory usage %
# TYPE frigate_gpu_mem_usage_percent gauge

Within Frigate it shows:

It seems like frigate_gpu_usage_percent{gpu_name="amd-vaapi"} 48.6 is missing the mem and should be frigate_gpu_mem_usage_percent{gpu_name="amd-vaapi"} 48.6

Running Exporter Image: latest@sha256:6dd7e8b5e7a6d624fd67f35cf65ec8167ee84e4bfe1722533ab169cf066697b9

With Frigate: 0.13.2@sha256:2906991ccad85035b176941f9dedfd35088ff710c39d45ef1baa9a49f2b16734

bairhys commented 3 months ago

Hi @reefland

Thanks for reporting this issue. I have found the mistake, try updating your Frigate Exporter and should work for you now.

Tests ok for me but my Intel iGPU doesnt report memory usage like your AMD so I never had this issue.

bairhys commented 3 months ago

My Frigate iGPU memory stats look like this

bairhys commented 3 months ago

And I just noticed Frigate web UI mentions this about Intel GPUs:

reefland commented 3 months ago

Thank you for the quick fix. Looking much better now:

Connecting to 10.42.3.233:9100 (10.42.3.233:9100)
writing to stdout
-                    100% |********************************| 31544  0:00:00 ETA
written to stdout
# HELP frigate_gpu_usage_percent GPU utilisation %
# TYPE frigate_gpu_usage_percent gauge
frigate_gpu_usage_percent{gpu_name="amd-vaapi"} 1.67
# HELP frigate_gpu_mem_usage_percent GPU memory usage %
# TYPE frigate_gpu_mem_usage_percent gauge
frigate_gpu_mem_usage_percent{gpu_name="amd-vaapi"} 48.6

bairhys / prometheus-frigate-exporter

Prometheus - Error on ingesting samples with different value but same timestamp #11