search

bairhys / prometheus-frigate-exporter

A Prometheus exporter for Frigate stats
MIT License
33 stars 4 forks source link

Prometheus v2.52.0 raises "Error on ingesting samples with different value but same timestamp" #9

Closed Rauks closed 4 months ago

Rauks commented 4 months ago

Hello,

Using the newly released Prometheus v2.52.0 (https://github.com/prometheus/prometheus/releases/tag/v2.52.0) errors are raised in Prometheus side because of duplicated series from this exporter.

The PR https://github.com/prometheus/prometheus/pull/12933, merged in Prometheus v2.52.0 highlight the issue. Duplicated series in a single scrape seem to no longer be ignored. The error below is raised in the logs and the prometheus_target_scrapes_sample_duplicate_timestamp_total metric is incremented (which I monitor using Alertmanager):

ts=2024-05-11T14:03:07.232Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=frigate-exporter target=http://10.0.10.10:9100/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=2

The two duplicated series in my case are from:

# HELP frigate_storage_mount_type_info Storage mount type
# TYPE frigate_storage_mount_type_info gauge
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0

I think it might be dependant on the Frigate container deployment (volumes, tmpfs...) but I found a similar problem in the sample included in the README of the repository. There is only one duplicate here (common with my case) and this will cause the same issue:

# HELP frigate_storage_mount_type_info Storage mount type
# TYPE frigate_storage_mount_type_info gauge
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="overlay",storage="/"} 1.0

You will find below a sample docker compose having the issue:

version: "3.9"

services:

  frigate:
    image: ghcr.io/blakeblackshear/frigate:stable
    container_name: frigate
    restart: always
    deploy:
      resources:
        limits:
          memory: 8G
    shm_size: 64mb
    cap_add:
      - CAP_PERFMON
    devices:
      - /dev/bus/usb:/dev/bus/usb # USB Coral passthrough
    networks:
      - default
    environment:
      FRIGATE_RTSP_PASSWORD: ${FRIGATE_RTSP_PASSWORD}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - data:/config
      - /mnt/nvr/frigate:/media/frigate
    tmpfs:
      - /tmp/cache:size=1G
    ports:
      - 5000:5000 # UI
      - 8554:8554 # RTSP feeds
      - 8555:8555/tcp # WebRTC over tcp
      - 8555:8555/udp # WebRTC over udp

  frigate-exporter:
    image: rhysbailey/prometheus-frigate-exporter:latest
    container_name: frigate-exporter
    read_only: true
    restart: always
    networks:
      - default
    environment:
      FRIGATE_STATS_URL: http://frigate:5000/api/stats
    ports:
      - 9100:9100

volumes:
  data:

networks:
  default:

And also the full output of my /metrics:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 327.0
python_gc_objects_collected_total{generation="1"} 294.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 42.0
python_gc_collections_total{generation="1"} 3.0
python_gc_collections_total{generation="2"} 0.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="8",version="3.11.8"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 3.1297536e+07
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.9369984e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.7154020025e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 10.39
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 6.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP frigate_audio_dBFS Audio dBFS for camera
# TYPE frigate_audio_dBFS gauge
frigate_audio_dBFS{camera_name="garage-1"} 0.0
frigate_audio_dBFS{camera_name="garage-2"} 0.0
# HELP frigate_audio_rms Audio RMS for camera
# TYPE frigate_audio_rms gauge
frigate_audio_rms{camera_name="garage-1"} 0.0
frigate_audio_rms{camera_name="garage-2"} 0.0
# HELP frigate_camera_fps Frames per second being consumed from your camera.
# TYPE frigate_camera_fps gauge
frigate_camera_fps{camera_name="garage-1"} 5.0
frigate_camera_fps{camera_name="garage-2"} 5.1
# HELP frigate_detection_enabled Detection enabled for camera
# TYPE frigate_detection_enabled gauge
frigate_detection_enabled{camera_name="garage-1"} 1.0
frigate_detection_enabled{camera_name="garage-2"} 1.0
# HELP frigate_detection_fps Number of times detection is run per second.
# TYPE frigate_detection_fps gauge
frigate_detection_fps{camera_name="garage-1"} 0.0
frigate_detection_fps{camera_name="garage-2"} 0.0
# HELP frigate_process_fps Frames per second being processed by frigate.
# TYPE frigate_process_fps gauge
frigate_process_fps{camera_name="garage-1"} 5.0
frigate_process_fps{camera_name="garage-2"} 5.1
# HELP frigate_skipped_fps Frames per second skip for processing by frigate.
# TYPE frigate_skipped_fps gauge
frigate_skipped_fps{camera_name="garage-1"} 0.0
frigate_skipped_fps{camera_name="garage-2"} 0.0
# HELP frigate_bandwidth_usages_kBps bandwidth usages kilobytes per second
# TYPE frigate_bandwidth_usages_kBps gauge
# HELP frigate_detection_total_fps Sum of detection_fps across all cameras and detectors.
# TYPE frigate_detection_total_fps gauge
frigate_detection_total_fps 0.0
# HELP frigate_detector_inference_speed_seconds Time spent running object detection in seconds.
# TYPE frigate_detector_inference_speed_seconds gauge
frigate_detector_inference_speed_seconds{name="coral"} 0.008150000000000001
# HELP frigate_detection_start Detector start time (unix timestamp)
# TYPE frigate_detection_start gauge
frigate_detection_start{name="coral"} 0.0
# HELP frigate_cpu_usage_percent Process CPU usage %
# TYPE frigate_cpu_usage_percent gauge
frigate_cpu_usage_percent{cmdline="ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.0.40.50:8554/unicast -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/garage-1@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale=1280:720 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:",name="garage-1",pid="295921",process="ffmpeg",type="Camera"} 14.6
frigate_cpu_usage_percent{cmdline="frigate.capture:garage-1",name="garage-1",pid="758",process="capture",type="Camera"} 1.1
frigate_cpu_usage_percent{cmdline="frigate.process:garage-1",name="garage-1",pid="753",process="detect",type="Camera"} 0.6
frigate_cpu_usage_percent{cmdline="ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.0.40.50:8555/unicast -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/garage-2@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale=1280:720 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:",name="garage-2",pid="295943",process="ffmpeg",type="Camera"} 16.0
frigate_cpu_usage_percent{cmdline="frigate.capture:garage-2",name="garage-2",pid="764",process="capture",type="Camera"} 2.1
frigate_cpu_usage_percent{cmdline="frigate.process:garage-2",name="garage-2",pid="755",process="detect",type="Camera"} 0.7
frigate_cpu_usage_percent{cmdline="frigate.detector.coral",name="coral",pid="744",process="coral",type="detectors"} 0.0
frigate_cpu_usage_percent{cmdline="/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml",name="go2rtc",pid="99",process="go2rtc",type="go2rtc"} 0.0
frigate_cpu_usage_percent{cmdline="frigate.logger       ",name="logger",pid="725",process="logger",type="logger"} 0.0
frigate_cpu_usage_percent{cmdline="frigate.recording_manager",name="recording",pid="734",process="recording",type="recording"} 0.1
frigate_cpu_usage_percent{cmdline="/package/admin/s6/command/s6-svscan -d4 -- /run/service",name="/package/admin/s6/command/s6-svscan -d4 -- /run/service",pid="1",process="/package/admin/s6/command/s6-svscan -d4 -- /run/service",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise s6-linux-init-shutdownd",name="s6-supervise s6-linux-init-shutdownd",pid="15",process="s6-supervise s6-linux-init-shutdownd",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B",name="/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B",pid="16",process="/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise nginx",name="s6-supervise nginx",pid="24",process="s6-supervise nginx",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise go2rtc-log",name="s6-supervise go2rtc-log",pid="25",process="s6-supervise go2rtc-log",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise frigate-log",name="s6-supervise frigate-log",pid="26",process="s6-supervise frigate-log",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise s6rc-fdholder",name="s6-supervise s6rc-fdholder",pid="27",process="s6-supervise s6rc-fdholder",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise go2rtc-healthcheck",name="s6-supervise go2rtc-healthcheck",pid="28",process="s6-supervise go2rtc-healthcheck",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise frigate",name="s6-supervise frigate",pid="29",process="s6-supervise frigate",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise go2rtc",name="s6-supervise go2rtc",pid="30",process="s6-supervise go2rtc",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise nginx-log",name="s6-supervise nginx-log",pid="31",process="s6-supervise nginx-log",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-supervise s6rc-oneshot-runner",name="s6-supervise s6rc-oneshot-runner",pid="32",process="s6-supervise s6rc-oneshot-runner",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --",name="/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --",pid="39",process="/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules",name="/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules",pid="43",process="/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc",name="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc",pid="78",process="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate",name="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate",pid="81",process="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx",name="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx",pid="82",process="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="bash ./run.user go2rtc-healthcheck",name="bash ./run.user go2rtc-healthcheck",pid="107",process="bash ./run.user go2rtc-healthcheck",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="python3 -u -m frigate",name="python3 -u -m frigate",pid="115",process="python3 -u -m frigate",type="Other"} 0.7
frigate_cpu_usage_percent{cmdline="nginx: master process nginx",name="nginx: master process nginx",pid="120",process="nginx: master process nginx",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="141",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="142",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="143",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="149",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="162",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="178",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="202",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="242",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="272",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="291",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="317",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="358",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="402",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="433",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="460",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="504",process="nginx: worker process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="nginx: cache manager process",name="nginx: cache manager process",pid="563",process="nginx: cache manager process",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(45)",name="/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(45)",pid="743",process="/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(45)",type="Other"} 0.1
frigate_cpu_usage_percent{cmdline="frigate.output       ",name="frigate.output       ",pid="749",process="frigate.output       ",type="Other"} 0.6
frigate_cpu_usage_percent{cmdline="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",name="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",pid="754",process="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",name="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",pid="762",process="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",name="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",pid="767",process="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="bash",name="bash",pid="304526",process="bash",type="Other"} 0.0
frigate_cpu_usage_percent{cmdline="sleep 30s",name="sleep 30s",pid="327203",process="sleep 30s",type="Other"} 0.0
# HELP frigate_mem_usage_percent Process memory usage %
# TYPE frigate_mem_usage_percent gauge
frigate_mem_usage_percent{cmdline="ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.0.40.50:8554/unicast -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/garage-1@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale=1280:720 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:",name="garage-1",pid="295921",process="ffmpeg",type="Camera"} 0.4
frigate_mem_usage_percent{cmdline="frigate.capture:garage-1",name="garage-1",pid="758",process="capture",type="Camera"} 0.7
frigate_mem_usage_percent{cmdline="frigate.process:garage-1",name="garage-1",pid="753",process="detect",type="Camera"} 1.1
frigate_mem_usage_percent{cmdline="ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.13.2-6476f8a -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.0.40.50:8555/unicast -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/garage-2@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale=1280:720 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:",name="garage-2",pid="295943",process="ffmpeg",type="Camera"} 0.4
frigate_mem_usage_percent{cmdline="frigate.capture:garage-2",name="garage-2",pid="764",process="capture",type="Camera"} 0.7
frigate_mem_usage_percent{cmdline="frigate.process:garage-2",name="garage-2",pid="755",process="detect",type="Camera"} 1.1
frigate_mem_usage_percent{cmdline="frigate.detector.coral",name="coral",pid="744",process="coral",type="detectors"} 0.7
frigate_mem_usage_percent{cmdline="/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml",name="go2rtc",pid="99",process="go2rtc",type="go2rtc"} 0.2
frigate_mem_usage_percent{cmdline="frigate.logger       ",name="logger",pid="725",process="logger",type="logger"} 0.5
frigate_mem_usage_percent{cmdline="frigate.recording_manager",name="recording",pid="734",process="recording",type="recording"} 1.3
frigate_mem_usage_percent{cmdline="/package/admin/s6/command/s6-svscan -d4 -- /run/service",name="/package/admin/s6/command/s6-svscan -d4 -- /run/service",pid="1",process="/package/admin/s6/command/s6-svscan -d4 -- /run/service",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise s6-linux-init-shutdownd",name="s6-supervise s6-linux-init-shutdownd",pid="15",process="s6-supervise s6-linux-init-shutdownd",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B",name="/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B",pid="16",process="/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise nginx",name="s6-supervise nginx",pid="24",process="s6-supervise nginx",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise go2rtc-log",name="s6-supervise go2rtc-log",pid="25",process="s6-supervise go2rtc-log",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise frigate-log",name="s6-supervise frigate-log",pid="26",process="s6-supervise frigate-log",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise s6rc-fdholder",name="s6-supervise s6rc-fdholder",pid="27",process="s6-supervise s6rc-fdholder",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise go2rtc-healthcheck",name="s6-supervise go2rtc-healthcheck",pid="28",process="s6-supervise go2rtc-healthcheck",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise frigate",name="s6-supervise frigate",pid="29",process="s6-supervise frigate",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise go2rtc",name="s6-supervise go2rtc",pid="30",process="s6-supervise go2rtc",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise nginx-log",name="s6-supervise nginx-log",pid="31",process="s6-supervise nginx-log",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-supervise s6rc-oneshot-runner",name="s6-supervise s6rc-oneshot-runner",pid="32",process="s6-supervise s6rc-oneshot-runner",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --",name="/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --",pid="39",process="/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules",name="/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules",pid="43",process="/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc",name="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc",pid="78",process="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate",name="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate",pid="81",process="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx",name="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx",pid="82",process="s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="bash ./run.user go2rtc-healthcheck",name="bash ./run.user go2rtc-healthcheck",pid="107",process="bash ./run.user go2rtc-healthcheck",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="python3 -u -m frigate",name="python3 -u -m frigate",pid="115",process="python3 -u -m frigate",type="Other"} 2.5
frigate_mem_usage_percent{cmdline="nginx: master process nginx",name="nginx: master process nginx",pid="120",process="nginx: master process nginx",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="141",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="142",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="143",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="149",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="162",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="178",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="202",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="242",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="272",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="291",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="317",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="358",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="402",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="433",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="460",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: worker process",name="nginx: worker process",pid="504",process="nginx: worker process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="nginx: cache manager process",name="nginx: cache manager process",pid="563",process="nginx: cache manager process",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(45)",name="/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(45)",pid="743",process="/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(45)",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="frigate.output       ",name="frigate.output       ",pid="749",process="frigate.output       ",type="Other"} 0.6
frigate_mem_usage_percent{cmdline="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",name="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",pid="754",process="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",name="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",pid="762",process="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",name="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",pid="767",process="ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="bash",name="bash",pid="304526",process="bash",type="Other"} 0.0
frigate_mem_usage_percent{cmdline="sleep 30s",name="sleep 30s",pid="327203",process="sleep 30s",type="Other"} 0.0
# HELP frigate_gpu_usage_percent GPU utilisation %
# TYPE frigate_gpu_usage_percent gauge
# HELP frigate_gpu_mem_usage_percent GPU memory usage %
# TYPE frigate_gpu_mem_usage_percent gauge
# HELP frigate_service_info Frigate version info
# TYPE frigate_service_info gauge
frigate_service_info{latest_version="0.13.2",version="0.13.2-6476f8a"} 1.0
# HELP frigate_service_uptime_seconds Uptime seconds
# TYPE frigate_service_uptime_seconds gauge
frigate_service_uptime_seconds 454909.0
# HELP frigate_service_last_updated_timestamp Stats recorded time (unix timestamp)
# TYPE frigate_service_last_updated_timestamp gauge
frigate_service_last_updated_timestamp 1.715436543e+09
# HELP frigate_device_temperature Device Temperature
# TYPE frigate_device_temperature gauge
# HELP frigate_storage_free_bytes Storage free bytes
# TYPE frigate_storage_free_bytes gauge
frigate_storage_free_bytes{storage="/dev/shm"} 5.86e+07
frigate_storage_free_bytes{storage="/media/frigate/clips"} 4.172461e+011
frigate_storage_free_bytes{storage="/media/frigate/recordings"} 4.172461e+011
frigate_storage_free_bytes{storage="/tmp/cache"} 1.0085e+09
# HELP frigate_storage_mount_type_info Storage mount type
# TYPE frigate_storage_mount_type_info gauge
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
# HELP frigate_storage_total_bytes Storage total bytes
# TYPE frigate_storage_total_bytes gauge
frigate_storage_total_bytes{storage="/dev/shm"} 6.4e+07
frigate_storage_total_bytes{storage="/media/frigate/clips"} 4.683557e+011
frigate_storage_total_bytes{storage="/media/frigate/recordings"} 4.683557e+011
frigate_storage_total_bytes{storage="/tmp/cache"} 1.024e+09
# HELP frigate_storage_used_bytes Storage used bytes
# TYPE frigate_storage_used_bytes gauge
frigate_storage_used_bytes{storage="/dev/shm"} 5.4e+06
frigate_storage_used_bytes{storage="/media/frigate/clips"} 5.10936e+010
frigate_storage_used_bytes{storage="/media/frigate/recordings"} 5.10936e+010
frigate_storage_used_bytes{storage="/tmp/cache"} 1.55e+07
# HELP frigate_camera_events_total Count of camera events since exporter started
# TYPE frigate_camera_events_total counter
frigate_camera_events_total{camera="garage-1",label="person"} 27.0
frigate_camera_events_total{camera="garage-1",label="car"} 14.0
frigate_camera_events_total{camera="garage-2",label="person"} 25.0
frigate_camera_events_total{camera="garage-2",label="car"} 22.0

Thanks,

bairhys commented 4 months ago

Wow thanks for finding this issue and creating a solution, I have tested and merged your change!