No camera or detector metrics

[bairhys]

Continues from here

Container is starting but http://*****:9100 or ../metrics shows no camera or detector metrics:

# HELP frigate_gpu_usage_percent GPU utilisation %
# TYPE frigate_gpu_usage_percent gauge
frigate_gpu_usage_percent{gpu="intel-qsv"} 6.0
# HELP frigate_gpu_mem_usage_percent GPU memory usage %
# TYPE frigate_gpu_mem_usage_percent gauge
# HELP frigate_service_info Frigate version info
# TYPE frigate_service_info gauge
frigate_service_info{latest_version="0.11.1",version="0.12.0-27a31e7"} 1.0
# HELP frigate_service_uptime_seconds Uptime seconds
# TYPE frigate_service_uptime_seconds gauge
frigate_service_uptime_seconds 54186.0
# HELP frigate_service_last_updated_timestamp Stats recorded time (unix timestamp)
# TYPE frigate_service_last_updated_timestamp gauge
frigate_service_last_updated_timestamp 1.676280155e+09
# HELP frigate_storage_free_bytes Storage free bytes
# TYPE frigate_storage_free_bytes gauge
frigate_storage_free_bytes{storage="/dev/shm"} 5.27e+08
frigate_storage_free_bytes{storage="/media/frigate/clips"} 8.97021e+011
frigate_storage_free_bytes{storage="/media/frigate/recordings"} 8.97021e+011
frigate_storage_free_bytes{storage="/tmp/cache"} 9.36e+08
# HELP frigate_storage_mount_type_info Storage mount type
# TYPE frigate_storage_mount_type_info gauge
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
# HELP frigate_storage_total_bytes Storage total bytes
# TYPE frigate_storage_total_bytes gauge
frigate_storage_total_bytes{storage="/dev/shm"} 5.36e+08
frigate_storage_total_bytes{storage="/media/frigate/clips"} 1.005441e+012
frigate_storage_total_bytes{storage="/media/frigate/recordings"} 1.005441e+012
frigate_storage_total_bytes{storage="/tmp/cache"} 1e+09
# HELP frigate_storage_used_bytes Storage used bytes
# TYPE frigate_storage_used_bytes gauge
frigate_storage_used_bytes{storage="/dev/shm"} 9e+06
frigate_storage_used_bytes{storage="/media/frigate/clips"} 5.7271e+010
frigate_storage_used_bytes{storage="/media/frigate/recordings"} 5.7271e+010
frigate_storage_used_bytes{storage="/tmp/cache"} 6.3e+07

[bairhys]

@ikavior

Not sure why it is not displaying the more interesting data like cameras and detectors, since I am also on same version as you (0.12.0-27a31e7).

It is like my python script is jumping straight to line 107 or your browser is chopping off the first ~200 lines on the ../metrics page, since it should at least show empty metrics for cameras and detectors if not on /api/stats . For example, your GPU memory usage % is empty - it has the # HELP frigate_gpu_mem_usage_percent GPU memory usage % help text but no metrics below it. My metrics are the same for GPU memory usage %.

Does it show more metrics on a different browser or device?

Does Prometheus show data for missing metrics (e.g. frigate_detector_inference_speed_seconds)?

My Prometheus metrics for frigate_detector_inference_speed_seconds and frigate_detection_total_fps look like this:

[ikavior]

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 221.0
python_gc_objects_collected_total{generation="1"} 156.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 43.0
python_gc_collections_total{generation="1"} 3.0
python_gc_collections_total{generation="2"} 0.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="10",patchlevel="10",version="3.10.10"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.6107904e+07
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.9996672e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.67629052713e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.28
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 6.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 524288.0
# HELP frigate_ffmpeg_pid PID for ffmpeg process
# TYPE frigate_ffmpeg_pid gauge
frigate_ffmpeg_pid{camera="Cam1"} 307.0
frigate_ffmpeg_pid{camera="Cam2"} 314.0
frigate_ffmpeg_pid{camera="Cam4"} 333.0
frigate_ffmpeg_pid{camera="Cam5"} 326.0
frigate_ffmpeg_pid{camera="Cam6"} 335.0
frigate_ffmpeg_pid{camera="Cam8"} 348.0
frigate_ffmpeg_pid{camera="Cam9"} 347.0
# HELP frigate_capture_pid PID for the ffmpeg process that consumes this camera
# TYPE frigate_capture_pid gauge
frigate_capture_pid{camera="Cam1"} 302.0
frigate_capture_pid{camera="Cam2"} 305.0
frigate_capture_pid{camera="Cam4"} 309.0
frigate_capture_pid{camera="Cam5"} 313.0
frigate_capture_pid{camera="Cam6"} 322.0
frigate_capture_pid{camera="Cam8"} 331.0
frigate_capture_pid{camera="Cam9"} 337.0
# HELP frigate_detect_pid PID for the process that runs detection for this camera
# TYPE frigate_detect_pid gauge
frigate_detect_pid{camera="Cam1"} 288.0
frigate_detect_pid{camera="Cam2"} 291.0
frigate_detect_pid{camera="Cam4"} 293.0
frigate_detect_pid{camera="Cam5"} 295.0
frigate_detect_pid{camera="Cam6"} 297.0
frigate_detect_pid{camera="Cam8"} 298.0
frigate_detect_pid{camera="Cam9"} 300.0
# HELP frigate_camera_fps Frames per second being consumed from your camera.
# TYPE frigate_camera_fps gauge
frigate_camera_fps{camera="Cam1"} 5.0
frigate_camera_fps{camera="Cam2"} 23.0
frigate_camera_fps{camera="Cam4"} 6.0
frigate_camera_fps{camera="Cam5"} 7.0
frigate_camera_fps{camera="Cam6"} 6.0
frigate_camera_fps{camera="Cam8"} 5.0
frigate_camera_fps{camera="Cam9"} 5.0
# HELP frigate_detection_fps Number of times detection is run per second.
# TYPE frigate_detection_fps gauge
frigate_detection_fps{camera="Cam1"} 2.0
frigate_detection_fps{camera="Cam2"} 0.0
frigate_detection_fps{camera="Cam4"} 0.0
frigate_detection_fps{camera="Cam5"} 0.0
frigate_detection_fps{camera="Cam6"} 0.0
frigate_detection_fps{camera="Cam8"} 0.0
frigate_detection_fps{camera="Cam9"} 0.0
# HELP frigate_process_fps Frames per second being processed by frigate.
# TYPE frigate_process_fps gauge
frigate_process_fps{camera="Cam1"} 5.0
frigate_process_fps{camera="Cam2"} 23.0
frigate_process_fps{camera="Cam4"} 6.0
frigate_process_fps{camera="Cam5"} 7.0
frigate_process_fps{camera="Cam6"} 6.0
frigate_process_fps{camera="Cam8"} 5.0
frigate_process_fps{camera="Cam9"} 5.0
# HELP frigate_skipped_fps Frames per second skip for processing by frigate.
# TYPE frigate_skipped_fps gauge
frigate_skipped_fps{camera="Cam1"} 0.0
frigate_skipped_fps{camera="Cam2"} 0.0
frigate_skipped_fps{camera="Cam4"} 0.0
frigate_skipped_fps{camera="Cam5"} 0.0
frigate_skipped_fps{camera="Cam6"} 0.0
frigate_skipped_fps{camera="Cam8"} 0.0
frigate_skipped_fps{camera="Cam9"} 0.0
# HELP frigate_detection_enabled Detection enabled for camera
# TYPE frigate_detection_enabled gauge
frigate_detection_enabled{camera="Cam1"} 1.0
frigate_detection_enabled{camera="Cam2"} 1.0
frigate_detection_enabled{camera="Cam4"} 1.0
frigate_detection_enabled{camera="Cam5"} 1.0
frigate_detection_enabled{camera="Cam6"} 1.0
frigate_detection_enabled{camera="Cam8"} 1.0
frigate_detection_enabled{camera="Cam9"} 1.0
# HELP frigate_detection_total_fps Sum of detection_fps across all cameras and detectors.
# TYPE frigate_detection_total_fps gauge
frigate_detection_total_fps 3.4
# HELP frigate_detector_inference_speed_seconds Time spent running object detection in seconds.
# TYPE frigate_detector_inference_speed_seconds gauge
frigate_detector_inference_speed_seconds{name="coral"} 0.014
# HELP frigate_detector_pid PID for the shared process that runs object detection on the detector
# TYPE frigate_detector_pid gauge
frigate_detector_pid{name="coral"} 280.0
# HELP frigate_detection_start Detector start time (unix timestamp)
# TYPE frigate_detection_start gauge
frigate_detection_start{name="coral"} 0.0
# HELP frigate_cpu_usage_percent Process CPU usage %
# TYPE frigate_cpu_usage_percent gauge
frigate_cpu_usage_percent{pid="1"} 0.0
frigate_cpu_usage_percent{pid="10388"} 0.0
frigate_cpu_usage_percent{pid="10398"} 0.0
frigate_cpu_usage_percent{pid="107"} 18.0
frigate_cpu_usage_percent{pid="112"} 0.0
frigate_cpu_usage_percent{pid="128"} 0.0
frigate_cpu_usage_percent{pid="129"} 0.0
frigate_cpu_usage_percent{pid="130"} 0.0
frigate_cpu_usage_percent{pid="142"} 0.0
frigate_cpu_usage_percent{pid="15"} 0.0
frigate_cpu_usage_percent{pid="17"} 0.0
frigate_cpu_usage_percent{pid="24"} 0.0
frigate_cpu_usage_percent{pid="25"} 0.0
frigate_cpu_usage_percent{pid="26"} 0.0
frigate_cpu_usage_percent{pid="27"} 0.0
frigate_cpu_usage_percent{pid="273"} 0.0
frigate_cpu_usage_percent{pid="279"} 2.0
frigate_cpu_usage_percent{pid="28"} 0.0
frigate_cpu_usage_percent{pid="280"} 4.0
frigate_cpu_usage_percent{pid="282"} 6.0
frigate_cpu_usage_percent{pid="287"} 0.0
frigate_cpu_usage_percent{pid="288"} 2.0
frigate_cpu_usage_percent{pid="29"} 0.0
frigate_cpu_usage_percent{pid="290"} 0.0
frigate_cpu_usage_percent{pid="291"} 10.0
frigate_cpu_usage_percent{pid="292"} 0.0
frigate_cpu_usage_percent{pid="293"} 3.0
frigate_cpu_usage_percent{pid="294"} 0.0
frigate_cpu_usage_percent{pid="295"} 9.0
frigate_cpu_usage_percent{pid="296"} 0.0
frigate_cpu_usage_percent{pid="297"} 2.0
frigate_cpu_usage_percent{pid="298"} 2.0
frigate_cpu_usage_percent{pid="299"} 0.0
frigate_cpu_usage_percent{pid="30"} 0.0
frigate_cpu_usage_percent{pid="300"} 2.0
frigate_cpu_usage_percent{pid="301"} 0.0
frigate_cpu_usage_percent{pid="302"} 1.0
frigate_cpu_usage_percent{pid="305"} 7.0
frigate_cpu_usage_percent{pid="306"} 0.0
frigate_cpu_usage_percent{pid="307"} 4.0
frigate_cpu_usage_percent{pid="309"} 2.0
frigate_cpu_usage_percent{pid="31"} 0.0
frigate_cpu_usage_percent{pid="313"} 2.0
frigate_cpu_usage_percent{pid="314"} 19.0
frigate_cpu_usage_percent{pid="321"} 2.0
frigate_cpu_usage_percent{pid="322"} 2.0
frigate_cpu_usage_percent{pid="326"} 9.0
frigate_cpu_usage_percent{pid="331"} 1.0
frigate_cpu_usage_percent{pid="332"} 2.0
frigate_cpu_usage_percent{pid="333"} 9.0
frigate_cpu_usage_percent{pid="335"} 7.0
frigate_cpu_usage_percent{pid="337"} 1.0
frigate_cpu_usage_percent{pid="347"} 4.0
frigate_cpu_usage_percent{pid="348"} 4.0
frigate_cpu_usage_percent{pid="352"} 2.0
frigate_cpu_usage_percent{pid="375"} 2.0
frigate_cpu_usage_percent{pid="377"} 2.0
frigate_cpu_usage_percent{pid="40"} 0.0
frigate_cpu_usage_percent{pid="402"} 2.0
frigate_cpu_usage_percent{pid="41"} 0.0
frigate_cpu_usage_percent{pid="429"} 2.0
frigate_cpu_usage_percent{pid="79"} 0.0
frigate_cpu_usage_percent{pid="80"} 0.0
frigate_cpu_usage_percent{pid="81"} 0.0
frigate_cpu_usage_percent{pid="88"} 47.0
frigate_cpu_usage_percent{pid="MiB"} 1760.0
frigate_cpu_usage_percent{pid="top"} 0.0
# HELP frigate_mem_usage_percent Process memory usage %
# TYPE frigate_mem_usage_percent gauge
frigate_mem_usage_percent{pid="%Cpu(s):"} 0.0
frigate_mem_usage_percent{pid="1"} 0.0
frigate_mem_usage_percent{pid="10388"} 0.0
frigate_mem_usage_percent{pid="10398"} 0.0
frigate_mem_usage_percent{pid="107"} 5.0
frigate_mem_usage_percent{pid="112"} 0.0
frigate_mem_usage_percent{pid="128"} 0.0
frigate_mem_usage_percent{pid="129"} 1.0
frigate_mem_usage_percent{pid="130"} 0.0
frigate_mem_usage_percent{pid="142"} 0.0
frigate_mem_usage_percent{pid="15"} 0.0
frigate_mem_usage_percent{pid="17"} 0.0
frigate_mem_usage_percent{pid="24"} 0.0
frigate_mem_usage_percent{pid="25"} 0.0
frigate_mem_usage_percent{pid="26"} 0.0
frigate_mem_usage_percent{pid="27"} 0.0
frigate_mem_usage_percent{pid="273"} 1.0
frigate_mem_usage_percent{pid="279"} 0.0
frigate_mem_usage_percent{pid="28"} 0.0
frigate_mem_usage_percent{pid="280"} 2.0
frigate_mem_usage_percent{pid="282"} 2.0
frigate_mem_usage_percent{pid="287"} 1.0
frigate_mem_usage_percent{pid="288"} 1.0
frigate_mem_usage_percent{pid="29"} 0.0
frigate_mem_usage_percent{pid="290"} 1.0
frigate_mem_usage_percent{pid="291"} 2.0
frigate_mem_usage_percent{pid="292"} 0.0
frigate_mem_usage_percent{pid="293"} 1.0
frigate_mem_usage_percent{pid="294"} 0.0
frigate_mem_usage_percent{pid="295"} 1.0
frigate_mem_usage_percent{pid="296"} 0.0
frigate_mem_usage_percent{pid="297"} 1.0
frigate_mem_usage_percent{pid="298"} 1.0
frigate_mem_usage_percent{pid="299"} 1.0
frigate_mem_usage_percent{pid="30"} 0.0
frigate_mem_usage_percent{pid="300"} 1.0
frigate_mem_usage_percent{pid="301"} 1.0
frigate_mem_usage_percent{pid="302"} 1.0
frigate_mem_usage_percent{pid="305"} 1.0
frigate_mem_usage_percent{pid="306"} 2.0
frigate_mem_usage_percent{pid="307"} 1.0
frigate_mem_usage_percent{pid="309"} 1.0
frigate_mem_usage_percent{pid="31"} 0.0
frigate_mem_usage_percent{pid="313"} 1.0
frigate_mem_usage_percent{pid="314"} 1.0
frigate_mem_usage_percent{pid="321"} 0.0
frigate_mem_usage_percent{pid="322"} 1.0
frigate_mem_usage_percent{pid="326"} 1.0
frigate_mem_usage_percent{pid="331"} 1.0
frigate_mem_usage_percent{pid="332"} 1.0
frigate_mem_usage_percent{pid="333"} 1.0
frigate_mem_usage_percent{pid="335"} 1.0
frigate_mem_usage_percent{pid="337"} 1.0
frigate_mem_usage_percent{pid="347"} 1.0
frigate_mem_usage_percent{pid="348"} 1.0
frigate_mem_usage_percent{pid="352"} 0.0
frigate_mem_usage_percent{pid="375"} 0.0
frigate_mem_usage_percent{pid="377"} 0.0
frigate_mem_usage_percent{pid="40"} 0.0
frigate_mem_usage_percent{pid="402"} 0.0
frigate_mem_usage_percent{pid="41"} 0.0
frigate_mem_usage_percent{pid="429"} 0.0
frigate_mem_usage_percent{pid="79"} 0.0
frigate_mem_usage_percent{pid="80"} 0.0
frigate_mem_usage_percent{pid="81"} 0.0
frigate_mem_usage_percent{pid="88"} 0.0
frigate_mem_usage_percent{pid="Tasks:"} 0.0
# HELP frigate_gpu_usage_percent GPU utilisation %
# TYPE frigate_gpu_usage_percent gauge
frigate_gpu_usage_percent{gpu="intel-qsv"} 5.0
# HELP frigate_gpu_mem_usage_percent GPU memory usage %
# TYPE frigate_gpu_mem_usage_percent gauge
# HELP frigate_service_info Frigate version info
# TYPE frigate_service_info gauge
frigate_service_info{latest_version="0.11.1",version="0.12.0-27a31e7"} 1.0
# HELP frigate_service_uptime_seconds Uptime seconds
# TYPE frigate_service_uptime_seconds gauge
frigate_service_uptime_seconds 3175.0
# HELP frigate_service_last_updated_timestamp Stats recorded time (unix timestamp)
# TYPE frigate_service_last_updated_timestamp gauge
frigate_service_last_updated_timestamp 1.676290545e+09
# HELP frigate_storage_free_bytes Storage free bytes
# TYPE frigate_storage_free_bytes gauge
frigate_storage_free_bytes{storage="/dev/shm"} 5.29e+08
frigate_storage_free_bytes{storage="/media/frigate/clips"} 8.88546e+011
frigate_storage_free_bytes{storage="/media/frigate/recordings"} 8.88546e+011
frigate_storage_free_bytes{storage="/tmp/cache"} 9.35e+08
# HELP frigate_storage_mount_type_info Storage mount type
# TYPE frigate_storage_mount_type_info gauge
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="ext4",storage="/"} 1.0
frigate_storage_mount_type_info{mount_type="tmpfs",storage="/"} 1.0
# HELP frigate_storage_total_bytes Storage total bytes
# TYPE frigate_storage_total_bytes gauge
frigate_storage_total_bytes{storage="/dev/shm"} 5.36e+08
frigate_storage_total_bytes{storage="/media/frigate/clips"} 1.005441e+012
frigate_storage_total_bytes{storage="/media/frigate/recordings"} 1.005441e+012
frigate_storage_total_bytes{storage="/tmp/cache"} 1e+09
# HELP frigate_storage_used_bytes Storage used bytes
# TYPE frigate_storage_used_bytes gauge
frigate_storage_used_bytes{storage="/dev/shm"} 7e+06
frigate_storage_used_bytes{storage="/media/frigate/clips"} 6.5745e+010
frigate_storage_used_bytes{storage="/media/frigate/recordings"} 6.5745e+010
frigate_storage_used_bytes{storage="/tmp/cache"} 6.4e+07

Thats full copy paste from Chrome. Frigate runs on machine 1, your container runs on machine 2.

[bairhys]

Thanks for sharing the output. Your prometheus exporter is working correctly since you have all the same metrics as I have. This exporter converts the Frigate stats into a Prometheus format. Then Prometheus "scrapes" this exporter url every ~15 seconds and records the data in its database. Can view the time series data in Prometheus or preferably Grafana for nice dashboards.

To get the graphs, need to

1. Create Prometheus and Grafana docker containers,

2. In Prometheus config file, add the frigate metrics url to scrape targets section, end of my config looks like this

   scrape_configs:
   # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
   - job_name: "frigate"
   static_configs:
     - targets: [
                   "<frigate_prometheus_exporter_ip>:9100"
                ]

3. In Grafana, connect Prometheus in Settings->data sources

4. Import my Grafana config in this repo or make a new dashboard

5. Should be able to view graphs

I might write up a detailed guide and add it to the read me later. An automated way of setting up Prometheus and Grafana containers is to install Internet Pi then edit the Prometheus config and setup Grafana

[bairhys]

Hi @ikavior, I have made a guide how to setup this exporter, prometheus and grafana, see here:

https://github.com/bairhys/prometheus-frigate-exporter