bairhys / prometheus-frigate-exporter

A Prometheus exporter for Frigate stats
MIT License
35 stars 5 forks source link

MiB 5k proc CPU usage #2

Closed PovilasID closed 1 year ago

PovilasID commented 1 year ago

Hey, thanks for awesome exporter, I ran it and got: frigate_cpu_usage_percent{pid="MiB"} 5613.0 I was happy to see my processor can turbot o 5000% 🚀 🤣 but little worried too. Here is my htop: image There is MiB under CPU usageprovided by /api/stats

"MiB": {
"cpu": "5614.3",
"mem": "avail"
},

I think it some kind of an average over time or lost pram not sure it should go to frigate or here. My frigate version: Frigate (0.12.0-da3e197)

bairhys commented 1 year ago

Hi @PovilasID, great to hear you like the exporter 👍

Could you post your Frigate CPU usage graph? I'm curious what it looks like

PovilasID commented 1 year ago

If MiB is not filtered it looks like this: image Is this what you meant?

bairhys commented 1 year ago

Yep that's the graph. Yeah it is not very insightful when MiB is so big. I'm not sure why this is happening but looks to be a Frigate API issue sending wrong data?

It is also interesting you have MiB instead of the pid number (you can see my exporter output on the readme)

You could filter out "MiB" in grafana like this (but sounds like you might be doing already). Could add , pid!="MiB" to the graph query like this to remove it:

image

PovilasID commented 1 year ago

Yup I did exactly that with the gauge up top and this graph. I am starting to suspect that this is a side effect of how frigate scrapes system data and my host OS being OpenWRT. It is a router OS and when I google MiB in relation to CPU it looked like it showed up in networking system prams for networking equipment.

bairhys commented 1 year ago

Wow that's cool you have it running on OpenWRT.

I use OpenWRT too but on my wifi router and Ubuntu for my server that has Frigate, this exporter, Prometheus & Grafana.

PovilasID commented 1 year ago

It is not cool... it's very janky. However, it is both a router and NVR in one box, so I am quite happy about that. This is setup for my garden, so I am running the exporter on a remote server using some mech networks to access the stats page. I think we can close this as solution seams to be outside of the scope for this project :)

SpangleLabs commented 9 months ago

I'm seeing this same issue btw, but running Frigate (and the exporter) on Ubuntu inside a docker image

SpangleLabs commented 9 months ago

Ah, trying to search frigate's github for similar issues is.. tricky, because almost every issue has a dump of the stats endpoint, which includes these false entries https://github.com/blakeblackshear/frigate/issues?q=%22MiB%22+stats+top+tasks

bairhys commented 9 months ago

@SpangleLabs How long have you ran this exporter? Has it only just started showing high cpu or have you only just started using the exporter?

SpangleLabs commented 9 months ago

@SpangleLabs How long have you ran this exporter? Has it only just started showing high cpu or have you only just started using the exporter?

I had literally just spun up the exporter and immediately it shows all those, as the exporter is just mirroring the stats from Frigate's stats endpoint I guess. I filed a ticket on Frigate's side for it, as that's clearly a bug on their end: https://github.com/blakeblackshear/frigate/issues/9373