ruscalworld / fabric-exporter

Fabric mod that adds a Prometheus exporter with general metrics of your server
https://modrinth.com/mod/fabricexporter
MIT License
33 stars 6 forks source link

Reporting 0 #12

Closed meza closed 2 months ago

meza commented 2 months ago

Hey,

I'm using grafana to keep track of my servers health and I've just noticed that when grafana doesn't receive a value, it just assumes the last known value which does lead to a lot of confusion on the charts.

I appreciate this might not be the mod's remit but how would you go about adding the output when "no data" means 0?

I thought I'd ask before changing the code to report 0s for all entities it has previously reported on and doesn't have anything coming from Minecraft.

ruscalworld commented 2 months ago

Hi!

I guess the mod just should clear all the labels before any of metric updates. This will cause all obsolete metrics not to be reported at all.

It should be even better to clear labels only after the stats are collected. Otherwise there would be a tiny probability that Prometheus will request metrics after all labels are cleared, but before the stats are actually collected, so some points on the charts would be missing.

meza commented 2 months ago

This seems to be the bigger issue with entities reported. I was thinking about maybe keeping track of what we knew of and combining the final report with the currently active entity numbers with the default 0 for the ones that didn't get an update

ruscalworld commented 2 months ago

But this way we will still have no metrics on count of entities that have never appeared in the world. So some "unknown" entities will have zero count value, but others will have no value at all. That doesn't seem to be a correct approach.

I see two ways of solving this:

  1. Clearing all metrics on entities we cannot calculate count of (as I suggested before)
  2. Iterating over all types of entities that are known to server and setting count value for all these types (which will allow us to set zero values or even clear obsolete labels)
meza commented 2 months ago

Yeah, I don't think I quite understand how Prometheus works under the hood :D The current Entity metric reporting conceptually makes sense to me as you're only ever reporting the active entities.

I guess the gague keeps track of the previous values and not Grafana?

I could see value in both options. If you clear them like you recommended, that's nice and clean. If you include all known entities, that could enable more deterministic logging and alerting for people.

ruscalworld commented 2 months ago

I guess the gague keeps track of the previous values and not Grafana?

Gauges just store current values of metrics. Then prometheus periodically polls your exporter and picks up the current values of gauges.

So yes, the problem is that mod stores obsolete values in gauges.

ruscalworld commented 2 months ago

I'll try to implement cleanups of obsolete labels by the end of the week

meza commented 2 months ago

Thanks! Let me know if you need some help, happy to chip in