Problem statistics in Prometheus at forwarder

[pe1mew]

Yesterday at approximately 18:20 a gateway reconnected to the forwarder. This gateway was onboarded and not online when I started monitoring the forwarder with Prometheus. the gateway is a Mikrotik wap8 LoRaWAN gateway.

Instantly after connection statistics in Prometheus do not show the number of packets per interval but cumulative. This seems incorrect. The gateway data is multiplexed to 2 forwarders connected to main-net and test-net. Both forwarders show the same result.

The forwarder on main-net is reset to see if the issue is persistent. Investigation continues.

[bas-vk]

This is intentional. The forwarder uses counters to count the number of up/downlinks since it was started. The forwarder has no notion of intervals. It just sees a packet and increments the associated counter. Typically you use something like grafana's rate or increase functions to convert counters to something it is more meaningful.

The whole prometheus integration is premature and quickly done. Would love to hear what kind of data you want out of the forwarder/router.

[pe1mew]

When I look at http://:8888/metrics of my forwarder I see:

# HELP gateways_rx_received Received packets by gateway
# TYPE gateways_rx_received gauge
gateways_rx_received{gw_network_id="0d8a014a81b80137",status="success"} 1
gateways_rx_received{gw_network_id="0dcc7bbb472f928a",status="success"} 1
gateways_rx_received{gw_network_id="19448279e5e8104b",status="success"} 16
gateways_rx_received{gw_network_id="2427d14d70814628",status="success"} 7
gateways_rx_received{gw_network_id="2b5e5d140e020ca7",status="success"} 2
gateways_rx_received{gw_network_id="3e1926fb40325641",status="success"} 1
gateways_rx_received{gw_network_id="5ee65225c031f93e",status="success"} 25
gateways_rx_received{gw_network_id="8fa03c6ce4fdccbb",status="success"} 658
gateways_rx_received{gw_network_id="a021046f2d0dfe9b",status="success"} 18
gateways_rx_received{gw_network_id="a841135142b75353",status="success"} 1
gateways_rx_received{gw_network_id="d271b971d25c540f",status="success"} 12
gateways_rx_received{gw_network_id="db5747c4da2cd660",status="success"} 25
gateways_rx_received{gw_network_id="ee37c478c382faaf",status="success"} 11
gateways_rx_received{gw_network_id="ef19dde8ed06a649",status="success"} 2
gateways_rx_received{gw_network_id="f5705810363e14a0",status="success"} 4

Every time I refresh I see new numbers. Only with gw_network_id="8fa03c6ce4fdccbb" count does not reset, it increments. this is not described as in: https://prometheus.io/docs/concepts/metric_types/#gauge

I do not believe this is intentional. sorry. I can accept an error at this stage of development as this feature might not have any priority.

I am happy to discuss, what I think, is useful infromation.

[timcooijmans]

You are both (a bit) right:

• We should do counters (as @bas-vk said) that count the packets and users should use a rate or increase function to make a sensible displays. Using a counter has several benefits:
  • You can pick your own display rate (per second, minute, hour)
  • You can pick your own polling rate
  • You don't mis any events as everything is counted
• We currently use the packets-stats provided by the gateways (that are not that useful), they should provide the packets received in the rate-period (and we provide them as a gauge, as they should reset every very seconds, that you shouldn't use rate on as @pe1mew correctly noted). However apparently the MikroTik implementation is different/wrong (by the way make sure you update the MikroTik firmware, they had a lot of LoRa issues).

My proposal would be to implement counters in our own forwarding code and ignore the (not so useful) stats provided by the gateways themselves. I will have a go at it.

[timcooijmans]

This has been fixed in #64