search

nats-io / prometheus-nats-exporter

A Prometheus exporter for NATS metrics
Apache License 2.0
375 stars 137 forks source link

Export connection info #78

Open rigtorp opened 5 years ago

rigtorp commented 5 years ago

What: Export information about individual connections

Why: To allow prometheus alerting on slow clients and track bandwidth usage by client.

How: Something like this:

gnatsd_connz_out_bytes{name="xxx", ip="host", port="port", cid="yyy"}
gnatsd_connz_in_bytes{name="xxx", ip="host", port="port", cid="yyy"}
... etc

SNMP exporter uses similar pattern:

ifHCOutOctets{ifAlias="",ifDescr="Unit: 1 Slot: 0 Port: 27 10G - Level",ifIndex="27",ifName="Te1/0/27"} 48839
ifInErrors{ifAlias="",ifDescr="Unit: 1 Slot: 0 Port: 23 10G - Level",ifIndex="23",ifName="Te1/0/23"} 0
wutkemtt commented 4 years ago

We need this feature too to be able to define limits and alerts based on number of connections and subscriptions per client. Furthermore showing additional connection option like MaxPubInFlight would help to prevent "rogue" clients from crashing the whole cluster.

ripienaar commented 4 years ago

I dont think this will work well, the way prom stores data this will lead to a huge explosion of data even on relatively small networks

ripienaar commented 4 years ago

Here are guidelines from the author/maintainers:

As a general rule of thumb I'd avoid having any metric whose cardinality on a /metrics could potentially go over 10 due to growth in label values.

https://www.robustperception.io/cardinality-is-key

Your proposal even if 1 client connects 10 times from the same host to the server we'd already be on 40 different metrics (port unique each time). Even just including the client name would quickly become a performance issue on the prom server.