search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.39k stars 4.43k forks source link

Metrics on K/V paths #7428

Open chuckyz opened 4 years ago

chuckyz commented 4 years ago

Feature Description

It would be great if we could have custom counters on K/V paths emitted. Even "x keys added since last flush" would be great.

Use Case(s)

If I have a lot of activity in my Vault, and I want to have a rough idea of what secret engine is getting the most activity, or I want to see if I have an abnormal number of leases being created. Even the anonymized data from knowing how many keys were added vault/auth/abcd1234-abcd1234-abcd1234-abcd1234 over a given period of time would be great.

I have no issues doing the actual summation of data in my graphing, just having the data available would be awesome. Right now it's required to do a recursive key lookup which is extremely heavy on Consul and I'd like to avoid it.

mkcp commented 4 years ago

Hey @chuckyz, we've discussed adding a counter per key in the past and it's not something that we're likely to do. Emitting a timeseries per KV key would mean we would emit an unbounded number of series. Rather than building out mechanisms only emit stats on certain keys, we thought that more data in our structured logs would allow users to solve this problem themselves. If we logged out the key and duration for each KV operation, this would allow users to load this up in a log aggregator and build up timeseries for the keys they're interested in. Would this solve your problem?

chuckyz commented 4 years ago

Hi @mkcp! If I could get something like this, which currently I gather with a pretty nasty 1-liner via consul trace, in a lighter more maintainable way I'd love that.