zendesk / samson

Web interface for deployments, with plugin architecture and kubernetes support
Other
1.45k stars 234 forks source link

Remove reporting of unqueried metrics. #4076

Closed sarahjmiller closed 8 months ago

sarahjmiller commented 8 months ago

Note: Samson is a public repo, do not include Zendesk-internal information, urls, etc.

The metrics cost optimization project requested that we remove the process stats being reported by Samson, which are not queried by any existing monitors. I've commented it out so that it will be easy to reinstate if we do need to monitor the process stats.

References

Risks

sarahjmiller commented 8 months ago

Thanks for the feedback, @grosser!

why is this even an issue when these have ~10 total series ?

In the metrics cost optimization request they mention both an improvement to ingest cost and cardinality cost, so even a small number of series might make a difference?

we should monitor periodicals taking too long because that means they are doing something funky (either consuming too many resources or stuck in a loop)

The request also says that these metrics aren't queried by any existing dashboards, monitors, or users. If Compute is monitoring these metrics, would you be willing to respond in the Slack thread with the rationale for keeping them?

grosser commented 8 months ago

ah nvm I read the code wrong ... this is just process list that would have big cardinality and be not generally useful 🎉