Data collection transparency

[QuentinBisson]

Motivation

Our Observability Platform uses a lot of resources. Those resources are needed to manage the big amount of data that we have to handle. And that data comes from sources that we do not control, but our internal and public customers control. So we need to make them aware of the resource usage and create a way on how we can make resource cost per source of data transparent.

Todo

• [ ] Create a monitoring dashboard that shows the resource usage per source of data
  • This source of data can be the service monitors for now
  • Do we have resource usage issues with logging right now?
• [ ] Enrich the documentation on how to monitor and app with a passage about resource cost and the dashboard:
  • see https://intranet.giantswarm.io/docs/monitoring/monitoring_of_apps/
  • https://intranet.giantswarm.io/docs/monitoring/monitoring-architecture/
  • https://intranet.giantswarm.io/docs/monitoring/monitoring-components-access/

Ideas:

• Monitoring: How to detect churn
• Monitoring: How to track down the resource usage per jobs https://github.com/giantswarm/giantswarm/issues/30727

Outcome

• We have a dashboard with which we can make transparent which source of data leads to high resource usage of our observability platform

[hervenicol]

We now have dashboards to investigate prometheus-agent targets: https://github.com/giantswarm/giantswarm/issues/30727. Does it cover the first part of the issue?

[hervenicol]

It's missing per-agent-pod metrics usage. I'll add it then I guess we can close this issue.

[hervenicol]

ServiceMonitors overview dashboard has been updated. I guess that's OK for now, and we can close this issue.