Add support for custom Prometheus scrape jobs

canonical / cos-configuration-k8s-operator

This charmed operator for Kubernetes enables you to provide configurations to various components of the Canonical Observability Stack (COS) bundle.

https://charmhub.io/cos-configuration-k8s

Apache License 2.0

4 stars 5 forks source link

Add support for custom Prometheus scrape jobs #86

Open przemeklal opened 3 months ago

przemeklal commented 3 months ago

Enhancement Proposal

It would be great if a user could manually add any scrape_jobs configuration to COS Prometheus using cos-configuration charm.

Historically, we relied on this config option https://charmhub.io/prometheus/configure#scrape-jobs in the LMA Prometheus charm to achieve that.

Use cases:

scraping non-charmed Prometheus targets
scraping exporters that don't support native COS relations (e.g. https://charmhub.io/prometheus-blackbox-peer-exporter)
scraping external targets

sed-i commented 3 months ago

Thanks for the proposal @przemeklal. Is this specifically to be able to see scrape targets up/down for the multi-target use case? (instead of the blackbox exporter itself)

przemeklal commented 3 months ago

It's kind of both actually, we have the multi-target use case because on many clouds we rely on blackbox-exporter meshes deployed using dedicated subordinate machine charms. But any "advanced" use case where one needs to scrape multiple targets, do a lot of relabeling, etc. and do other things that https://github.com/canonical/prometheus-scrape-target-k8s-operator doesn't support.

It also feels a bit awkward that I must define custom alerting rules and dashboards for these "external" exporters here in the cos-config charm but I don't have any way to configure how I want to scrape them in the same charm.