Add support for custom Prometheus scrape jobs

[przemeklal]

Enhancement Proposal

It would be great if a user could manually add any scrape_jobs configuration to COS Prometheus using cos-configuration charm.

Historically, we relied on this config option https://charmhub.io/prometheus/configure#scrape-jobs in the LMA Prometheus charm to achieve that.

Use cases:

• scraping non-charmed Prometheus targets
• scraping exporters that don't support native COS relations (e.g. https://charmhub.io/prometheus-blackbox-peer-exporter)
• scraping external targets

[sed-i]

Thanks for the proposal @przemeklal. Is this specifically to be able to see scrape targets up/down for the multi-target use case? (instead of the blackbox exporter itself)

[przemeklal]

It's kind of both actually, we have the multi-target use case because on many clouds we rely on blackbox-exporter meshes deployed using dedicated subordinate machine charms. But any "advanced" use case where one needs to scrape multiple targets, do a lot of relabeling, etc. and do other things that https://github.com/canonical/prometheus-scrape-target-k8s-operator doesn't support.

It also feels a bit awkward that I must define custom alerting rules and dashboards for these "external" exporters here in the cos-config charm but I don't have any way to configure how I want to scrape them in the same charm.

[dstathis]

I think we should do this. It would unify the "external targets" story and be much less cumbersome than using prometheus-scape-target.