The Prometheus Charmed Operator for Juju provides a monitoring solution using Prometheus, which is an open-source monitoring system and alerting toolkit. Besides it handles instantiation, scaling, configuration, and Day 2 operations specific to Prometheus.
This Charm is also part of the Canonical Observability Stack and deploys only the monitoring component of Prometheus in a Kubernetes cluster. An alerting service for Prometheus is offered through a separate Charm.
Create a Juju model for your operator, say "observability"
juju add-model observability
The Prometheus Charmed Operator may now be deployed using the Juju command line as in
juju deploy prometheus-k8s --channel=stable
Progress of the Prometheus charm deployment and its current status may be viewed anytime using the Juju command line.
juju status --color --relations
Once the Prometheus charm deployments completes, you may expect to see a status result such as
$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
cos charm-dev-batteries microk8s/localhost 3.0.2 unsupported 14:27:58-03:00
App Version Status Scale Charm Channel Rev Address Exposed Message
prometheus-k8s 2.33.5 active 1 prometheus-k8s stable 79 10.152.183.227 no
Unit Workload Agent Address Ports Message
prometheus-k8s/0* active idle 10.1.36.90
Relation provider Requirer Interface Type Message
prometheus-k8s:prometheus-peers prometheus-k8s:prometheus-peers prometheus_peers peer
Prometheus provides a user interface that let us explore the data that has collected such as metrics and associated alerts.
This UI is accessed on port 9090 at the Prometheus charm's address. This assumes the Prometheus host address is accessible from the host your browser is running on. For example the Juju status message shown above is from a microk8s
cluster, hence navigating to http://10.152.183.227:9090
will show us the Prometheus UI.
When Prometheus is deployed it will only scrape metrics from itself. This is the default behavior of Prometheus. Additional metrics endpoints may be added to the Prometheus charm through relations with other charms. Currently the following charms are supported
prometheus_scrape
interface to provide a metrics endpoint and optionally alert rules for Prometheus.At present the Prometheus Charmed Operator for Kubernetes supports eight relations.
metrics-endpoint:
interface: prometheus_scrape
Charms may forward information about their metrics endpoints and associated alert rules to the Prometheus charm over the metrics-endpoint
relation using the prometheus_scrape
interface. In order for these metrics to be aggregated by this Prometheus charm all that is required is to relate the two charms as in:
juju relate kube-state-metrics-k8s prometheus-k8s
Charms that seek to provide metrics endpoints and alert rules for Prometheus must do so using the provided prometheus_scrape
charm library. This library by implementing the metrics-endpoint
relation, not only ensures that scrape jobs and alert rules are forward to Prometheus but also that these are updated any time the metrics provider charm is upgraded. For example new alert rules may be added or old ones removed by updating and releasing a new version of the metrics provider charm. While it is safe to update alert rules as desired, care must be taken when updating scrape job specifications as this has the potential to break the continuity of the scraped metrics time series. In particular changing the following keys in the scrape job can break time series continuity
job_name
relabel_configs
metric_relabel_configs
static_configs
Evaluation of alert rules forwarded through the prometheus_scrape
interface are automatically limited to the charmed application that provided these rules. This ensures that alert rule evaluation is scoped down to the charm providing the rules.
alertmanager:
interface: alertmanager_dispatch
The Alertmanager Charm aggregates, deduplicates, groups and routes alerts to selected "receivers". Alertmanager receives its alerts from Prometheus and this interaction is set up and configured using the alertmanager
relation through the alertmanager_dispatch
interface. Over this relation the Alertmanager charm keeps Prometheus informed of all Alertmanager instances (units) to which alerts must be forwarded. If your charm sets any alert rules then almost always it would need a relation with an Alertmanager charm which had been configured to forward alerts to specific receivers. In the absence of such a relation alerts even when raised will only be visible in the Prometheus user interface. A prudent approach to setting up an Observability stack is to do so in a manner such that it draws your attention to alarms as and when they are raised, without you having to periodically check a dashboard.
ingress:
interface: ingress_per_unit
limit: 1
Interactions with the Prometheus charm can not be assumed to originate within the same Juju model, let alone the same Kubernetes cluster, or even the same Juju cloud. Hence the Prometheus charm also supports an Ingress relation. There are multiple use cases that require an ingress, in particular
Prometheus typical needs a "per unit" Ingress. This per unit ingress is necessary since Prometheus exposes a remote write endpoint on a per unit basis. A per unit ingress relation is available in the traefik-k8s charm and this Prometheus charm does support that relation over ingress_per_unit
interface.
catalogue:
interface: catalogue
Through this relation, Prometheus provides its URL to Catalogue K8s Charmed Operator which is a landing page that helps users to locate the user interfaces of charms it relates to.
grafana-source:
interface: grafana_datasource
The Grafana Charm provides a data visualization solution for metrics aggregated by Prometheus and supports the creation of bespoke dashboards for such visualization. Grafana requires a data source for its dashboards and this Prometheus charm provides the data source through the grafana-source
relation using the grafana_datasource
interface. To visualize your charms metrics using Grafana the following steps are required
cassandra-k8s
) and Prometheus so that Prometheus can aggregate the metrics.For example
juju relate cassandra-k8s prometheus-k8s
juju relate prometheus-k8s grafana-k8s
juju relate cassandra-k8s grafana-k8s
receive-remote-write:
interface: prometheus_remote_write
Metrics may also be pushed to this Prometheus charm through the receive-remote-write
relation using the prometheus_remote_write
interface, which can be used with the Grafana Agent Charm to have metrics scraped by the Grafana Agent sent over to Prometheus.
self-metrics-endpoint:
interface: prometheus_scrape
This Prometheus charm may forward information about its metrics endpoint and associated alert rules to another Prometheus charm over the self-metrics-endpoint
relation using the prometheus_scrape
interface. In order for these metrics to be aggregated by the remote Prometheus charm all that is required is to relate the two charms as in:
juju relate \
prometheus-k8s:self-metrics-endpoint \
remote-prometheus-charm:metrics-endpoint
grafana-dashboard:
interface: grafana_dashboard
Over the grafana-dashboard
relation using the grafana-dashboard
interface, this Prometheus charm also provides meaningful dashboards about its metrics to be shown in a Grafana Charm .
In order to add these dashboards to Grafana all that is required is to relate the two charms in the following way:
juju relate \
prometheus-k8s:grafana-dashboard \
grafana-k8s:grafana-dashboard
This charm by default uses the latest version of the canonical/prometheus image.