Observability addon configuration

[Azbesciak]

Hi, I have just installed the observability addon. I have searched through the deployments and config maps, and tried to find the way out, but failed, so let me ask - how to supply custom Prometheus (alert.rules, prometheus.yml) and alertmanager rules?

Also please notice that this addon is not mentioned on https://microk8s.io/docs/addons (there is a Prometheus addon [obsolete], but url is 404 anyway).

Just wonder if passing own kube-prometheus repo deployment would fix it - just to override, but it is a little overkill IMO, and maybe something microk8s may be overridden?

Thanks.

[Azbesciak]

Hello :)? Anything on this?

[Azbesciak]

Hellooooo :)

[kamontat]

Yes can use microk8s enable observability -f values.yaml command to deploy your own custom config. For values.yaml file, you can read more detail on this.

I assume you familiar with helm chart and values file.

[Azbesciak]

@kamontat Ok, thanks but for example, I want to provide custom values for grafana - I suppose it would be in custom.ini. Is it somehow possible in other way that during the runtime update the config map? Can I somehow persist the configuration from grafana, to recreate it on other machine when I will make a new microk8s installation?

BTW maybe it is possible to provide some env variables in that values.yml?

BTW if any would wonder, values for grafana, etc, needs to be inside its section, including grafana.ini, so it will be flattened grafana.grafana.ini

[amartincolby]

I am also wandering by wondering about the current state of observability and best practices around it. What are the current plans because I recently lost my Grafana integration with Linkerd and want to make sure that any methods I implement to solve this don't get broken in the future.

[amartincolby]

I ended up abandoning the addon and just installing its constituent parts myself. Ultimately it ended up being better for me.

[anastazya]

I ended up abandoning the addon and just installing its constituent parts myself. Ultimately it ended up being better for me.

Yep, cause without documentation there's no point.

[davidwincent]

Yes can use microk8s enable observability -f values.yaml command to deploy your own custom config. For values.yaml file, you can read more detail on this.

I assume you familiar with helm chart and values file.

This did not work for me:

❯ microk8s enable observability -f helm/values.yaml
Infer repository core for addon observability
Addon core/dns is already enabled
Addon core/helm3 is already enabled
Addon core/hostpath-storage is already enabled
Enabling observability
Unknown option -f

[anastazya]

Yes can use microk8s enable observability -f values.yaml command to deploy your own custom config. For values.yaml file, you can read more detail on this. I assume you familiar with helm chart and values file.

This did not work for me:

❯ microk8s enable observability -f helm/values.yaml
Infer repository core for addon observability
Addon core/dns is already enabled
Addon core/helm3 is already enabled
Addon core/hostpath-storage is already enabled
Enabling observability
Unknown option -f

Also i DO NOT want 'core/hostpath-storage' addon as i have longhorn distributed storage or other better options. There's no way to do anything useful with this, as it's presented right now.

[DuyAnhTran95]

Yes can use microk8s enable observability -f values.yaml command to deploy your own custom config. For values.yaml file, you can read more detail on this. I assume you familiar with helm chart and values file.

This did not work for me:

❯ microk8s enable observability -f helm/values.yaml
Infer repository core for addon observability
Addon core/dns is already enabled
Addon core/helm3 is already enabled
Addon core/hostpath-storage is already enabled
Enabling observability
Unknown option -f

A quick look at the source code shows me that the correct flag would be --kube-prometheus-stack-values. I haven't done any test on that though.

[danielabbatt]

Yes can use microk8s enable observability -f values.yaml command to deploy your own custom config. For values.yaml file, you can read more detail on this. I assume you familiar with helm chart and values file.

This did not work for me:

❯ microk8s enable observability -f helm/values.yaml
Infer repository core for addon observability
Addon core/dns is already enabled
Addon core/helm3 is already enabled
Addon core/hostpath-storage is already enabled
Enabling observability
Unknown option -f

A quick look at the source code shows me that the correct flag would be --kube-prometheus-stack-values. I haven't done any test on that though.

Yes this is correct, I had to update my own usage to do this, if you look at the source code, as @DuyAnhTran95 says, you can see there are multiple options now for providing different values files.

[rodrigo-galba]

Hi there. To pass custom values to the observability addon, use:

microk8s enable observability --kube-prometheus-stack-values=values.yml

Later, to get all values from the observability release created by Helm, use:

helm get values -n observability kube-prom-stack > prom-values.yaml

[josedgm]

Later, to get all values from the observability release created by Helm, use:

And what if you later want to change a value? Do you have to remove and install the addon again while pointing to the updated yaml file?

[neoaggelos]

@josedgm An alternative would be to update the helm installation directly. The command should look like this:

helm upgrade -n observability kube-prom-stack kube-prometheus-stack \
  --repo https://prometheus-community.github.io/helm-charts \
  --reuse-values -f new_values.yaml

Keep in mind that the intent of the addon is to simply make it easy to deploy the stack, not handle lifecycle and further operations.

[filip-franek]

On top of what was mentioned above how about patching the current deployment?

Here is an example of how I patched my Grafana deployment to mount the Grafana data location to my persistent volume. I have it applied over Flux or it could be applied over kubectl apply -f grafana.yml. It assumes the Grafana Persistent Volume Claim (PVC) is already created.

# grafana.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-prom-stack-grafana
  namespace: observability
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: kube-prom-stack
      app.kubernetes.io/name: grafana
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: kube-prom-stack
        app.kubernetes.io/name: grafana
    spec:
      containers:
      - name: grafana
        volumeMounts:
        - mountPath: "/var/lib/grafana"
          name: <storage-class>
      volumes:
      - name: <storage-class>
        persistentVolumeClaim:
          claimName: <grafana-pvc>

There is also a question posted on kube-prometheus discussion board regarding how to integrate the Grafana Operator in the stack which would also be another way how to manage Grafana. The Prometheus Operator is in the kube-prometheus stack already unlike the Grafana Operator to manage Prometheus.

Here is an example of how to change the Prometheus configuration for scraping metrics and its storage with the Prometheus Operator. It also assumes the Prometheus Persistent Volume Claim (PVC) is already created.

# prometheus-cr-patch.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: kube-prom-stack-kube-prome-prometheus
  namespace: observability
spec:
  scrapeInterval: "120s"
  evaluationInterval: "120s" 
  retention: "15d"
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: <storage-class>
        resources:
          requests:
            storage: 15Gi

[fiedlerNr9]

Hi there. To pass custom values to the observability addon, use:

microk8s enable observability --kube-prometheus-stack-values=values.yml

Later, to get all values from the observability release created by Helm, use:

helm get values -n observability kube-prom-stack > prom-values.yaml

I tried it that way and it was not working as expected. I think they correct way of passing the values file is happening like this:

microk8s enable observability --kube-prometheus-stack-values values.yml --kube-prometheus-stack-version 55.8.1

[tonyskulk]

Using the link to the source code I could figure out about the underlying helm charts in use.

My intention is to add persistence. I could find the configuration for setting up persistence for prometheus, alertmanager and tempo, but for loki it seems that it is not supported by the helm chart in use loki-stack.

How can I add persistence to the loki-stack helm chart without manually manipulating the k8s resource files afterwards?

[IonBoleac]

@tonyskulk I recommend to use InfluxDB to save all data. I supposed to add this layer is not too difficult. What do you think?

[amartincolby]

Is this add-on no longer going to be maintained? It is in this repo but is no longer included in the official docs.

[tonyskulk]

It definitely lacks of documentation. For my solution I finally got it to work and took the following steps to make the grafana, tempo, loki, prometheus, alertmanager stack work with persistence. For loki and temp I needed to delete the statefulsets before beeing able to upgrade the helm installation:

--> lookup helmchart versions and values.yml files https://github.com/canonical/microk8s-core-addons/blob/main/addons/observability/enable --> configure the helm values files

microk8s enable observability
helm upgrade -n observability kube-prom-stack kube-prometheus-stack --repo https://prometheus-community.github.io/helm-charts --reuse-values -f kube-prometheus-stack-values.yml --version 45.5.0
kubectl delete statefulset -n observability loki --cascade=false
helm upgrade -n observability loki loki-stack --repo https://grafana.github.io/helm-charts --reuse-values -f loki-stack-values.yml --version 2.9.9
kubectl delete statefulset -n observability tempo --cascade=false
helm upgrade -n observability tempo tempo --repo https://grafana.github.io/helm-charts --reuse-values -f tempo-values.yml --version 1.0.0
--> delete/restart loki and tempo pods to fix persistent volume claims

[codespearhead]

@amartincolby it's probably the other way around:

root@ubuntu-s-4vcpu-8gb-nyc1-01:~# microk8s version

MicroK8s v1.29.2 revision 6641

root@ubuntu-s-4vcpu-8gb-nyc1-01:~# microk8s enable prometheus

Infer repository core for addon prometheus
DEPRECATION WARNING: 'prometheus' is deprecated and will soon be removed. Please use 'observability' instead.

Infer repository core for addon observability
Addon core/observability is already enabled

[benwah92]

Has anyone deployed their own kube-prometheus stack into MicroK8s other than using this addon? I also require persistence. Thinking about switching to k3s otherwise.