Upgrade required from grafana in observability - CVE-2023-3128

[ISAF87]

Summary

We are running a microk8s HA cluster on version v1.28.2 rev 5960 We enabled dashboarding using observability.

This uses grafana v 9.3.8

However, our security scans return several critical issues with this version: CVE-2023-3128

According to our tooling, it should be resolved in version 9.3.16

But if I check the referred link downloads A lot of later versions (minor and major) are present. At the time of writing up to 10.1.4

I tried updating using the sudo snap refresh, but according to snap everything is up to date.

Truth be told, I am relatively new with kubernetes/microk8s. I could install it using the suggested steps on the download link, but as I don't know the impact / workings in combination with the observability addon. I am kind of hesitant to do so.

[neoaggelos]

Hi @ISAF87, thank you for raising this.

The microk8s observability addon currently installs the kube-prom-stack chart version 45.5.0, which is a bit old.

To resolve this, I expect that installing a newer kube-prom-stack version helps. e should be able to bump the chart version (similar for tempo and loki). This is possible through a command-line argument.

Can you check whether enabling the addon as shown below helps:

microk8s enable observability --kube-prometheus-stack-version=51.8.1

From our side, I believe we can be more attentive for the core addons and keep the default version up to date.

Thanks!

[ISAF87]

I executed the following:

microk8s enable observability --kube-prom-stack-version=51.8.1 Output: Infer repository core for addon observability Addon core/observability is already enabled

Also tried to disable the addon, and re enable it using above command.

When I check the deployment for kube-prom-stack-grafana:

Name: kube-prom-stack-grafana Namespace: observability CreationTimestamp: Mon, 16 Oct 2023 15:28:02 +0200 Labels: app.kubernetes.io/instance=kube-prom-stack app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=grafana app.kubernetes.io/version=9.3.8 helm.sh/chart=grafana-6.51.2 Annotations: deployment.kubernetes.io/revision: 1 meta.helm.sh/release-name: kube-prom-stack meta.helm.sh/release-namespace: observability Selector: app.kubernetes.io/instance=kube-prom-stack,app.kubernetes.io/name=grafana Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app.kubernetes.io/instance=kube-prom-stack app.kubernetes.io/name=grafana Annotations: checksum/config: 984721f4a9edbcbb3089100a378a2f1bdcb05f0206d388fe85af46a4c6fea8bf checksum/dashboards-json-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b checksum/sc-dashboard-provider-config: 1bedc881a103b6af0b5a9282f2085a08440edbf1412796a59b6fb4b546b4a98d checksum/secret: 3494940d8e928be51c06b551afc839a94c2bd812a4b46487316cd9380ff9b7f9 Service Account: kube-prom-stack-grafana Containers: grafana-sc-dashboard: Image: quay.io/kiwigrid/k8s-sidecar:1.22.0 Port: Host Port: Environment: METHOD: WATCH LABEL: grafana_dashboard LABEL_VALUE: 1 FOLDER: /tmp/dashboards RESOURCE: both

It is still the outdated version. If I check the helm repos using microk8s helm search repo grafana it does list: prometheus-community/kube-prometheus-stack 51.2.0 v0.68.0 kube-prometheus-stack collects Kubernetes manif...

Which I could update to 51.8.1, but I am not sure it is related to the observability addon. (it could be from a previous setup for grafana which we all cleaned before enabling observability)

Any advise?

[ISAF87]

plugging: #3678 would certainly help for scenarios like this.

[neoaggelos]

Okay, apologies for the typo in the argument. Please try the following on a fresh cluster:

$ microk8s enable observability --kube-prometheus-stack-version=51.8.1

This gives me grafana version 10.1.5:

$ microk8s kubectl describe -n observability deploy/kube-prom-stack-grafana  | grep Image | grep grafana
    Image:           docker.io/grafana/grafana:10.1.5

can you check if it helps? I have also edited my previous comment for future onlookers

[ISAF87]

Thanks for the update.

I don't have a fresh cluster, so I took the following steps:

1. Check the current version: microk8s kubectl describe -n observability deploy/kube-prom-stack-grafana | grep Image | grep grafana which outputs: Image: grafana/grafana:9.3.8

2. As I don't have a fresh cluster, I proceeded by disabling the observability addon: $ microk8s disable observability

Output: Infer repository core for addon observability Disabling observability service "kube-prom-stack-kube-prome-kubelet" deleted

3. Next enable the observability addon again using the command suggested:

$ microk8s enable observability --kube-prometheus-stack-version=51.8.1 Which outputs:

Infer repository core for addon observability Addon core/dns is already enabled Addon core/helm3 is already enabled Addon core/hostpath-storage is already enabled Enabling observability "prometheus-community" already exists with the same configuration, skipping "grafana" already exists with the same configuration, skipping Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "grafana" chart repository ...Successfully got an update from the "prometheus-community" chart repository Update Complete. ⎈Happy Helming!⎈ Release "kube-prom-stack" does not exist. Installing it now. NAME: kube-prom-stack LAST DEPLOYED: Tue Oct 17 08:35:18 2023 NAMESPACE: observability STATUS: deployed REVISION: 1 NOTES: kube-prometheus-stack has been installed. Check its status by running: kubectl --namespace observability get pods -l "release=kube-prom-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator. Release "loki" does not exist. Installing it now. NAME: loki LAST DEPLOYED: Tue Oct 17 08:35:50 2023 NAMESPACE: observability STATUS: deployed REVISION: 1 NOTES: The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.

See http://docs.grafana.org/features/datasources/loki/ for more detail. Release "tempo" does not exist. Installing it now. NAME: tempo LAST DEPLOYED: Tue Oct 17 08:35:52 2023 NAMESPACE: observability STATUS: deployed REVISION: 1 TEST SUITE: None

4. Check if update succeeded: $ microk8s kubectl describe -n observability deploy/kube-prom-stack-grafana | grep Image | grep grafana Image: grafana/grafana:9.3.8

Still on the same version

On a side note: in a production environment disabling the addon re enabling it might not be the best update path. Is there a way to push new versions as its running? In this case it might be less critical, but would be annoying to setup dashboards / monitoring/alerting again etc.

[neoaggelos]

On a side note: in a production environment disabling the addon re enabling it might not be the best update path. Is there a way to push new versions as its running? In this case it might be less critical, but would be annoying to setup dashboards / monitoring/alerting again etc.

This is true, and thanks for putting a +1 in #3678, it helps us more than you might imagine with prioritization.

In this particular case, the observability addon deploys a helm chart, so you should be able to do something like this:

microk8s helm repo add https://prometheus-community.github.io/helm-charts prometheus
microk8s helm repo update
microk8s helm upgrade -n observability kube-prom-stack prometheus/kube-prom-stack --version 51.8.1 --install --reuse-values

[ISAF87]

Thank you for the quick response!

Did some checks based on your suggestion:

1. microk8s helm repo list => Shows prometheus-community helm repo was already present.
2. microk8s helm repo update => indicated it retrieved updates from the chart repo
3. microk8s helm search repo prometheus-community (as that is the repo name in our case) listed: ... prometheus-community/kube-prometheus-stack 51.9.0 v0.68.0 kube-prometheus-stack collects Kubernetes manif... ...
4. microk8s helm upgrade -n observability kube-prom-stack prometheus-community/kube-prometheus-stack --version 51.9.0 --install --reuse-values to upgrade to version 51.9.0, as that is present in our repo (51.8.1 couldnt be found) The output: Error: UPGRADE FAILED: template: kube-prometheus-stack/templates/prometheus/rules-1.14/windows.pod.rules.yaml:7:140: executing "kube-prometheus-stack/templates/prometheus/rules-1.14/windows.pod.rules.yaml" at <.Values.windowsMonitoring.enabled>: nil pointer evaluating interface {}.enabled

Edit: As it is related to the prometheus helm chart. Also logged an issue there: prometheus-community issue: 3908

[ISAF87]

Been digigng a bit in the source of windows.pod.rules..yaml

It contains these lines:

{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
{{- if and (semverCompare ">=1.14.0-0" $kubeTargetVersion) (semverCompare "<9.9.9-9" $kubeTargetVersion) .Values.defaultRules.create .Values.windowsMonitoring.enabled .Values.defaultRules.rules.windows }}

I suspect the values file use by the --reuse-values argument doesn't contain the windowsMonitoring value.

• Where is the values file located / can I check that somewhere?
• What would the impact be if I try to run it without the --reuse-value argument?

[ISAF87]

Decided to bite the bullet and run it without the --reuse-values argument Seems to be working, although I did get a couple of warnings

Did get a couple of warnings: microk8s helm upgrade -n observability kube-prom-stack prometheus-community/kube-prometheus-stack --version 51.9.0 --install W1019 08:25:17.774450 171499 warnings.go:70] unknown field "spec.scrapeConfigNamespaceSelector" W1019 08:25:17.776643 171499 warnings.go:70] unknown field "spec.scrapeConfigSelector" Release "kube-prom-stack" has been upgraded. Happy Helming! NAME: kube-prom-stack LAST DEPLOYED: Thu Oct 19 08:24:59 2023 NAMESPACE: observability STATUS: deployed REVISION: 2 NOTES: kube-prometheus-stack has been installed. Check its status by running: kubectl --namespace observability get pods -l "release=kube-prom-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

Update: In the prometheus issue it was suggested to also update the CRD's for these warnings. However, those are controlled by the helm chart.

What would the go to process be for the addon version updates? For now it seems to be resolved. The dashboard is running on 10.1.5 and still awaiting security scan results

[ISAF87]

The vulnerability was no longer listed in the security scan so everything seems fine now. Are actions required to ensure the used charts are updated with the microk8s release?

[neoaggelos]

Hi @ISAF87, indeed, we could update the chart version. Happy to accept PRs!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.