search

canonical / grafana-k8s-operator

This charmed operator automates the operational procedures of running Grafana, an open-source visualization toolkit, on Kubernetes.
https://charmhub.io/grafana-k8s
Apache License 2.0
6 stars 23 forks source link

`failed to verify` Let's Encrypt certificate #307

Closed mthaddon closed 4 months ago

mthaddon commented 4 months ago

Bug Description

We're getting the following on a loki dashboard in grafana after having enabled certificates (Let's Encrypt):

Get "https://${hostname}/${model-name}-loki-0/loki/api/v1/query_range?direction=backward&end=1709726886797000000&limit=1000&query=%7Bjuju_charm%3D%22django-k8s%22%2C+juju_application%3D~%22.%2A%22%2C+juju_model%3D~%22prod-httprequest-lego-provider-k8s%22%2C+juju_model_uuid%3D~%22.%2A%22%2C+juju_unit%3D~%22.%2A%22%7D+%7C%3D+%22%22&start=1709726586797000000&step=100ms": x509: certificate signed by unknown authority

To Reproduce

  1. Deploy COS
  2. Add Let's Encrypt certificates to Traefik-k8s
  3. Bring up a dashboard in Grafana that includes data from Loki.

Environment

Here's the versions of charms we're currently running:

alertmanager          0.26.0                          active      1  alertmanager-k8s      latest/stable      96  10.152.183.48   no       
catalogue                                             active      1  catalogue-k8s         latest/stable      31  10.152.183.38   no       
cos-ingress           2.10.4                          active      1  traefik-k8s           latest/stable     166  10.141.53.97    yes      
grafana               9.2.1                           active      1  grafana-k8s           latest/stable      92  10.152.183.46   no       
httprequest-lego-k8s                                  active      1  httprequest-lego-k8s  stable             40  10.152.183.126  no       
karma                 0.114                           active      1  karma-k8s             latest/candidate   20  10.152.183.245  no       
loki                  2.7.4                           active      1  loki-k8s              latest/stable     105  10.152.183.231  no       
metallb-controller    res:metallb-controller-imag...  active      1  metallb-controller    latest/stable      46  10.152.183.58   no       
metallb-speaker       res:metallb-speaker-image@6...  active      6  metallb-speaker       latest/stable      41  10.152.183.136  no       
prometheus            2.47.2                          active      1  prometheus-k8s        latest/stable     156  10.152.183.188  no

Relevant log output

Get "https://${hostname}/${model-name}-loki-0/loki/api/v1/query_range?direction=backward&end=1709726886797000000&limit=1000&query=%7Bjuju_charm%3D%22django-k8s%22%2C+juju_application%3D~%22.%2A%22%2C+juju_model%3D~%22prod-httprequest-lego-provider-k8s%22%2C+juju_model_uuid%3D~%22.%2A%22%2C+juju_unit%3D~%22.%2A%22%7D+%7C%3D+%22%22&start=1709726586797000000&step=100ms": x509: certificate signed by unknown authority

Additional context

This is similar to https://github.com/canonical/karma-k8s-operator/issues/41.

Abuelodelanada commented 4 months ago

Hi @mthaddon

Is this the only extra relation you have added??

httprequest-lego-k8s <-- tls-certificates --> traefik

mcarvalhor commented 4 months ago

Hi @Abuelodelanada ,

This is the timeline:

  1. Previously we were using manual-tls-certificates, and related it to traefik via tls-certificates interface.
  2. We removed the manual-tls-certificates application (and thus its relation to traefik).
  3. Added httprequest-lego-k8s and related it to traefik via tls-certificates interface.
  4. Deleted the traefik pod to let it be recreated and request a certificate again.
  5. No issues seen on logs or when using COS on Chrome/latest.
lucabello commented 4 months ago

The rock for Grafana has been fixed to include the base certificates in /etc/ssl/certs, which are normally bundled with the ca-certificates package.

The charms have been updated to use that image (they're using a newer resource but they are the same revision).

If you deploy grafana-k8s now, and you juju ssh --container grafana grafana/0, you can see:

root@grafana-0:/# ll /etc/ssl/certs/
total 604

Closing!