loki ingress endpoint doesn't work

[simondeziel]

Bug Description

After deploying the COS Lite bundle (https://github.com/canonical/cos-lite-bundle), the Grafana dashboard datasources section looks like this:

The Alertmanager and Prometheus datasources successfully pass the Grafana checks but the Loki one fails due to receiving a 404 error.

To Reproduce

1. Follow the "Deploy the COS Lite bundle with overlays" section of https://charmhub.io/topics/canonical-observability-stack/install/microk8s
2. juju deploy cos-lite --trust --channel=edge --overlay ./offers-overlay.yaml --overlay ./storage-small-overlay.yaml

Environment

1. juju status

   
   ubuntu@microk8s:~$ juju status --relations
   Model  Controller  Cloud/Region         Version  SLA          Timestamp
   cos    overlord    k8s-cloud/localhost  2.9.34   unsupported  15:41:05Z

App Version Status Scale Charm Channel Rev Address Exposed Message alertmanager 0.23.0 active 1 alertmanager-k8s edge 32 10.152.183.115 no
grafana 8.2.6 active 1 grafana-k8s edge 44 10.152.183.212 no
loki 2.4.1 active 1 loki-k8s edge 45 10.152.183.113 no
prometheus 2.33.5 active 1 prometheus-k8s edge 71 10.152.183.116 no
traefik active 1 traefik-k8s edge 88 10.76.66.104 no

Unit Workload Agent Address Ports Message alertmanager/0 active idle 10.1.128.202
grafana/0 active idle 10.1.128.251
loki/0 active idle 10.1.128.252
prometheus/0 active idle 10.1.128.200
traefik/0* active idle 10.1.128.249

Offer Application Charm Rev Connected Endpoint Interface Role alertmanager-karma-dashboard alertmanager alertmanager-k8s 32 0/0 karma-dashboard karma_dashboard provider grafana-dashboards grafana grafana-k8s 44 0/0 grafana-dashboard grafana_dashboard requirer loki-logging loki loki-k8s 45 0/0 logging loki_push_api provider prometheus-receive-remote-write prometheus prometheus-k8s 71 0/0 receive-remote-write prometheus_remote_write provider prometheus-scrape prometheus prometheus-k8s 71 0/0 metrics-endpoint prometheus_scrape requirer

Relation provider Requirer Interface Type Message alertmanager:alerting loki:alertmanager alertmanager_dispatch regular
alertmanager:alerting prometheus:alertmanager alertmanager_dispatch regular
alertmanager:grafana-dashboard grafana:grafana-dashboard grafana_dashboard regular
alertmanager:grafana-source grafana:grafana-source grafana_datasource regular
alertmanager:replicas alertmanager:replicas alertmanager_replica peer
alertmanager:self-metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular
grafana:grafana grafana:grafana grafana_peers peer
loki:grafana-dashboard grafana:grafana-dashboard grafana_dashboard regular
loki:grafana-source grafana:grafana-source grafana_datasource regular
loki:metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular
prometheus:grafana-dashboard grafana:grafana-dashboard grafana_dashboard regular
prometheus:grafana-source grafana:grafana-source grafana_datasource regular
prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer
traefik:ingress-per-unit loki:ingress ingress_per_unit regular
traefik:ingress-per-unit prometheus:ingress ingress_per_unit regular
traefik:metrics-endpoint prometheus:metrics-endpoint prometheus_scrape regular

2. `snap list`
```ubuntu@microk8s:~$ snap list
Name        Version      Rev    Tracking       Publisher   Notes
charmcraft  2.0.0        1033   latest/stable  canonical✓  classic
core18      20220831     2566   latest/stable  canonical✓  base
core20      20220826     1623   latest/stable  canonical✓  base
juju        2.9.34       20510  2.9/stable     canonical✓  classic
lxd         5.6-794016a  23680  latest/stable  canonical✓  -
microk8s    v1.25.0      3883   1.25/stable    canonical✓  classic
snapd       2.57.2       17029  latest/stable  canonical✓  snapd

Relevant log output

`juju debug-log --replay`: https://termbin.com/wndl

Additional context

No response

[simskij]

I can confirm this is the case. It seems like the webroot isn't properly set as removing the relation between loki and traefik and refreshing the source turns it green. Adding it back again and refreshing the datasources, however, it continues to work, so this might very well be connected to https://github.com/canonical/traefik-k8s-operator/issues/78, as that usually manifests by deploying it as a bundle.

[dstathis]

When I deploy the bundle this does not happen for me. Can anyone confirm that this bug still exists?

[dstathis]

When I deploy the bundle this does not happen for me. Can anyone confirm that this bug still exists?

@simskij

[simskij]

I can't reproduce it either, so seems like the fixes to traefik solved it.