search

canonical / tempo-k8s-operator

This charmed operator automates the operational procedures of running Grafana Tempo, an open-source tracing backend.
https://charmhub.io/tempo-k8s
Apache License 2.0
5 stars 3 forks source link

Upgrade Tempo to 2.4.0 with adjusting config properties/defaults #74

Closed mmkay closed 4 months ago

mmkay commented 4 months ago

Issue

Upgrade Tempo to the latest image (2.4.0).

Testing Instructions

Run charmcraft pack on the charm and run the following commands:

juju deploy cos-lite --trust --channel edge
juju deploy ./tempo-k8s_ubuntu-22.04-amd64.charm tempo --resource tempo-image=grafana/tempo:2.4.0
(once everything settles down)
jhack imatrix fill

Bundle:

bundle: kubernetes
applications:
  alertmanager:
    charm: alertmanager-k8s
    channel: edge
    revision: 103
    series: focal
    resources:
      alertmanager-image: 87
    scale: 1
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
    trust: true
  catalogue:
    charm: catalogue-k8s
    channel: edge
    revision: 33
    series: focal
    resources:
      catalogue-image: 32
    scale: 1
    options:
      description: "Canonical Observability Stack Lite, or COS Lite, is a light-weight,
        highly-integrated, \nJuju-based observability suite running on Kubernetes.\n"
      tagline: Model-driven Observability Stack deployed with a single command.
      title: Canonical Observability Stack
    constraints: arch=amd64
    trust: true
  grafana:
    charm: grafana-k8s
    channel: edge
    revision: 106
    series: focal
    resources:
      grafana-image: 67
      litestream-image: 43
    scale: 1
    constraints: arch=amd64
    storage:
      database: kubernetes,1,1024M
    trust: true
  loki:
    charm: loki-k8s
    channel: edge
    revision: 122
    series: focal
    resources:
      loki-image: 91
    scale: 1
    constraints: arch=amd64
    storage:
      active-index-directory: kubernetes,1,1024M
      loki-chunks: kubernetes,1,1024M
    trust: true
  prometheus:
    charm: prometheus-k8s
    channel: edge
    revision: 171
    series: focal
    resources:
      prometheus-image: 140
    scale: 1
    constraints: arch=amd64
    storage:
      database: kubernetes,1,1024M
    trust: true
  tempo:
    charm: local:tempo-k8s-0
    scale: 1
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
  traefik:
    charm: traefik-k8s
    channel: edge
    revision: 170
    series: focal
    resources:
      traefik-image: 158
    scale: 1
    constraints: arch=amd64
    storage:
      configurations: kubernetes,1,1024M
    trust: true
relations:
- - traefik:ingress-per-unit
  - prometheus:ingress
- - traefik:ingress-per-unit
  - loki:ingress
- - traefik:traefik-route
  - grafana:ingress
- - traefik:ingress
  - alertmanager:ingress
- - prometheus:alertmanager
  - alertmanager:alerting
- - grafana:grafana-source
  - prometheus:grafana-source
- - grafana:grafana-source
  - loki:grafana-source
- - grafana:grafana-source
  - alertmanager:grafana-source
- - loki:alertmanager
  - alertmanager:alerting
- - prometheus:metrics-endpoint
  - traefik:metrics-endpoint
- - prometheus:metrics-endpoint
  - alertmanager:self-metrics-endpoint
- - prometheus:metrics-endpoint
  - loki:metrics-endpoint
- - prometheus:metrics-endpoint
  - grafana:metrics-endpoint
- - grafana:grafana-dashboard
  - loki:grafana-dashboard
- - grafana:grafana-dashboard
  - prometheus:grafana-dashboard
- - grafana:grafana-dashboard
  - alertmanager:grafana-dashboard
- - catalogue:ingress
  - traefik:ingress
- - catalogue:catalogue
  - grafana:catalogue
- - catalogue:catalogue
  - prometheus:catalogue
- - catalogue:catalogue
  - alertmanager:catalogue
- - catalogue:catalogue
  - loki:catalogue
- - loki:logging
  - tempo:logging
- - loki:logging
  - traefik:logging
- - tempo:tracing
  - alertmanager:tracing
- - tempo:grafana-dashboard
  - grafana:grafana-dashboard
- - tempo:grafana-source
  - grafana:grafana-source
- - tempo:tracing
  - grafana:tracing
- - tempo:tracing
  - loki:tracing
- - tempo:metrics-endpoint
  - prometheus:metrics-endpoint
- - tempo:tracing
  - prometheus:tracing
- - tempo:tracing
  - traefik:tracing
- - traefik:grafana-dashboard
  - grafana:grafana-dashboard
- - traefik:ingress
  - tempo:ingress

Release Notes

Upgrade of Tempo workload to 2.4.0, adjusting configuration properties names and default values according to Tempo's changelogs.

PietroPasotti commented 4 months ago

After some testing with cos-lite plus tempo, I found out that the tracing databag was empty. after a good hour of confusion, frustration, life choice doubts and so on, it turns out it was an issue with the signature of the tracing .publish_receivers signature. It said: receivers: Iterable[RawReceiver] but the loop below was accessing receivers multiple times. As a result only the first relation got the receivers list, all other relations got no receivers.

Upgrading that type to receivers: Sequence[RawReceiver] and materializing the iterator in _update_tracing_relations fixed the issue.