search

suxess-it / sx-cnp-oss

1 stars 0 forks source link

[loki] GrafanaAgent stuck in pending deletion #293

Open jkleinlercher opened 3 days ago

jkleinlercher commented 3 days ago

Shutdown-test Job in github action https://github.com/suxess-it/sx-cnp-oss/actions/runs/9785529844/job/27018726355 often shows loki stuck in deletion. Root cause is GrafanaAgent:

Group: monitoring.grafana.com
      health:
        message: Pending deletion
        status: Progressing
      kind: GrafanaAgent
      name: sx-loki
      namespace: monitoring
      status: Synced
      version: v1alpha1

one possible hint could be https://github.com/grafana/agent/issues/4177

jkleinlercher commented 3 days ago

I removed the shutdown test with https://github.com/suxess-it/sx-cnp-oss/commit/ffc3cdc90d66dc2146c1235804eabba5daf882ee because otherwise it always fails. I will try to recreate the problem locally and maybe we can fix it. On the other hand GrafanaAgent in loki is only needed for Loki self-monitoring and GrafanaAgent is deprecated. Maybe in the future they switch to alloy according to https://github.com/grafana/loki/issues/12627