search

grafana / tempo-operator

Grafana Tempo Kubernetes operator
https://grafana.com/docs/tempo/latest/setup/operator/
GNU Affero General Public License v3.0
55 stars 28 forks source link

Fix clearing instance metrics when namespace is deleted #1024

Closed pavolloffay closed 2 weeks ago

pavolloffay commented 2 weeks ago

Related to #1019

The reconciliation loop was run after the deletion webohok causing the instance metric to be set as failed. 

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
running create or update
{"level":"error","ts":"2024-08-29T09:41:44.80454309Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"546c572e-a89f-4960-9e6f-6adaa4d6bdb2","objectName":"tempo-simplest","objectKind":"/, Kind=","error":"configmaps \"tempo-simplest\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/controllers/tempo.reconcileManagedObjects\n\t/workspace/controllers/tempo/common.go:84\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).createOrUpdate\n\t/workspace/controllers/tempo/tempostack_create_or_update.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:133\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
create or update error failed to create objects for simplest: configmaps "tempo-simplest" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated
{"level":"error","ts":"2024-08-29T09:41:44.884563417Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"546c572e-a89f-4960-9e6f-6adaa4d6bdb2","error":"failed to create objects for simplest: configmaps \"tempo-simplest\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.886623001Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a944d820-ecb1-45c4-83a5-f2cde9dfb99b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.908728513Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a944d820-ecb1-45c4-83a5-f2cde9dfb99b","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.911132343Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a9ed6fbb-d457-4d13-8288-c55b07089611","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.920784312Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"a9ed6fbb-d457-4d13-8288-c55b07089611","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.923598451Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"b5dac258-688f-468b-8ac7-8b0971b03996","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.9326483Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"b5dac258-688f-468b-8ac7-8b0971b03996","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:44.943632907Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"0e75583c-028c-46b3-9616-3c0235ba2cf2","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:44.956636985Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"0e75583c-028c-46b3-9616-3c0235ba2cf2","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.04003063Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"54110e40-726f-4473-a2a7-456335daad9f","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.054945442Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"54110e40-726f-4473-a2a7-456335daad9f","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.217640355Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"818703ff-42da-4c4a-90a2-dcfd1713fd7d","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.244609271Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"818703ff-42da-4c4a-90a2-dcfd1713fd7d","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.568455632Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"ce344116-39bb-4ecd-95c8-958107bbc14d","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.591105486Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"ce344116-39bb-4ecd-95c8-958107bbc14d","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.759866099Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"f659dad9-5b7d-48e9-a494-77d4f9d9e832","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:45.772825122Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"f659dad9-5b7d-48e9-a494-77d4f9d9e832","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:45.990902279Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"7445b22d-1f8c-4bbe-90ee-06de160559d7","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:46.013553615Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"7445b22d-1f8c-4bbe-90ee-06de160559d7","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:46.204762724Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"6676fb07-9628-453d-afb5-8749b4c4f1c0","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:46.221985526Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"6676fb07-9628-453d-afb5-8749b4c4f1c0","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
{"level":"error","ts":"2024-08-29T09:41:46.325294554Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-signing-ca","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-signing-ca\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.326617046Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.327672887Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ingester-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-ingester-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.330892036Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-querier-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-querier-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.33226402Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-query-frontend-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-query-frontend-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.333397086Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-compactor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-compactor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.335907255Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-gateway-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-gateway-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:46.336978796Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-distributor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-distributor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates
{"level":"error","ts":"2024-08-29T09:41:46.343896764Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"39a865e7-e584-4283-81d9-cbe9a5539c8b","error":"built in cert manager error: failed to create or rotate TempoStack certificates","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
instance is found, running reconciliation
running build in cert management
<--- delete webhook
{"level":"info","ts":"2024-08-29T09:41:46.945062798Z","logger":"certrotation-reconcile","msg":"Skipping reconciliation for unmanaged TempoStack resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"7de9981d-088e-4c45-97ec-d43b06778ddc","tempo":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"name":"chainsaw-loyal-drake/simplest"}
{"level":"error","ts":"2024-08-29T09:41:47.119842764Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-signing-ca","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-signing-ca\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.12123206Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ca-bundle","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"configmaps \"tempo-simplest-ca-bundle\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.122862429Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-ingester-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-ingester-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.125637264Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-querier-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-querier-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.128165588Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-query-frontend-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-query-frontend-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.13160468Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-compactor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-compactor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.133025777Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-gateway-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-gateway-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-08-29T09:41:47.140857109Z","logger":"tempostack-reconcile","msg":"failed to configure resource","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","tempostacks":"chainsaw-loyal-drake/simplest","event":"createOrRotateCerts","object_name":"tempo-simplest-distributor-mtls","object_kind":"&TypeMeta{Kind:,APIVersion:,}","error":"secrets \"tempo-simplest-distributor-mtls\" is forbidden: unable to create new content in namespace chainsaw-loyal-drake because it is being terminated","stacktrace":"github.com/grafana/tempo-operator/internal/certrotation/handlers.CreateOrRotateCertificates\n\t/workspace/internal/certrotation/handlers/rotate_certs.go:78\ngithub.com/grafana/tempo-operator/controllers/tempo.(*TempoStackReconciler).Reconcile\n\t/workspace/controllers/tempo/tempostack_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}
error creating or rotate certificates failed to create or rotate TempoStack certificates

-->Reconciling tempostack chainsaw-loyal-drake/simplest
not found error TempoStack.tempo.grafana.com "simplest" not found
{"level":"error","ts":"2024-08-29T09:41:47.158544812Z","msg":"Reconciler error","controller":"tempostack","controllerGroup":"tempo.grafana.com","controllerKind":"TempoStack","TempoStack":{"name":"simplest","namespace":"chainsaw-loyal-drake"},"namespace":"chainsaw-loyal-drake","name":"simplest","reconcileID":"cd3b8daf-83ff-49c4-8acd-4ef8359e7803","error":"tempostacks.tempo.grafana.com \"simplest\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"}

-->Reconciling tempostack chainsaw-loyal-drake/simplest
not found error TempoStack.tempo.grafana.com "simplest" not found

The alert TempoOperatorReconcileError is still firing, but that is expected. 

      - alert: TempoOperatorReconcileError
        annotations:
          message: Tempo Operator failed to reconcile.
          runbook_url: https://github.com/grafana/tempo-operator/blob/main/operations/runbook.md#TempoOperatorReconcileError
        expr: |
          increase(controller_runtime_reconcile_errors_total{service="tempo-operator-controller-manager-metrics-service"}[15m]) > 0
          and
          increase(controller_runtime_terminal_reconcile_errors_total{service="tempo-operator-controller-manager-metrics-service"}[15m]) == 0
        for: 15m
        labels:
          severity: warning
codecov-commenter commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.92%. Comparing base (bdf452d) to head (3df7293).

Files with missing lines Patch % Lines
controllers/tempo/tempomonolithic_controller.go 0.00% 1 Missing :warning:
controllers/tempo/tempostack_controller.go 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1024 +/- ## ========================================== - Coverage 72.94% 72.92% -0.03% ========================================== Files 106 106 Lines 6575 6577 +2 ========================================== Hits 4796 4796 - Misses 1489 1491 +2 Partials 290 290 ``` | [Flag](https://app.codecov.io/gh/grafana/tempo-operator/pull/1024/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grafana) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/grafana/tempo-operator/pull/1024/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grafana) | `72.92% <0.00%> (-0.03%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grafana#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

IshwarKanse commented 2 weeks ago

@pavolloffay Verified the fix with the steps from the jira issue. We are good to merge the PR.