sustainable-computing-io / kepler-operator

Kepler Operator
Apache License 2.0
25 stars 26 forks source link

Kepler Instance gets timeout at time of deletion #271

Closed vprashar2929 closed 11 months ago

vprashar2929 commented 11 months ago

At the time of deleting the Kepler Instance, it usually takes time for deletion to actually happen. By looking at the controller manager logs looks like there is a timeout when deleting configmap for dashboards.

Attached below are logs for reference:

2023-10-06T08:22:48Z    LEVEL(-5)       Reconciling     {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler
", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138"}
2023-10-06T08:22:48Z    INFO    Start of  reconcile     {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler
", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138"}
2023-10-06T08:23:23Z    LEVEL(-3)       continue reconciliation despite error   {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138", "error": "power-monitoring-overview (/v1, Kind=ConfigMap): deleter: timed out waiting for deletion : timed out waiting for the condition"}
2023-10-06T08:23:53Z    LEVEL(-3)       continue reconciliation despite error   {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138", "error": "power-monitoring-by-ns (/v1, Kind=ConfigMap): deleter: timed out waiting for deletion : timed out waiting for the condition"}
2023-10-06T08:24:03Z    LEVEL(-3)       finalizer state {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138", "reconciler": "finalizer", "deleted": true, "finalizer": true}
2023-10-06T08:24:03Z    LEVEL(-3)       removing finalizer      {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138", "reconciler": "finalizer"}
2023-10-06T08:24:03Z    LEVEL(-3)       stopping further reconciliation as requested    {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138"}
2023-10-06T08:24:03Z    LEVEL(-3)       kepler could not be found; may be marked for deletion   {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138"}
2023-10-06T08:24:03Z    INFO    End of reconcile        {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138"}
2023-10-06T08:24:03Z    ERROR   Reconciler error        {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "514af8c3-7d06-4dde-ac80-daffd24c5138", "error": "power-monitoring-by-ns (/v1, Kind=ConfigMap): deleter: timed out waiting for deletion : timed out waiting for the condition"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227
2023-10-06T08:24:03Z    LEVEL(-5)       Reconciling     {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "dcd5b5b8-bb3e-4e36-978c-876d0548932a"}
2023-10-06T08:24:03Z    INFO    Start of  reconcile     {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "dcd5b5b8-bb3e-4e36-978c-876d0548932a"}
2023-10-06T08:24:03Z    LEVEL(-3)       kepler could not be found; may be marked for deletion   {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "dcd5b5b8-bb3e-4e36-978c-876d0548932a"}
2023-10-06T08:24:03Z    INFO    End of reconcile        {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "dcd5b5b8-bb3e-4e36-978c-876d0548932a"}
2023-10-06T08:24:03Z    LEVEL(-5)       Reconcile successful    {"controller": "kepler", "controllerGroup": "kepler.system.sustainable.computing.io", "controllerKind": "Kepler", "Kepler": {"name":"kepler"}, "namespace": "", "name": "kepler", "reconcileID": "dcd5b5b8-bb3e-4e36-978c-876d0548932a"}

However, after some time, Kepler instance along with related resources gets deleted