search

newrelic / newrelic-kubernetes-operator

Operator to create New Relic configuration in Kubernetes
Apache License 2.0
43 stars 21 forks source link

Deleting an AlertPolicy doesn't remove the AlertsApmCondition #106

Open AdrienKuhn opened 4 years ago

AdrienKuhn commented 4 years ago

Description

When creating an AlertPolicy containing an AlertsApmCondition, removing the policy will not remove the AlertsApmCondition from Kubernetes resources and the operator will be in an infinite error loop.

Current behavior

The AlertsApmCondition is not removed from Kubernetes cluster after removing an AlertPolicy, even if the condition doesn't exist on NewRelic anymore.

Expected behavior

The AlertsApmCondition should be removed from Kubernetes cluster after removing an AlertPolicy.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Deploy a policy

    $ cat <<EOF | kubectl apply -f -
apiVersion: nr.k8s.newrelic.com/v1
kind: AlertsPolicy
metadata:
name: my-alert-policy
spec:
account_id: [account_id]
api_key_secret:
name: nr-api-key
namespace: newrelic-kubernetes-operator-system
key_name: api-key
name: "My test policy"
incidentPreference: "PER_CONDITION_AND_TARGET"
region: "US"
conditions:
- spec:
    type: "apm_app_metric"
    enabled: true
    metric: "apdex"
    condition_scope: application
    entities:
      - "[entity_id]"
    apm_terms:
      - threshold: "0.7"
        time_function: "all"
        duration: "10"
        priority: "warning"
        operator: "below"
      - threshold: "0.7"
        time_function: "all"
        duration: "15"
        priority: "critical"
        operator: "below"
EOF
  2. 
$ kubectl get AlertsPolicy
NAME              AGE
my-alert-policy   38s

kubectl get AlertsApmCondition NAME CREATED my-alert-policy-condition-2pw8d


3. 
```bash
$ kubectl delete AlertsPolicy my-alert-policy

  1. In the manager logs

    {"level":"error","ts":1598903639.5313737,"logger":"controllers.AlertsAPMCondition","msg":"Failed to delete API Condition","conditionId":15881394,"region":"US","Api Key":"...","error":"resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/controllers.(*AlertsAPMConditionReconciler).Reconcile\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/controllers/alerts_apmcondition_controller.go:110\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"}
{"level":"info","ts":1598903639.5314548,"logger":"controllers.AlertsAPMCondition","msg":"New Relic API returned resource not found, deleting condition resource"}
{"level":"info","ts":1598903639.5314605,"logger":"controllers.AlertsAPMCondition","msg":"New Relic Alert condition deleted, Removing finalizer"}
{"level":"info","ts":1598903639.5360534,"logger":"alertsapmcondition-resource","msg":"default","name":"my-alert-policy-condition-2pw8d"}
{"level":"info","ts":1598903639.5399203,"logger":"alertsapmcondition-resource","msg":"validate update","name":{"apiVersion":"nr.k8s.newrelic.com/v1","kind":"AlertsAPMCondition","namespace":"newrelic-kubernetes-operator-system","name":"my-alert-policy-condition-2pw8d"}}
{"level":"info","ts":1598903639.5399554,"logger":"alertsapmcondition-resource","msg":"Checking existing","policyId":"1025375"}
{"level":"info","ts":1598903639.8808038,"logger":"alertsapmcondition-resource","msg":"Deleting resource","errAlertPolicy":"resource not found"}
{"level":"error","ts":1598903639.8809204,"logger":"alertsapmcondition-resource","msg":"failed to get policy","policyId":"1025375","API Key":"...","region":"US","error":"resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).CheckExistingPolicyID\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:276\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).ValidateUpdate\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:117\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/validator.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/webhook.go:135\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/http.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook.instrumentedHook.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/server.go:129\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2041\nnet/http.(*ServeMux).ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2416\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2836\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:1924"}
{"level":"info","ts":1598903639.8881001,"logger":"alertsapmcondition-resource","msg":"default","name":"my-alert-policy-condition-2pw8d"}
{"level":"info","ts":1598903639.8928833,"logger":"alertsapmcondition-resource","msg":"validate update","name":{"apiVersion":"nr.k8s.newrelic.com/v1","kind":"AlertsAPMCondition","namespace":"newrelic-kubernetes-operator-system","name":"my-alert-policy-condition-2pw8d"}}
{"level":"info","ts":1598903639.89293,"logger":"alertsapmcondition-resource","msg":"Checking existing","policyId":"1025375"}
{"level":"info","ts":1598903640.4122937,"logger":"alertsapmcondition-resource","msg":"Deleting resource","errAlertPolicy":"resource not found"}
{"level":"error","ts":1598903640.412395,"logger":"alertsapmcondition-resource","msg":"failed to get policy","policyId":"1025375","API Key":"...","region":"US","error":"resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).CheckExistingPolicyID\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:276\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).ValidateUpdate\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:117\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/validator.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/webhook.go:135\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/http.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook.instrumentedHook.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/server.go:129\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2041\nnet/http.(*ServeMux).ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2416\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2836\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:1924"}
{"level":"error","ts":1598903640.414743,"logger":"controllers.AlertsAPMCondition","msg":"Failed to update condition after deleting New Relic Alert condition","error":"admission webhook \"valertsapmcondition.kb.io\" denied the request: resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/controllers.(*AlertsAPMConditionReconciler).Reconcile\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/controllers/alerts_apmcondition_controller.go:130\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"}
{"level":"error","ts":1598903640.4149141,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"alertsapmcondition","request":"newrelic-kubernetes-operator-system/my-alert-policy-condition-2pw8d","error":"admission webhook \"valertsapmcondition.kb.io\" denied the request: resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"}

  2. AlertsApmCondition is still there and there is no way to delete it

$ kubectl get AlertsApmCondition
NAME                                                   CREATED
my-alert-policy-condition-2pw8d

$ kubectl edit AlertsApmCondition my-alert-policy-condition-2pw8d # Remove the `alertsapmconditions.finalizers.nr.k8s.newrelic.com` finalizer
error: alertsapmconditions.nr.k8s.newrelic.com "my-alert-policy-condition-2pw8d" could not be patched: admission webhook "valertsapmcondition.kb.io" denied the request: resource not found
guohaolee commented 2 years ago

Just to add a note if someone faced this issue.

You'll need to edit the existing policy id in the manifest with any policy ID that exist in your newrelic. Once done that it will be deleted.

If still not deleted, patch to remove the finalizers