Issue with minCount not functioning as Expected

[danieltaub]

I am encountering an issue with the minCount configuration in kubernetes-event-exporter, where it doesn't seem to be working as expected. Despite setting minCount to 3, I am receiving notifications for events with a count of 1. Interestingly, after cloning the repository and examining the minCount tests, everything seems to be in order, which makes this behavior quite puzzling. This issue does happens in the opposite side sometimes where count of the event greater them min count and we not receiving any alert/logs in the pod.

Configuration:

Here is the relevant part of my configuration:

route:
  routes:
  - drop:
      - type: "Normal"
    match:
      - receiver: dump
  - drop:
      - type: "Normal"
    match:
      - receiver: slack_webhook
  - drop:
     - minCount: 3
     - type: "Normal"
    match:
      - namespace: "server"
        labels:
          app.kubernetes.io/instance: server-prod
        receiver: pagerduty

Issue:

The configuration should theoretically prevent PagerDuty alerts for events with a count less than 3. However, I am still receiving alerts for events with a count of 1.

Example:

An example of an alert received, which is visible in the stdout of the exporter pod, is as follows (with sensitive information replaced):

{
  "metadata": {
    "name": "server-prod-6cc7d64c6f-2pbs6.17ba8670bd1e88ca",
    "namespace": "server",
    "uid": "5dea720f-2f25-43db-bb64-5cbeeec95953",
    "resourceVersion": "12194664",
    "creationTimestamp": "2024-03-07T15:53:47Z"
  },
  "reason": "FailedScheduling",
  "message": "0/4 nodes are available: 4 Insufficient cpu. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.",
  "source": {
    "component": "default-scheduler"
  },
  "firstTimestamp": "2024-03-07T15:53:47Z",
  "lastTimestamp": "2024-03-07T15:53:47Z",
  "count": 1,
  "type": "Warning",
  "eventTime": null,
  "reportingComponent": "default-scheduler",
  "reportingInstance": "",
  "clusterName": "",
  "involvedObject": {
    "kind": "Pod",
    "namespace": "server",
    "name": "server-prod-6cc7d64c6f-2pbs6",
    "uid": "34990f72-0e4b-4d85-9eec-d030f1259c7b",
    "apiVersion": "v1",
    "resourceVersion": "12194663",
    "labels": {
      "app.kubernetes.io/instance": "server-prod",
      "app.kubernetes.io/name": "server",
      "pod-template-hash": "6cc7d64c6f"
    },
    "annotations": {
      "kubectl.kubernetes.io/restartedAt": "2024-02-10T13:29:05Z"
    },
    "ownerReferences": [
      {
        "apiVersion": "apps/v1",
        "kind": "ReplicaSet",
        "name": "server-prod-6cc7d64c6f",
        "uid": "ba9c9cf3-1e91-4bb0-9026-1479e9921866",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ],
    "deleted": false
  }
}

As seen in the above JSON, the event count is 1, which should not trigger an alert based on my configuration.

Any assistance in resolving this issue would be greatly appreciated. Thank you for your time and help.

[danieltaub]

After looking in the code it seems minCount will work only within match rule, and not drop rule, maybe it's worth do add some docs on minCount conf

There is a test for Rule of minCount, but not via a Route For example, this test will fail becuase the event will proceed

func TestRouteCount(t *testing.T) {
    ev1 := kube.EnhancedEvent{}
    ev1.Type = "Warning"
    ev1.Count = 1
    ev1.Namespace = "server"

    reg := testReceiverRegistry{}

    r := Route{
        Drop: []Rule{{
            MinCount: 3,
            Type:     "Normal",
        }},
        Match: []Rule{{
            Namespace: "server",
            Receiver:  "elastic",
        }},
    }

    r.ProcessEvent(&ev1, &reg)

    assert.False(t, reg.isEventRcvd("elastic", &ev1))
}

But if the minCount will move to the Match section it will work and drop the event

func TestRouteCount(t *testing.T) {
    ev1 := kube.EnhancedEvent{}
    ev1.Type = "Warning"
    ev1.Count = 1
    ev1.Namespace = "server"

    reg := testReceiverRegistry{}

    r := Route{
        Drop: []Rule{{
            Type:     "Normal",
        }},
        Match: []Rule{{
                         MinCount: 3,
            Namespace: "server",
            Receiver:  "elastic",
        }},
    }

    r.ProcessEvent(&ev1, &reg)

    assert.False(t, reg.isEventRcvd("elastic", &ev1))
}