search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.01k stars 1.01k forks source link

Apache Kafka limitToPartitionsWithLag is not respected #5880

Open kamilglod opened 2 weeks ago

kamilglod commented 2 weeks ago

Report

When limitToPartitionsWithLag is set to true and have some partitions without lag keda is still keeping max number of pods scaled

Expected Behavior

Keda will scaled up to the number of partitions with lag

Actual Behavior

Keda is scaling to max number of pods

Steps to Reproduce the Problem

  1. Have keda scaledobject with limitToPartitionsWithLag = true
  2. Have a lot of events on topic distributed unevenly on all partitions that would result in scaling max number of pods
  3. Observe that when keda will start MAX number of pods instead of number of partitions - partitions without lag

Logs from KEDA operator

2024-06-12T08:10:21Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "75c29f13-3068-455d-acc1-6b228ed37030"}
2024-06-12T08:10:36Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "7b0ac5eb-f1a4-4460-857d-8e7204cdecd9"}
2024-06-12T08:10:52Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "a1adb2b8-636e-4dcc-8e92-e87421976070"}
2024-06-12T08:11:07Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "5b899b41-a293-4ca8-8553-bc18f849c70b"}
2024-06-12T08:11:22Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "9566c55b-f2bb-473b-888a-4faf38e3c1c3"}
2024-06-12T08:11:38Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "e0739a9b-1ca1-44b2-af44-7c43b1d5826d"}
2024-06-12T08:25:02Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "dd7e3f55-bd64-4ab9-9a2e-0bd26bf0ebea"}
2024-06-12T08:26:35Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "0048c21e-834d-418f-8ce0-a2443cc5c575"}
2024-06-12T08:26:50Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "52a36446-e4ff-48c6-9b27-c75283a69c2f"}
2024-06-12T08:27:06Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "93cbee63-53c1-40ff-afda-996aaeb0bf4e"}
2024-06-12T08:27:21Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "d3a84b4a-7f67-481b-8d75-4bc5f8e27d96"}
2024-06-12T08:27:37Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "90e07ab4-0f35-416d-8340-b950d211345b"}
2024-06-12T08:27:52Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "6d524afb-5954-47cb-a3c3-19ebd3d3b03f"}
2024-06-12T08:29:09Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "14b6c5f0-b0e3-44d0-a680-a663006b39e9"}
2024-06-12T08:29:25Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "ab86f102-1dbb-4e0a-92df-ddfdc08edc19"}
2024-06-12T08:30:57Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "3e3112db-65c3-4b9e-9558-421269e74e7e"}

KEDA Version

2.12.0

Kubernetes Version

1.28

Platform

Google Cloud

Scaler Details

Kafka

Anything else?

Scaled object definition

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  labels:
    k8s.prisjakt.se/app: feed-ingestion-scaled-object
    k8s.prisjakt.se/owner: team-data-operations
    k8s.prisjakt.se/tier: backend
    team: ingestion
    component: feed-ingestion-service
    environment: dev
  name: fis-process-offer-scaled-object
  namespace: data-integration
spec:
  scaleTargetRef:
    name: fis-process-offer
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 60
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: xxx
        consumerGroup: xxx
        topic: xxx
        lagThreshold: '10000'
        offsetResetPolicy: earliest
        tls: enable
        sasl: plaintext
        limitToPartitionsWithLag: "true"
      authenticationRef:
        name: kafka-trigger-auth