HTTPScaledObject restarts existing pods

Report

I am using Keda http add-on to scale pods based on pending request. After installing it and setup forward request to keda-add-ons-http-interceptor-proxy service, I have simulated 502 requests to test scale pod automatically. But, number of pending request is equal or greater than spec.targetPendingRequests: 5, I see that HTTPScaledObject has restarted my old pods and then continuing to scale more pods to spec.replicas.max: 7.

I do not know this is bug or feature of http-add-on. Please helping me fix this ?

Expected Behavior

I expect http-add-on still keeps old pods and scales new pods basing on specification defined

Actual Behavior

My old pods is restarted by http-add-on when pending request is equal or greater spec.targetPendingRequests: 5

Steps to Reproduce the Problem

To simulate 502 requests
To watch pods by command watch kubectl -n <namesapce> get pod
When pending request is equal or greater than spec.targetPendingRequests: 5, I see that HTTPScaledObject restarted my old pods

Logs from KEDA HTTP operator

2024-06-07T02:58:44Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "e96a4b20-9d5d-41cd-8d03-1ff1e861a10d", "resource": "apps/v1.Deployment", "name": "my-scaler"}
2024-06-07T02:58:49Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "df6be875-be0c-4e6f-97ac-47c636af38b8"}
2024-06-07T02:58:49Z    ERROR   external_push_scaler    error running internalRun   {"type": "ScaledObject", "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "error": "rpc error: code = Canceled desc = context canceled"}
github.com/kedacore/keda/v2/pkg/scalers.(*externalPushScaler).Run.func1
    /workspace/pkg/scalers/external_scaler.go:249
github.com/kedacore/keda/v2/pkg/scalers.(*externalPushScaler).Run
    /workspace/pkg/scalers/external_scaler.go:268
2024-06-07T02:58:49Z    INFO    Successfully restored scaleTarget's replica count back to the original  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "df6be875-be0c-4e6f-97ac-47c636af38b8", "replicaCount": 3}
2024-06-07T02:58:49Z    INFO    Successfully finalized ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "df6be875-be0c-4e6f-97ac-47c636af38b8"}
2024-06-07T02:59:10Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b"}
2024-06-07T02:59:10Z    INFO    Adding Finalizer for the ScaledObject   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b"}
2024-06-07T02:59:10Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b", "resource": "apps/v1.Deployment", "name": "my-scaler"}
2024-06-07T02:59:10Z    INFO    Creating a new HPA  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b", "HPA.Namespace": "my-scaler", "HPA.Name": "keda-hpa-my-scaler-keda-autoscaler"}
2024-06-07T02:59:10Z    INFO    Initializing Scaling logic according to ScaledObject Specification  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b"}
2024-06-07T02:59:10Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "0e430f33-840c-42fa-9df3-df7825f9d8e2"}
2024-06-07T02:59:10Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "0e430f33-840c-42fa-9df3-df7825f9d8e2", "resource": "apps/v1.Deployment", "name": "my-scaler"}
2024-06-07T02:59:40Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "7c077642-456b-4f24-8ab2-1cfb3c785d17"}
2024-06-07T02:59:40Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "7c077642-456b-4f24-8ab2-1cfb3c785d17", "resource": "apps/v1.Deployment", "name": "my-scaler"}

HTTP Add-on Version

0.7.0

Kubernetes Version

1.29

Platform

Other

Anything else?

apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  name: my-scaler-name-keda-autoscaler
  namespace: my_namespace
spec:
  hosts:
  - sub.domain.com
  pathPrefixes:
  - /api/v1
  targetPendingRequests: 5
  scaleTargetRef:
    name: my-deployment-name
    kind: Deployment
    apiVersion: apps/v1
    service: my-service-svc
    port: 8000
  replicas:
    min: 3
    max: 7

kedacore / http-add-on