If `pendingPodConditions` isn't set, KEDA never detects any pending jobs

Makeshift commented 1 month ago

Report

If pendingPodConditions is not set on a ScaledJob with strategy accurate, KEDA appears to incorrectly set the number of pending jobs as always being 0. This means that if a job takes longer to start up than the poll time of the trigger, you end up with duplicate jobs.

Expected Behavior

When using scalingStrategy.strategy: accurate, I'd expect KEDA to correctly count the number of pods that had been scheduled but are not yet running, and calculate the quantity to scale up by as QueueLength-RunningJobs-PendingJobs.

Actual Behavior

If pendingPodConditions is not set in your ScaledJob, KEDA will always calculate the number of pending jobs as being 0, resulting in duplicate jobs if your job has a long startup time.

Steps to Reproduce the Problem

Define a ScaledJob with:
```
jobTargetRef:
pollingInterval: 5
scalingStrategy:
strategy: accurate
```
- a pod spec that will spend at least 10 seconds in the pending state
Add 3 items to the trigger such that KEDA will scale up to 3 replicas. These 3 replicas should stay in the 'pending' state, however the KEDA operator will log "Number of pending Jobs": 0
After 5 seconds (the next poll), KEDA will attempt to launch 3 additional replicas (bringing the total to 6) because it does not see the pending jobs.

Reset and modify your ScaledJob to add all possible pendingPodConditions:

jobTargetRef:
pollingInterval: 5
scalingStrategy:
strategy: accurate
pendingPodConditions:
  - Ready
  - PodReadyToStartContainers
  - ContainersReady
  - Initialized
  - PodScheduled

Add 3 items to the trigger such that KEDA will scale up to 3 replicas. These 3 replicas should stay in the 'pending' state, and KEDA should log
```
"Number of pending Jobs": 4
No need to create jobs - all requested jobs already exist
```

Logs from KEDA operator

15s poll time with pendingPodConditions unset, comments mine:

2024-09-12T14:48:13Z    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of running Jobs": 0}
2024-09-12T14:48:13Z    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 0}
# 3 items submitted to queue - 3 jobs created and pending
2024-09-12T14:48:13Z    INFO    scaleexecutor   Creating jobs   {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Effective number of max jobs": 3}
2024-09-12T14:48:13Z    INFO    scaleexecutor   Creating jobs   {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
2024-09-12T14:48:13Z    INFO    scaleexecutor   Created jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
# KEDA claims 0 pending jobs
2024-09-12T14:48:28Z    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of running Jobs": 0}
2024-09-12T14:48:28Z    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 0}
# No new items added, previous 3 jobs still pending, KEDA creates 3 more jobs
2024-09-12T14:48:28Z    INFO    scaleexecutor   Creating jobs   {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Effective number of max jobs": 6}
2024-09-12T14:48:28Z    INFO    scaleexecutor   Creating jobs   {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
2024-09-12T14:48:28Z    INFO    scaleexecutor   Created jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
# The first 3 jobs are finally now running, KEDA still can't see the most recent 3 jobs that are pending
2024-09-12T14:48:43Z    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of running Jobs": 3}
2024-09-12T14:48:43Z    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 0}

15s poll time with pendingPodConditions set to all conditions, only part of the log but shows it's working correctly:

2024-09-12T14:42:13Z    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 4}
2024-09-12T14:42:13Z    INFO    scaleexecutor   No need to create jobs - all requested jobs already exist   {"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "jobs": 0}

KEDA Version

2.15.1

Kubernetes Version

1.28

Platform

Amazon Web Services

Scaler Details

SQS FIFO queue

Anything else?

I dug into this a tiny bit and I think the culprit is here. It looks like this function might be returning that a pod is running or complete when it's actually pending.

My conclusion that this is a bug is based on how the default behaviour is described here:

Default behavior - Job that have not finished yet and the underlying pod is either not running or has not been completed yet

quiqueg commented 2 days ago

I noticed the same behavior and can confirm that adding pendingPodConditions fixed it for us.

andretibolaintelipost commented 1 day ago

Adding pendingPodConditions also fixed for me, but it took me a while to get this fixed. I resorted to a large polling time at first, but that was affecting end users. Only after coming across this issue that i finally got to scale without duplicates and with a short polling period

kedacore / keda