Keda Cron Scaler can't be unsuspended

[arnoldyahad]

Report

Hello, We have a ScaledObject that is scaling our application based on the Cron Scaler.

Sometimes we have production events that cause us to want even more replicas than what is configured in the cron, so we just change the number of desired replicas using the suspend annotation:

kubectl annotate scaledObject -n my-apps my-app autoscaling.keda.sh/paused-replicas=25

this works, and immediatly changes the replicas to the number that is stated in the annotation. The problem is - unpause, when removing the annotation:

kubectl annotate scaledObject -n my-apps my-app autoscaling.keda.sh/paused-replicas-

this doesnt do anything, so basically once we add a pause-replicas annotations we are stuck with this number of replicas. we also dont see anything in the logs regarding the removal of the annotation like we see with other scalers when we remove the annotation. so it seems like there is a bug to get the event that the annotation was removed when using cron.

this graph shows how it looks:

at first we were using paused-replicas on 30, after removing it nothing happened so we deleted and recreated the scaled object - and everything is back to normal, than we did paused-replicas again on 25 replicas - removed it and now we stuck on 25 replicas.

Expected Behavior

when removing the annotation of paused-replicas return to the needed number of replicas.

Actual Behavior

nothing happens when removing the annotations.

Steps to Reproduce the Problem

1. use cron scaler
2. add an annotation of pause-replicas
3. remove the annotation of pause-replicas

Logs from KEDA operator

No response

KEDA Version

2.7.1

Kubernetes Version

< 1.23

Platform

Amazon Web Services

Scaler Details

Cron

Anything else?

No response

[zroubalik]

Thanks for reporting, this definitely sounds like a bug. Are you willing to contribute fix?

[rayjanoka]

I noticed something weird like this today, it seems like setting the autoscaling.keda.sh/paused-replicas value on the ScaledObject to something equal or higher than the maxReplicas results in a condition that KEDA cannot clear. If I remove the paused-replicas annotation from the scaled object, nothing happens to the HPA. Then if I remove the paused-replicas annotation from the HPA, KEDA resumes management of the HPA.

I may be able to circle back to this in a few days and take a peek at the code.

[tomkerkhove]

Hm interesting - Nice find. Let us know what you can find out!