ScaledObject no being cached correctly at all times

[gauron99]

This is a tracking issue of a bug for which a workaround has been implemented in number of places for this time until its resolved globally.

What happens

ScaledObject is sometimes not cached correctly -> for example the metrics are an empty list or the .Status.ScaleTargetGVKR is nil

Workaround

A workaround has been implemented in a number of places. See below.

• pkg/scaling/scale_handler.go in func (h *scaleHandler) getTrueMetricArray()
• pkg/scaling/resolver/scale_resolvers.go in func ResolveScaleTargetPodSpec()

Other info

initial related issue link

/kind bug

[BehbudSh]

Any updates on this issue? We are facing a problem with a scaling of the Production instance

[nrukavkov]

The same here

[nuved]

Hey The same here, After migrating the k8s cluster from 1.22.6 to 1.23.17 and upgrading keda from 2.8.2 to 2.9.4, we've faced the same issue. keda is connecting to rabbitmq cluster and based on the count of messages in the queue, will adjust the HPA.

I even tried to delete the helm chart of keda and install the keda from scratch but still I can see the same errors.

[gauron99]

@zroubalik ^

[nuved]

I'm just wondering why I can still see errors when I already deleted all TriggerAuthentication and ScaledObject and deleted the chart of keda and all its crd .

when I install keda I can see these errors in operator

23-11-15T13:30:42Z ERROR scalehandler failed to get ScaledObject {"name": "", "namespace": "default", "error": "ScaledObject.keda.sh \"\" not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
 /workspace/pkg/scaling/scale_handler.go:240
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalersCacheForScaledObject
 /workspace/pkg/scaling/scale_handler.go:198
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
 /workspace/pkg/scaling/scale_handler.go:389
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
 /workspace/pkg/metricsservice/server.go:45
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
 /workspace/pkg/metricsservice/api/metrics_grpc.pb.go:79
google.golang.org/grpc.(*Server).processUnaryRPC
 /workspace/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
 /workspace/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
 /workspace/vendor/google.golang.org/grpc/server.go:965

while I don't have any ScaledObject in the default ns and the name of scaledObject also is empty in the log.

in operator-metrics , also I can see these errors

E1115 14:23:56.826045       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc000116628)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:00.678277       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc0013781e0)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:01.420055       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc000116668)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:11.853357       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc000330e60)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:15.703234       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc000469708)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:16.435198       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc0017fc0f8)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:26.862525       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc000469758)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:30.713697       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc0004a2340)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found
E1115 14:24:31.449717       1 status.go:71] apiserver received an error that is not an metav1.Status: &status.Error{s:(*status.Status)(0xc0004a2390)}: rpc error: code = Unknown desc = error when getting metric values error when getting scalers ScaledObject.keda.sh "" not found

[janmara]

We are facing the same issue. :(

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[dontbesatisfied]

I'm facing this issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.