kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.27k stars 1.05k forks source link

Keda operator pod restarts because of objectScaler #5817

Closed tomerle03 closed 1 month ago

tomerle03 commented 3 months ago

Report

A scaledobject that causing the keda operator pod to reset

Expected Behavior

The scaledobject will not cause the keda operator pod to restart and affect the whole cluster

Actual Behavior

We are using Openshift in version 4.12 and keda in version 2.11. We have created a scaledobject that will sample Eck and will scale a DeploymentConfig in the cluster. Somehow this scaledobject caused the operator to restart a lot

Steps to Reproduce the Problem

  1. Install keda in the operatorHub in version 2.11.2-322 2.deploy keda controller
  2. Deploy scaledobject on deploymentConfig

Logs from KEDA operator


2024-05-20T14:53:52Z ERROR scale_resolvers target resource doesn't exist {"scaledobject.Namespace": "NAMESPACE" , "scaledobject.Name": "example-scaledobject", "resource": "/, Kind=", "name": "DEPLOYMENTCONFIG_NAME",
"error": "Obiect 'Kind' is missing
in 'unstructured object has no kind'"}
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveScaleTargetPodSpec
/remote-source/keda/app/pkg/scaling/resolver/scale_resolvers.go: 125
github.com/kedacore/keda/v2/pkg/scaling- (*scaleHandler)-performGetScalersCache
/remote-source/keda/app/pkg/scaling/scale_handler-go:355
github.com/kedacore/keda/v2/pkg/scaling- (*scaleHandler) •getScalersCacheForScaledObject
/remote-source/keda/app/pkg/scaling/scale_handler-go: 289
github.com/kedacore/keda/v2/pkg/scaling- (*scaleHandler).GetScaledObjectMetrics
/remote-source/keda/app/pkg/scaling/scale_handler. go:414
github.com/kedacore/keda/v2/pkg/metricsservice. (*GrpcServer).GetMetrics
/remote-source/keda/app/pkg/metricsservice/server-go:47
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
/remote-source/keda/app/pkg/metricsservice/api/metrics_grpc.pb-go:99
google-golang.org/grpc. (*Server) - processUnaryRPC
/remote-source/keda/app/vendor/google-golang.org/grpc/server-go:1337
google-golang.org/grpc. (*5erver) -handleStream
/remote-source/keda/app/vendor/google-golang.org/grpc/server-go:1714
google-golang.org/grpc. (*Server) -serveStreams.func1.1
/remote-source/keda/app/vendor/google-golang.org/grpc/server. go:959

KEDA Version

2.11.2

Kubernetes Version

< 1.27

Platform

Red Hat OpenShift

Scaler Details

ElasticSearch

Anything else?

No response

SpiritZhou commented 3 months ago

Could you also provide the ScaledObject yaml?

tomerle03 commented 3 months ago

Could you also provide the ScaledObject yaml?

apiVersion: keda.sh/v1alpha1
kind: Scaledobject
metadata:
  labels:
    scaledobject.keda.sh/name: Scaledobject_NAME
  name: Scaledobject_NAME
  namespace: NAMESPACE
spec:
  cooldownPeriod: 180
  maxReplicaCount: 20
  minReplicaCount: 2
  pollingInterval: 5
  scaleTargetRef:
    apiVersion: apps.openshift.io/v1
    kind: DeploymentConfig
    name: DeploymentConfig_NAME
  triggers:
    - authenticationRef:
        name: AUTHREF_NAME
      metadata:
        addresses: 'https://eck.example.com:443'
        index: index
        parameters: example
        searchTemplateName: desired_pods_count
        targetValue: '1'
        unsafeSs1: 'true'
        username: user
        valueLocation: aggregations.0.buckets.0.pods_count.value
      type: elasticsearch
zroubalik commented 3 months ago

could you paste here output of kubectl get scaledobject -oyaml -n NAMESPACE ?

And also more logs, as the attached are not correctly pasted, see this part: "resource": "/, Kind=", "

2024-05-20T14:53:52Z ERROR scale_resolvers target resource doesn't exist {"scaledobject.Namespace": "NAMESPACE" , "scaledobject.Name": "example-scaledobject", "resource": "/, Kind=", "name": "DEPLOYMENTCONFIG_NAME",
"error": "Obiect 'Kind' is missing
in 'unstructured object has no kind'"}
tomerle03 commented 3 months ago

could you paste here output of kubectl get scaledobject -oyaml -n NAMESPACE ?

And also more logs, as the attached are not correctly pasted, see this part: "resource": "/, Kind=", "


2024-05-20T14:53:52Z ERROR scale_resolvers target resource doesn't exist {"scaledobject.Namespace": "NAMESPACE" , "scaledobject.Name": "example-scaledobject", "resource": "/, Kind=", "name": "DEPLOYMENTCONFIG_NAME",

"error": "Obiect 'Kind' is missing

in 'unstructured object has no kind'"}

About the first question, do you need the status?

And secondly, the paste is correct I just checked. That's what weird.

zroubalik commented 3 months ago

Yes, full status.

stale[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 1 month ago

This issue has been automatically closed due to inactivity.