search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.59k stars 1.09k forks source link

ScaledObject with argo Rollout scaleTargetRef fails to reconcile #4344

Closed golanpa closed 1 year ago

golanpa commented 1 year ago

Report

ScaledObject with Argo Rollout scaleTargetRef fails to reconcile in GKE and HPA resource is not created.

I am using Keda 2.9.3 (installed via https://kedacore.github.io/charts, version: 2.9.4).

Expected Behavior

HPA resource should be created for the ScaledObject.

Actual Behavior

Reconciliation fails with the following error and HPA resource is not created.

If a Deployment ref is used then the reconciliation succeeds, e.g:

scaleTargetRef
    apiVersion: apps/v1
    kind: Deployment
    name: archive-creator

Steps to Reproduce the Problem

Having an Argo Rollout resource named archive-creator use the following Keda resources:

ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: archive-creator-subscription
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: archive-creator
  minReplicaCount: 2
  maxReplicaCount: 200
  pollingInterval: 30
  cooldownPeriod: 180
  fallback:
    failureThreshold: 3
    replicas: 1
  triggers:
    - type: gcp-pubsub
      authenticationRef:
        name: keda-clustertrigger-auth-gcp-credentials
        kind: ClusterTriggerAuthentication
      metadata:
        subscriptionName: "projects/project/—–subscriptions/archive-creator"
        mode: "SubscriptionSize"
        metricType: "AverageValue"
        value: "16"

ClusterTriggerAuthentication:

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-clustertrigger-auth-gcp-credentials
spec:
  podIdentity:
    provider: gcp

Logs from KEDA operator

2023-03-08T14:53:47Z    ERROR   Failed to ensure HPA is correctly created for ScaledObject  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "67d2f6a9-9a9a-4ef4-b9fe-fa585da0c206", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}

KEDA Version

2.9.3

Kubernetes Version

None

Platform

Google Cloud

Scaler Details

gcp-pubsub

Anything else?

No response

golanpa commented 1 year ago

some additional information. The failure is happening when the Argo Rollout is defined with a workloadRef:

workloadRef:
     apiVersion: apps/v1
     kind: Deployment
     name: archive-creator

There is no issue if the Argo Rollout defines a pod template within the Spec. See for more information about workloadRef

JorTurFer commented 1 year ago

Hello, Could you share KEDA Operator logs?

zroubalik commented 1 year ago

In the error message I can see: "error parsing PubSub metadata: GoogleApplicationCredentials not found" please check the scaler config

golanpa commented 1 year ago

@zroubalik, i use the above ClusterTriggerAuthentication configuration. It works when I use Deployment, when I use Rollout with embedded pod spec. The error occurs only when I use Rollout with workloadRef.

Here is the problematic Rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: archive-creator
  namespace: production
  labels:
    app: archive-creator
  annotations:
    notifications.argoproj.io/subscribe.on-rollout-step-completed.slack: wixmp-argo-rollouts-notifications
    notifications.argoproj.io/subscribe.on-rollout-completed.slack: wixmp-argo-rollouts-notifications
    notifications.argoproj.io/subscribe.on-rollout-aborted.slack: wixmp-argo-rollouts-notifications
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: archive-creator
  **workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: archive-creator**
  strategy:
    canary:
      maxSurge: "25%"
      maxUnavailable: 0
      steps:
        - setWeight: 100
        - pause: { duration: 3m }
        - analysis:
            templates:
              - templateName: archive-creator-success-rate
                clusterScope: true
            args:
              - name: app
                value: archive-creator
              - name: namespace
                value: production
golanpa commented 1 year ago

@JorTurFer , here are more logs:

2023-03-14T07:10:37Z    INFO    Creating a new HPA  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "ede9c889-660c-4af1-9304-a1a3d14146af", "HPA.Namespace": "production", "HPA.Name": "keda-hpa-archive-creator-subscription"}
2023-03-14T07:10:37Z    INFO    Initializing Scaling logic according to ScaledObject Specification  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "ede9c889-660c-4af1-9304-a1a3d14146af"}
2023-03-14T07:10:37Z    ERROR   scalehandler    error resolving auth params {"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startPushScalers
    /workspace/pkg/scaling/scale_handler.go:313
2023-03-14T07:10:37Z    ERROR   scalehandler    Error getting scalers   {"type": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription", "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startPushScalers
    /workspace/pkg/scaling/scale_handler.go:315
2023-03-14T07:10:37Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9"}
2023-03-14T07:10:37Z    ERROR   scalehandler    error resolving auth params {"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:345
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:162
2023-03-14T07:10:37Z    ERROR   scalehandler    Error getting scalers   {"object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:347
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:162
E0314 07:10:37.818060       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c8741b21e", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71837", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34125421", FieldPath:""}, Reason:"KEDAScalerFailed", Message:"error parsing PubSub metadata: GoogleApplicationCredentials not found", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 813298441, time.Local), Count:2, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c8741b21e" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
2023-03-14T07:10:37Z    ERROR   scalehandler    error resolving auth params {"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
    /workspace/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
    /workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
    /workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Error getting scalers   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
    /workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
    /workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
    /workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Failed to create new HPA resource   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "HPA.Namespace": "production", "HPA.Name": "keda-hpa-archive-creator-subscription", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
    /workspace/controllers/keda/hpa.go:154
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Failed to check HPA for possible update {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:429
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Failed to ensure HPA is correctly created for ScaledObject  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:179
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
E0314 07:10:37.822198       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c8741b21e", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71837", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34125421", FieldPath:""}, Reason:"KEDAScalerFailed", Message:"error parsing PubSub metadata: GoogleApplicationCredentials not found", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 819924869, time.Local), Count:3, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c8741b21e" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
2023-03-14T07:10:37Z    ERROR   Reconciler error    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "cb49de5f-ab9f-4f16-b4a6-4c3a516d5ec9", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c"}
2023-03-14T07:10:37Z    ERROR   scalehandler    error resolving auth params {"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
    /workspace/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
    /workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
    /workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Error getting scalers   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
    /workspace/controllers/keda/hpa.go:202
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
    /workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
    /workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Failed to create new HPA resource   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "HPA.Namespace": "production", "HPA.Name": "keda-hpa-archive-creator-subscription", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
    /workspace/controllers/keda/hpa.go:154
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Failed to check HPA for possible update {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:429
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Failed to ensure HPA is correctly created for ScaledObject  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:179
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
E0314 07:10:37.852262       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c8741b21e", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71837", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34126099", FieldPath:""}, Reason:"KEDAScalerFailed", Message:"error parsing PubSub metadata: GoogleApplicationCredentials not found", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 849263448, time.Local), Count:4, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c8741b21e" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
E0314 07:10:37.854537       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"archive-creator-subscription.174c378c881519bb", GenerateName:"", Namespace:"production", SelfLink:"", UID:"", ResourceVersion:"71838", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ScaledObject", Namespace:"production", Name:"archive-creator-subscription", UID:"4976322f-917f-44c7-b518-48542c1d5c6f", APIVersion:"keda.sh/v1alpha1", ResourceVersion:"34126099", FieldPath:""}, Reason:"ScaledObjectCheckFailed", Message:"Failed to ensure HPA is correctly created for ScaledObject", Source:v1.EventSource{Component:"keda-operator", Host:""}, FirstTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 0, time.Local), LastTimestamp:time.Date(2023, time.March, 14, 7, 10, 37, 849420970, time.Local), Count:2, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "archive-creator-subscription.174c378c881519bb" is forbidden: User "system:serviceaccount:kedacore:keda-operator" cannot patch resource "events" in API group "" in the namespace "production"' (will not retry!)
2023-03-14T07:10:37Z    ERROR   Reconciler error    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "9778841f-dec2-4332-9321-aeeb94e1fa6c", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "c9d93f9d-f876-4bfb-a065-8b614f758c8d"}
2023-03-14T07:10:37Z    ERROR   scalehandler    error resolving auth params {"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "production", "name": "archive-creator-subscription"}, "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
    /workspace/pkg/scaling/scale_handler.go:543
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
    /workspace/pkg/scaling/scale_handler.go:269
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:190
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
    /workspace/controllers/keda/hpa.go:200
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
    /workspace/controllers/keda/hpa.go:74
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
    /workspace/controllers/keda/hpa.go:152
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
    /workspace/controllers/keda/scaledobject_controller.go:427
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
    /workspace/controllers/keda/scaledobject_controller.go:230
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
2023-03-14T07:10:37Z    ERROR   Error getting scalers   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"archive-creator-subscription","namespace":"production"}, "namespace": "production", "name": "archive-creator-subscription", "reconcileID": "c9d93f9d-f876-4bfb-a065-8b614f758c8d", "error": "error parsing PubSub metadata: GoogleApplicationCredentials not found"}
zroubalik commented 1 year ago

@golanpa gotcha, the problem is that workloadRef is not supported at the moment, you need to use Rollout with podSpec

golanpa commented 1 year ago

@zroubalik, thanks for the insight. Is there a request for that? should we transition this "bug" to a feature request?

zroubalik commented 1 year ago

@golanpa did that. Are you willing to contribute this?

golanpa commented 1 year ago

@zroubalik , i will try to take a look at that

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 1 year ago

This issue has been automatically closed due to inactivity.