search

SeleniumHQ / docker-selenium

Provides a simple way to run Selenium Grid with Chrome, Firefox, and Edge using Docker, making it easier to perform browser automation
http://www.selenium.dev/docker-selenium/
Other
7.78k stars 2.5k forks source link

[🐛 Bug]: Issue with keda operator #2275

Open amardeep2006 opened 1 month ago

amardeep2006 commented 1 month ago

What happened?

I have grid deployed with autoscaling in kubernetes. All of sudden I am seeing keda-operator pod in crash back loop. I have tried multiple options Like Jobs/deployment but it always crashes. This was working fine before and I have tried multiple keda versions and selenium chart versions.

Need help with troubleshooting. If this needs to be raised with KEDA, please let me know .

Command used to start Selenium Grid with Docker (or Kubernetes)

global:
  seleniumGrid:
    # Image registry for all selenium components
    imageRegistry: myrepo/testautomation/selenium-grid
    # Image tag for all selenium components
    imageTag: "$IMAGE_TAG_GRID"
    # Image tag for browser's nodes
    nodesImageTag: "$IMAGE_TAG_NODES"
    # Image tag for browser's video recorder
    # videoImageTag: "ffmpeg-6.1-latest"
    # Pull secret for all components, can be overridden individually
    imagePullSecret: ""
    # Log level for all components. Possible values describe here: https://www.selenium.dev/documentation/grid/configuration/cli_options/#logging
    logLevel: INFO
    # kubectl image is used to execute kubectl commands in utility jobs
    kubectlImage: myrepo/testautomation/bitnami/kubectl:latest
isolateComponents: $ISOLATE_COMPONENTS
# Basic auth settings for Selenium Grid
basicAuth:
  # Enable or disable basic auth
  enabled: $BASIC_AUTH_ENABLED
autoscaling:
  enabled: $AUTOSCALING_ENABLED
  scalingType: $SCALING_TYPE
  # patchObjectFinalizers:
  #   enabled: false
  # Options for KEDA scaled resources (keep only common options used for both ScaledJob and ScaledObject)
  terminationGracePeriodSeconds: 5400 #default
  scaledOptions:
    minReplicaCount: 0
    maxReplicaCount: $MAX_REPLICAS_COUNT
customLabels: {"app-id": "selgrid", "app-tier": "application"}
ingress:
  # Name of ingress class to select which controller will implement ingress resource
  # Custom annotations for ingress resource
  annotations:
    cert-manager.io/cluster-issuer: "ci-0-acme-sectigo"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
  # Default host for the ingress resource
  hostname: $INGRESS_DOMAIN
  tls:
    - secretName: sel-grid-tls-secret
      hosts:
        - $INGRESS_DOMAIN
router:
  imagePullPolicy: Always
distributor:
  imagePullPolicy: Always  
eventBus:
  imagePullPolicy: Always
sessionMap:
  imagePullPolicy: Always
sessionQueue:
  imagePullPolicy: Always    
hub:
  # Custom sub path for the hub deployment
  # subPath: /selenium  
  imagePullPolicy: Always
  # Resources for container
  resources:
    requests:
      memory: "2Gi"
      cpu: "3"
    limits:
      memory: "8Gi"
      cpu: "4"
  extraEnvironmentVariables:
    - name: SE_JAVA_OPTS
      value: "-Xms1024m -Xmx8192m"
chromeNode:
  imagePullPolicy: Always
  # /dev/shm volume
  dshmVolumeSizeLimit: "2Gi"
  # Resources for chrome-node container
  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"
  extraEnvironmentVariables: 
    - name: "SE_NODE_ENABLE_MANAGED_DOWNLOADS"
      value: "true"
    # - name: "SE_VNC_NO_PASSWORD"
    #   value: "1"
    # - name: "SE_VNC_VIEW_ONLY"
    #   value: "1"      
  terminationGracePeriodSeconds: 5400
edgeNode:
  imagePullPolicy: Always
  # /dev/shm volume
  dshmVolumeSizeLimit: "2Gi"
  # Resources for edge-node container
  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"
  extraEnvironmentVariables: 
    - name: "SE_NODE_ENABLE_MANAGED_DOWNLOADS"
      value: "true"
    # - name: "SE_VNC_NO_PASSWORD"
    #   value: "1"
    - name: "SE_VNC_VIEW_ONLY"
      value: "1"         
  terminationGracePeriodSeconds: 5400
firefoxNode:
  enabled: true
  imagePullPolicy: Always
  # /dev/shm volume
  dshmVolumeSizeLimit: "2Gi"
  # Resources for firefox-node container
  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"
  extraEnvironmentVariables: 
    - name: "SE_NODE_ENABLE_MANAGED_DOWNLOADS"
      value: "true"
    # - name: "SE_VNC_NO_PASSWORD"
    #   value: "1"
    - name: "SE_VNC_VIEW_ONLY"
      value: "1"         
  autoscaling:
    scaledOptions:
      minReplicaCount: 0
      maxReplicaCount: 3
  terminationGracePeriodSeconds: 5400
keda:
  image:
    keda:
      # -- Image name of KEDA operator
      repository: myrepo/testautomation/kedacore/keda
      # -- Image tag of KEDA operator. Optional, given app version of Helm chart is used by default
      tag: "2.14.0"
    metricsApiServer:
      # -- Image name of KEDA Metrics API Server
      repository: myrepo/testautomation/kedacore/keda-metrics-apiserver
      # -- Image tag of KEDA Metrics API Server. Optional, given app version of Helm chart is used by default
      tag: "2.14.0"
    webhooks:
      # -- Image name of KEDA admission-webhooks
      repository: myrepo/testautomation/kedacore/keda-admission-webhooks
      # -- Image tag of KEDA admission-webhooks . Optional, given app version of Helm chart is used by default
      tag: "2.14.0"
    # -- Image pullPolicy for all KEDA components
  podLabels:
    # -- Pod labels for KEDA operator
    keda: {"app-id": "selgrid", "app-tier": "application"}
    # -- Pod labels for KEDA Metrics Adapter
    metricsAdapter: {"app-id": "selgrid", "app-tier": "application"}
    # -- Pod labels for KEDA Admission webhooks
    webhooks: {"app-id": "selgrid", "app-tier": "application"}

Relevant log output

2024/06/01 06:19:23 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2024-06-01T06:19:23.959283100Z 2024-06-01T06:19:23Z INFO    setup   Starting manager
2024-06-01T06:19:23.959319013Z 2024-06-01T06:19:23Z INFO    setup   KEDA Version: 2.14.0
2024-06-01T06:19:23.959323025Z 2024-06-01T06:19:23Z INFO    setup   Git Commit: 6681d5ecbc82293626cab693625c9cc9fb453381
2024-06-01T06:19:23.959326564Z 2024-06-01T06:19:23Z INFO    setup   Go Version: go1.21.9
2024-06-01T06:19:23.959329802Z 2024-06-01T06:19:23Z INFO    setup   Go OS/Arch: linux/amd64
2024-06-01T06:19:23.959332675Z 2024-06-01T06:19:23Z INFO    setup   Running on Kubernetes 1.26  {"version": "v1.26.15"}
2024-06-01T06:19:23.959335699Z 2024-06-01T06:19:23Z INFO    setup   WARNING: KEDA 2.14.0 hasn't been tested on Kubernetes v1.26.15
2024-06-01T06:19:23.959338504Z 2024-06-01T06:19:23Z INFO    setup   You can check recommended versions on https://keda.sh
2024-06-01T06:19:24.060591378Z 2024-06-01T06:19:24Z INFO    starting server {"kind": "health probe", "addr": "0.0.0.0:8081"}
2024-06-01T06:19:24.060639676Z I0601 06:19:24.060156       1 leaderelection.go:250] attempting to acquire leader lease mer-docconv-dev-selgrid/operator.keda.sh...
2024-06-01T06:19:59.614663604Z I0601 06:19:59.614492       1 leaderelection.go:260] successfully acquired lease mer-docconv-dev-selgrid/operator.keda.sh
2024-06-01T06:19:59.615050145Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
2024-06-01T06:19:59.615078181Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v2.HorizontalPodAutoscaler"}
2024-06-01T06:19:59.615082910Z 2024-06-01T06:19:59Z INFO    Starting Controller {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2024-06-01T06:19:59Z    INFO    Starting EventSource    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
2024-06-01T06:19:59.615621834Z 2024-06-01T06:19:59Z INFO    Starting Controller {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2024-06-01T06:19:59.615682544Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
2024-06-01T06:19:59Z    INFO    Starting Controller {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2024-06-01T06:19:59.615804081Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource", "source": "kind source: *v1alpha1.CloudEventSource"}
2024-06-01T06:19:59.615824547Z 2024-06-01T06:19:59Z INFO    Starting Controller {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}
2024-06-01T06:19:59.615920589Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
2024-06-01T06:19:59.615931419Z 2024-06-01T06:19:59Z INFO    Starting Controller {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2024-06-01T06:19:59.615975240Z 2024-06-01T06:19:59Z INFO    cert-rotation   starting cert rotator controller
2024-06-01T06:19:59.616026241Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2024-06-01T06:19:59.616078745Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-06-01T06:19:59.616083841Z 2024-06-01T06:19:59Z INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-06-01T06:19:59.616086600Z 2024-06-01T06:19:59Z INFO    Starting Controller {"controller": "cert-rotator"}
2024-06-01T06:19:59.721049127Z 2024-06-01T06:19:59Z INFO    cert-rotation   no cert refresh needed
2024-06-01T06:19:59.721224072Z 2024-06-01T06:19:59Z INFO    cert-rotation   certs are ready in /certs
2024-06-01T06:19:59.723513000Z 2024-06-01T06:19:59Z INFO    Starting workers    {"controller": "scaledjob", "
controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "worker count": 1}
2024-06-01T06:19:59.723530674Z 2024-06-01T06:19:59Z INFO    Starting workers    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "worker count": 1}
2024-06-01T06:19:59.723544490Z 2024-06-01T06:19:59Z INFO    Starting workers    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "worker count": 5}
2024-06-01T06:19:59.723895251Z 2024-06-01T06:19:59Z INFO    Reconciling ScaledJob   {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-firefox-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-firefox-node", "reconcileID": "91eeb527-77ba-44e2-9bbf-8f9b974548f9"}
2024-06-01T06:19:59.741719285Z 2024-06-01T06:19:59Z INFO    Starting workers    {"controller": "cert-rotator", "worker count": 1}
2024-06-01T06:19:59.742308407Z 2024-06-01T06:19:59Z INFO    cert-rotation   no cert refresh needed
2024-06-01T06:19:59.742672421Z 2024-06-01T06:19:59Z ERROR   cert-rotation   Webhook not found. Unable to update certificate.    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"keda-admission\" not found"}
2024-06-01T06:19:59.742698026Z github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
2024-06-01T06:19:59.742701209Z  /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:816
2024-06-01T06:19:59.742703999Z github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
2024-06-01T06:19:59.742706825Z  /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:785
2024-06-01T06:19:59.742709576Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:19:59.742712088Z  /workspace
/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:19:59.742714688Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:19:59.742717197Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:19:59.742720034Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:19:59.742722615Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:19:59.742725178Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:19:59.742727682Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:19:59.742732394Z 2024-06-01T06:19:59Z INFO    cert-rotation   Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-06-01T06:19:59Z    INFO    cert-rotation   no cert refresh needed
2024-06-01T06:19:59.752905149Z 2024-06-01T06:19:59Z ERROR   cert-rotation   Webhook not found. Unable to update certificate.    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"keda-admission\" not found"}
2024-06-01T06:19:59.752930920Z github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
2024-06-01T06:19:59.752936143Z  /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:816
2024-06-01T06:19:59.752940874Z github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
2024-06-01T06:19:59.752945688Z  /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:785
2024-06-01T06:19:59.752950259Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:19:59.752960600Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:19:59.752965286Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:19:59.752970516Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:19:59.752991374Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:19:59.752996065Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:19:59.753000469Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:19:59.753005058Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:19:59.753009491Z 2024-06-01T06:19:59Z INFO    cert-rotation   Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1
.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-06-01T06:19:59.834407875Z 2024-06-01T06:19:59Z INFO    Starting workers    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource", "worker count": 1}
2024-06-01T06:19:59.838461478Z 2024-06-01T06:19:59Z INFO    Starting workers    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "worker count": 1}
2024-06-01T06:20:00.792037447Z 2024-06-01T06:20:00Z INFO    cert-rotation   CA certs are injected to webhooks
2024-06-01T06:20:00.801565413Z 2024-06-01T06:20:00Z INFO    grpc_server Starting Metrics Service gRPC Server    {"address": ":9666"}
2024-06-01T06:20:35Z    INFO    Stopping and waiting for non leader election runnables
2024-06-01T06:20:35.241592692Z 2024-06-01T06:20:35Z INFO    Stopping and waiting for leader election runnables
2024-06-01T06:20:35Z    ERROR   scale_handler   error resolving auth params {"type": "ScaledJob", "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-firefox-node", "triggerIndex": 0, "error": "error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:mer-selgrid-selenium-secrets,},Optional:nil,} on namespace mer-docconv-dev-selgrid: Timeout: failed waiting for *v1.Secret Informer to sync"}
2024-06-01T06:20:35.260645768Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
2024-06-01T06:20:35.260650435Z  /workspace/pkg/scaling/scalers_builder.go:99
2024-06-01T06:20:35.260653893Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
2024-06-01T06:20:35.260657229Z  /workspace/pkg/scaling/scale_handler.go:357
2024-06-01T06:20:35.260660192Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).reconcileScaledJob
    /workspace/controllers/keda/scaledjob_controller.go:199
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
    /workspace/controllers/keda/scaledjob_controller.go:150
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:20:35.260698058Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:20:35.260701280Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.260704344Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.260707069Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.260709737Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.260712444Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.260715235Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35Z    ERROR   Error getting scalers   {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-firefox-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-firefox-node", "reconcileID": "91eeb527-77ba-44e2-9bbf-8f9b974548f9", "error": "error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:mer-selgrid-selenium-secrets,},Optional:nil,} on namespace mer-docconv-dev-selgrid: Timeout: failed waiting for *v1.Secret Informer to sync"}
2024-06-01T06:20:35.264246893Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).reconcileScaledJob
2024-06-01T06:20:35.264253624Z  /workspace/controllers/keda/scaledjob_controller.go:201
2024-06-01T06:20:35.264258286Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
2024-06-01T06:20:35.264262828Z  /workspace/controllers/keda/scaledjob_controller.go:150
2024-06-01T06:20:35.264267353Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:20:35.264271866Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.264279944Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.264297932Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.264307587Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.264311486Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35.264330571Z 2024-06-01T06:20:35Z ERROR   Failed to ensure ScaledJob is correctly created {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-firefox-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-firefox-node", "reconcileID": "91eeb527-77ba-44e2-9bbf-8f9b974548f9", "error": "error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:mer-selgrid-selenium-secrets,},Optional:nil,} on namespace mer-docconv-dev-selgrid: Timeout: failed waiting for *v1.Secret Informer to sync"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
2024-06-01T06:20:35.264342827Z  /workspace/controllers/keda/scaledjob_controller.go:152
2024-06-01T06:20:35.264347384Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:20:35.264352041Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:20:35.264356373Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.264374288Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.264378957Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.264384163Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.264388595Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.264393193Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2024-06-01T06:20:35.273107564Z 2024-06-01T06:20:35Z INFO    Shutdown signal received, waiting for all workers to finish {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}
2024-06-01T06:20:35.273115440Z 2024-06-01T06:20:35Z INFO    Shutdown signal received, waiting for all workers to finish {"controller": "cert-rotator"}
2024-06-01T06:20:35.273120763Z 2024-06-01T06:20:35Z INFO    Shutdown signal received, waiting for all workers to finish {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2024-06-01T06:20:35.273126276Z 2024-06-01T06:20:35Z INFO    Shutdown signal received, waiting for all workers to finish {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2024-06-01T06:20:35Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2024-06-01T06:20:35Z    INFO    cert-rotation   stopping cert rotator controller
2024-06-01T06:20:35Z    INFO    All workers finished    {"controller": "cert-rotator"}
2024-06-01T06:20:35.287156572Z 2024-06-01T06:20:35Z INFO    All workers finished    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2024-06-01T06:20:35Z    INFO    All workers finished    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}
2024-06-01T06:20:35.287192854Z 2024-06-01T06:20:35Z INFO    All workers finished    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2024-06-01T06:20:35.287197694Z 2024-06-01T06:20:35Z INFO    All workers finished    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2024-06-01T06:20:35Z    ERROR   failed to patch Objects {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-firefox-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-firefox-node", "reconcileID": "91eeb527-77ba-44e2-9bbf-8f9b974548f9", "error": "client rate limiter Wait returned an error: context canceled"}
2024-06-01T06:20:35.430317295Z github.com/kedacore/keda/v2/pkg/status.TransformObject
2024-06-01T06:20:35.430322731Z  /workspace/pkg/status/status.go:195
2024-06-01T06:20:35.430327522Z github.com/kedacore/keda/v2/pkg/status.SetStatusConditions
2024-06-01T06:20:35.430333453Z  /workspace/pkg/status/status.go:51
2024-06-01T06:20:35.430336826Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
    /workspace/controllers/keda/scaledjob_controller.go:165
2024-06-01T06:20:35.430343128Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:20:35.430350417Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.430353365Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.430356498Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.430359534Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.430362466Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.430365370Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35.430368285Z 2024-06-01T06:20:35Z ERROR   Reconciler error    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-firefox-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-firefox-node", "reconcileID": "91eeb527-77ba-44e2-9bbf-8f9b974548f9", "error": "client rate limiter Wait returned an error: context canceled"}
2024-06-01T06:20:35.430398384Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.430409866Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.430414487Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.430420283Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35Z    INFO    Reconciling ScaledJob   {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-chrome-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-chrome-node", "reconcileID": "5f3a640a-8385-40f8-b5db-9ee4daa00f33"}
2024-06-01T06:20:35Z    ERROR   scale_handler   error resolving auth params {"type": "ScaledJob", "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-chrome-node", "triggerIndex": 0, "error": "error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:mer-selgrid-selenium-secrets,},Optional:nil,} on namespace mer-docconv-dev-selgrid: Timeout: failed waiting for *v1.Secret Informer to sync"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
2024-06-01T06:20:35.471582702Z  /workspace/pkg/scaling/scalers_builder.go:99
2024-06-01T06:20:35.471594193Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
2024-06-01T06:20:35.471597500Z  /workspace/pkg/scaling/scale_handler.go:357
2024-06-01T06:20:35.471600783Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
    /workspace/pkg/scaling/scale_handler.go:282
2024-06-01T06:20:35.471606986Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).reconcileScaledJob
    /workspace/controllers/keda/scaledjob_controller.go:199
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
    /workspace/controllers/keda/scaledjob_controller.go:150
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:20:35.471620446Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.471645306Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.471650862Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35.472312786Z 2024-06-01T06:20:35Z ERROR   Error getting scalers   {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-chrome-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-chrome-node", "reconcileID": "5f3a640a-8385-40f8-b5db-9ee4daa00f33", "error": "error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:mer-selgrid-selenium-secrets,},Optional:nil,} on namespace mer-docconv-dev-selgrid: Timeout: failed waiting for *v1.Secret Informer to sync"}
2024-06-01T06:20:35.472342642Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).reconcileScaledJob
2024-06-01T06:20:35.472346572Z  /workspace/controllers/keda/scaledjob_controller.go:201
2024-06-01T06:20:35.472349662Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
2024-06-01T06:20:35.472352540Z  /workspace/controllers/keda/scaledjob_controller.go:150
2024-06-01T06:20:35.472355618Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:20:35.472358645Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:20:35.472361305Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.472364044Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.472366739Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.472369571Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.472372252Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-0
6-01T06:20:35.472374943Z    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35.472482956Z 2024-06-01T06:20:35Z ERROR   Failed to ensure ScaledJob is correctly created {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-chrome-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-chrome-node", "reconcileID": "5f3a640a-8385-40f8-b5db-9ee4daa00f33", "error": "error resolving secrets for ScaleTarget: error reading secret ref &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:mer-selgrid-selenium-secrets,},Optional:nil,} on namespace mer-docconv-dev-selgrid: Timeout: failed waiting for *v1.Secret Informer to sync"}
2024-06-01T06:20:35.472500960Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
2024-06-01T06:20:35.472505783Z  /workspace/controllers/keda/scaledjob_controller.go:152
2024-06-01T06:20:35.472511085Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:20:35.472516418Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:20:35.472520626Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.472525174Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.472529313Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.472533328Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.472541414Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35Z    ERROR   failed to patch Objects {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-chrome-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-chrome-node", "reconcileID": "5f3a640a-8385-40f8-b5db-9ee4daa00f33", "error": "client rate limiter Wait returned an error: context canceled"}
2024-06-01T06:20:35.499425464Z github.com/kedacore/keda/v2/pkg/status.TransformObject
2024-06-01T06:20:35.499429654Z  /workspace/pkg/status/status.go:195
2024-06-01T06:20:35.499432647Z github.com/kedacore/keda/v2/pkg/status.SetStatusConditions
2024-06-01T06:20:35.499435566Z  /workspace/pkg/status/status.go:51
2024-06-01T06:20:35.499438537Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
2024-06-01T06:20:35.499441851Z  /workspace/controllers/keda/scaledjob_controller.go:165
2024-06-01T06:20:35.499444576Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:20:35.499449963Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.499452693Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.499455516Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.499458342Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.499470783Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.499473835Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35.499476819Z 2024-06-01T06:20:35Z ERROR   Reconciler error    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-chrome-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-chrome-node", "reconcileID": "5f3a640a-8385-40f8-b5db-9ee4daa00f33", "error": "client rate limiter Wait returned an error: context canceled"}
2024-06-01T06:20:35.499480943Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.499484674Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
2024-06-01T06:20:35.499487434Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.499492747Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.499495472Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35Z    INFO    Reconciling ScaledJob   {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-edge-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-edge-node", "reconcileID": "9dbfdce1-01f5-4ba0-a540-f878b6fa0343"}
2024-06-01T06:20:35.501225853Z 2024-06-01T06:20:35Z INFO    Adding Finalizer for the ScaledJob  {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-edge-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-edge-node", "reconcileID": "9dbfdce1-01f5-4ba0-a540-f878b6fa0343"}
2024-06-01T06:20:35Z    ERROR   Failed to update ScaledJob with a finalizer {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-edge-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-edge-node", "reconcileID": "9dbfdce1-01f5-4ba0-a540-f878b6fa0343", "finalizer": "finalizer.keda.sh", "error": "client rate limiter Wait returned an error: context canceled"}
2024-06-01T06:20:35.505671652Z github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).ensureFinalizer
    /workspace/controllers/keda/scaledjob_finalizer.go:73
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
    /workspace/controllers/keda/scaledjob_controller.go:128
2024-06-01T06:20:35.505695527Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
2024-06-01T06:20:35.505698750Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
2024-06-01T06:20:35.505701766Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.505704665Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
2024-06-01T06:20:35.505707550Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.505710386Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.505713081Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35.505718452Z 2024-06-01T06:20:35Z ERROR   Reconciler error    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"mer-selgrid-selenium-edge-node","namespace":"mer-docconv-dev-selgrid"}, "namespace": "mer-docconv-dev-selgrid", "name": "mer-selgrid-selenium-edge-node", "reconcileID": "9dbfdce1-01f5-4ba0-a540-f878b6fa0343", "error": "client rate limiter Wait returned an error: context canceled"}
2024-06-01T06:20:35.505722405Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
2024-06-01T06:20:35.505726077Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
2024-06-01T06:20:35.505744258Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
2024-06-01T06:20:35.505747306Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
2024-06-01T06:20:35.505750039Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
2024-06-01T06:20:35.505752843Z  /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-06-01T06:20:35.505755589Z 2024-06-01T06:20:35Z INFO    All workers finished    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2024-06-01T06:20:35.505758612Z 2024-06-01T06:20:35Z INFO    Stopping and waiting for caches
I0601 06:20:35.525106       1 trace.go:236] Trace[1528225064]: "Reflector ListAndWatch" name:sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 (01-Jun-2024 06:20:00.028) (total time: 35496ms):
Trace[1528225064]: [35.496160664s] [35.496160664s] END
2024-06-01T06:20:35Z    INFO    Stopping and waiting for webhooks
2024-06-01T06:20:35Z    INFO    Stopping and waiting for HTTP servers
2024-06-01T06:20:35Z    INFO    shutting down server    {"kind": "health probe", "addr": "0.0.0.0:8081"}
2024-06-01T06:20:35.549862722Z 2024-06-01T06:20:35Z INFO    Wait completed, proceeding to shutdown the manager
E0601 06:20:35.555281       1 leaderelection.go:369] Failed to update lock: client rate limiter Wait returned an error: context canceled
I0601 06:20:35.555334       1 leaderelection.go:285] failed to renew lease mer-docconv-dev-selgrid/operator.keda.sh: timed out waiting for the condition

Operating System

Kubernetes 1.26.15

Docker Selenium version (image tag)

4.21.0

Selenium Grid chart version (chart version)

0.31.1

github-actions[bot] commented 1 month ago

@amardeep2006, thank you for creating this issue. We will troubleshoot it as soon as we can.

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

VietND96 commented 3 weeks ago

@amardeep2006, I don't have any clue about this issue. CI tests are running with the latest KEDA and a range of K8s versions but didn't see this issue, also the CI test disables the webhook service in KEDA.

amardeep2006 commented 3 weeks ago

Thanks @VietND96 I will retry enabling this week . If no clue then I will seek help with KEDA forums. If I find some RCA I will share here as well.