search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.27k stars 1.05k forks source link

Reconciler crashes with ScaledObject postgres trigger and Vault for authenticationRef #5625

Closed caoimheharvey closed 1 month ago

caoimheharvey commented 5 months ago

Report

I am deploying the ScaledObject and TriggerAuthentication via Helm (in a custom chart) alongside the Target of the scaling (a deployment). The TriggerAuthentication fetches the credentials for the postgres trigger from Vault. This seems to cause the keda-operator to crash with the error listed below.

Expected Behavior

Logging to indicate that it is connecting to postgres or authentication issue, basically not to crash.

Actual Behavior

It is crashing with the following error: 

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x32915f5]

goroutine 272 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:115 +0x1fa
panic({0x3837a00, 0x6a730a0})

Steps to Reproduce the Problem

  1. Configure your deployment, in my case I have a deployment which sends an email every 10 seconds and fetches the mailing queue from a postgres database. Unfortunately I cannot share the yaml for this.
  2. Configure the ScaledObject and the TriggerAuthentication: 
    
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: deployment-auth
spec:
hashiCorpVault:
address: {{ .Values.vault_host }}
authentication: kubernetes
role: read-secrets
mount: /vault/secrets
secrets:
- parameter: userName
  key: DATABASE_USER
  path: /secret/deploy/{{ $.Values.environment }}/secret
- parameter: password
  key: DATABASE_PASSWORD
  path: /secret/deploy/{{ $.Values.environment }}/secret
- parameter: host
  key: DATABASE_DEFAULT_REPLICA_HOST
  path: /secret/deploy/{{ $.Values.environment }}/inventory
- parameter: dbName
  key: DEFAULT__DATABASE_NAME
  path: /secret/deploy/{{ $.Values.environment }}/setting
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: deployment-name
spec:
scaleTargetRef:
kind:    Deployment
name:    deployment-name
pollingInterval:  10
cooldownPeriod:   300
minReplicaCount:  1
maxReplicaCount:  10
advanced:
restoreToOriginalReplicaCount: false
horizontalPodAutoscalerConfig:
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 180
      policies:
      - type: Percent
        value: 50
        periodSeconds: 30
triggers:
- type: postgresql
metadata:
  query: "SELECT count(id) FROM queue"
  activationTargetQueryValue: "1"
  targetQueryValue: "0"
  sslmode: "require"
authenticationRef:
  name: deployment-auth

Logs from KEDA operator

2024-03-26T13:14:46Z    INFO    Creating a new HPA  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"deployment","namespace":"namespace"}, "namespace": "namespace", "name": "deployment", "reconcileID": "8249403d-916f-49de-8dca-3f8b26dd4772", "HPA.Namespace": "namespace", "HPA.Name": "keda-hpa-deployment"}
2024-03-26T13:14:46Z    INFO    Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"deployment","namespace":"namespace"}, "namespace": "namespace", "name": "deployment", "reconcileID": "8249403d-916f-49de-8dca-3f8b26dd4772"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x32915f5]

goroutine 272 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:115 +0x1fa
panic({0x3837a00, 0x6a730a0})
    /usr/local/go/src/runtime/panic.go:884 +0x213
github.com/kedacore/keda/v2/pkg/scaling/resolver.(*HashicorpVaultHandler).token(0xc001e79db8, 0xc0010abb60?)
    /workspace/pkg/scaling/resolver/hashicorpvault_handler.go:110 +0xf5
github.com/kedacore/keda/v2/pkg/scaling/resolver.(*HashicorpVaultHandler).Initialize(0xc001e79db8, {{0x4754cc8?, 0xc001f0fd40?}, 0xc000610750?})
    /workspace/pkg/scaling/resolver/hashicorpvault_handler.go:61 +0x9e
github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthRef({0x474d110, 0xc001f0f860}, {0x4764880, 0xc000610750}, {{0x4754cc8?, 0xc001f0fd40?}, 0x40dcca?}, 0xc001e2f0c0, 0xc000995968, {0xc00091fc50, ...}, ...)
    /workspace/pkg/scaling/resolver/scale_resolvers.go:243 +0x337
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity({0x474d110, 0xc001f0f860}, {0x4764880?, 0xc000610750?}, {{0x4754cc8?, 0xc001f0fd40?}, 0x4754cc8?}, 0xc001f0fd40?, 0xc000995880, {0xc00091fc50, ...}, ...)
    /workspace/pkg/scaling/resolver/scale_resolvers.go:182 +0xc9
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1()
    /workspace/pkg/scaling/scalers_builder.go:68 +0x3f4
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers(0xc0004e6cb0, {0x474d110?, 0xc001f0f860}, 0xc001f1e3c0, 0xc000995880, {0x0, 0x0})
    /workspace/pkg/scaling/scalers_builder.go:78 +0x5d6
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache(0xc0004e6cb0, {0x474d110, 0xc001f0f860}, {0xc001f30330, 0x30}, {0x3e7de80, 0xc001ee4a00}, 0xc0020b88f8, {0x0, 0x0}, ...)
    /workspace/pkg/scaling/scale_handler.go:360 +0x71c
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache(0xc001ee4600?, {0x474d110, 0xc001f0f860}, {0x3e7de80, 0xc001ee4a00})
    /workspace/pkg/scaling/scale_handler.go:281 +0xf6
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs(0xc0006894a0, {0x474d110, 0xc001f0f860}, {{0x4754cc8?, 0xc001f0f890?}, 0xc178d0719eb81499?}, 0xc001ee4600)
    /workspace/controllers/keda/hpa.go:209 +0xda
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject(0xc0006894a0, {0x474d110?, 0xc001f0f860?}, {{0x4754cc8?, 0xc001f0f890?}, 0xc001ee8d00?}, 0xc001ee4600, 0xc0020b90e0)
    /workspace/controllers/keda/hpa.go:75 +0x66
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA(0xc0006894a0, {0x474d110, 0xc001f0f860}, {{0x4754cc8?, 0xc001f0f890?}, 0xc001f0f860?}, 0xc001ee4600, 0x0?)
    /workspace/controllers/keda/hpa.go:48 +0x1d6
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists(0xc0006894a0, {0x474d110, 0xc001f0f860}, {{0x4754cc8?, 0xc001f0f890?}, 0x4754cc8?}, 0xc001ee4600, 0xc0002c4c00?)
    /workspace/controllers/keda/scaledobject_controller.go:394 +0x356
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject(0xc0006894a0?, {0x474d110, 0xc001f0f860}, {{0x4754cc8?, 0xc001f0f890?}, 0xc000567f68?}, 0xc001ee4600, 0xc001e79b60)
    /workspace/controllers/keda/scaledobject_controller.go:254 +0x845
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile(0xc0006894a0, {0x474d110, 0xc001f0f860}, {{{0xc00091fc50?, 0x0?}, {0xc000567f68?, 0x40e0a7?}}})
    /workspace/controllers/keda/scaledobject_controller.go:177 +0x555
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x474d110?, {0x474d110?, 0xc001f0f860?}, {{{0xc00091fc50?, 0x3610820?}, {0xc000567f68?, 0x472f3f0?}}})
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006a03c0, {0x474d068, 0xc000578000}, {0x3a332e0?, 0xc0000b6220?})
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314 +0x377
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006a03c0, {0x474d068, 0xc000578000})
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222 +0x587

KEDA Version

2.11.2

Kubernetes Version

1.29

Platform

Amazon Web Services

Scaler Details

Postgresql

Anything else?

Helm Chart version: 2.11.2

caoimheharvey commented 5 months ago

Was missing the Service Account credential

zroubalik commented 5 months ago

@caoimheharvey Could you please provide a snippet of configuration of what was missing? We shouldn't crash if there's missing config, this should be fixed.

stale[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 1 month ago

This issue has been automatically closed due to inactivity.