search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.38k stars 1.06k forks source link

Keda Version 2.13 Installation via YAML Files Error #5444

Closed UGS-DanielSoriano closed 5 months ago

UGS-DanielSoriano commented 8 months ago

Report

I am encountering error in installing the provided keda-2.13.0.yaml file. The pods are encountering errors and not running. The keda operator is logging an error related to open /certs/ca.crt: permission denied

Expected Behavior

Installation should work and pods must be on running state.

Actual Behavior

Pods are on error state

Steps to Reproduce the Problem

  1. kubectl apply -f keda-2.13.0.yaml

Logs from KEDA operator

2024/01/30 06:28:18 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2024-01-30T06:28:18Z    INFO    setup   Starting manager
2024-01-30T06:28:18Z    INFO    setup   KEDA Version: 2.13.0
2024-01-30T06:28:18Z    INFO    setup   Git Commit: 367fcd32f1e72be504ada53165d07c33d75fc0f7
2024-01-30T06:28:18Z    INFO    setup   Go Version: go1.21.6
2024-01-30T06:28:18Z    INFO    setup   Go OS/Arch: linux/amd64
2024-01-30T06:28:18Z    INFO    setup   Running on Kubernetes 1.28+ {"version": "v1.28.4-eks-8cb36c9"}
2024-01-30T06:28:18Z    INFO    controller-runtime.metrics  Starting metrics server
2024-01-30T06:28:18Z    INFO    controller-runtime.metrics  Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-01-30T06:28:18Z    INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
I0130 06:28:18.852090       1 leaderelection.go:250] attempting to acquire leader lease keda/operator.keda.sh...
I0130 06:28:34.618066       1 leaderelection.go:260] successfully acquired lease keda/operator.keda.sh
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v2.HorizontalPodAutoscaler"}
2024-01-30T06:28:34Z    INFO    Starting Controller {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
2024-01-30T06:28:34Z    INFO    Starting Controller {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
2024-01-30T06:28:34Z    INFO    Starting Controller {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource", "source": "kind source: *v1alpha1.CloudEventSource"}
2024-01-30T06:28:34Z    INFO    Starting Controller {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
2024-01-30T06:28:34Z    INFO    Starting Controller {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2024-01-30T06:28:34Z    INFO    cert-rotation   starting cert rotator controller
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-01-30T06:28:34Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-01-30T06:28:34Z    INFO    Starting Controller {"controller": "cert-rotator"}
2024-01-30T06:28:34Z    INFO    cert-rotation   no cert refresh needed
2024-01-30T06:28:34Z    INFO    cert-rotation   certs are ready in /certs
2024-01-30T06:28:34Z    INFO    Starting workers    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "worker count": 1}
2024-01-30T06:28:34Z    INFO    Starting workers    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "worker count": 1}
2024-01-30T06:28:34Z    INFO    Starting workers    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "worker count": 5}
2024-01-30T06:28:34Z    INFO    Starting workers    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource", "worker count": 1}
2024-01-30T06:28:34Z    INFO    Starting workers    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "worker count": 1}
2024-01-30T06:28:34Z    INFO    Starting workers    {"controller": "cert-rotator", "worker count": 1}
2024-01-30T06:28:34Z    INFO    cert-rotation   no cert refresh needed
2024-01-30T06:28:34Z    INFO    cert-rotation   Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-01-30T06:28:34Z    INFO    cert-rotation   Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-01-30T06:28:34Z    INFO    cert-rotation   no cert refresh needed
2024-01-30T06:28:34Z    INFO    cert-rotation   Ensuring CA cert    {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-01-30T06:28:34Z    INFO    cert-rotation   Ensuring CA cert    {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-01-30T06:28:36Z    INFO    cert-rotation   CA certs are injected to webhooks
2024-01-30T06:28:36Z    INFO    Stopping and waiting for non leader election runnables
2024-01-30T06:28:36Z    INFO    Stopping and waiting for leader election runnables
2024-01-30T06:28:36Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "cert-rotator"}
2024-01-30T06:28:36Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2024-01-30T06:28:36Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}
2024-01-30T06:28:36Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2024-01-30T06:28:36Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2024-01-30T06:28:36Z    INFO    Shutdown signal received, waiting for all workers to finish {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2024-01-30T06:28:36Z    INFO    cert-rotation   stopping cert rotator controller
2024-01-30T06:28:36Z    INFO    All workers finished    {"controller": "cert-rotator"}
2024-01-30T06:28:36Z    INFO    All workers finished    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2024-01-30T06:28:36Z    INFO    All workers finished    {"controller": "cloudeventsource", "controllerGroup": "eventing.keda.sh", "controllerKind": "CloudEventSource"}
2024-01-30T06:28:36Z    INFO    All workers finished    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2024-01-30T06:28:36Z    INFO    All workers finished    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2024-01-30T06:28:36Z    INFO    All workers finished    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2024-01-30T06:28:36Z    INFO    Stopping and waiting for caches
2024-01-30T06:28:36Z    INFO    Stopping and waiting for webhooks
2024-01-30T06:28:36Z    INFO    Stopping and waiting for HTTP servers
2024-01-30T06:28:36Z    INFO    shutting down server    {"kind": "health probe", "addr": "[::]:8081"}
2024-01-30T06:28:36Z    INFO    controller-runtime.metrics  Shutting down metrics server with timeout of 1 minute
2024-01-30T06:28:36Z    INFO    Wait completed, proceeding to shutdown the manager
2024-01-30T06:28:36Z    ERROR   setup   problem running manager {"error": "open /certs/ca.crt: permission denied"}
main.main
    /workspace/cmd/operator/main.go:319
runtime.main
    /usr/local/go/src/runtime/proc.go:267

KEDA Version

2.13.0

Kubernetes Version

Other

Platform

Amazon Web Services

Scaler Details

No response

Anything else?

AWS EKS 1.28

JorTurFer commented 8 months ago

Hello I can't reproduce the issue. If I execute kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.13.0/keda-2.13.0.yaml it works as expected. The operator generates the cert and restarts as part of the initial configuration, and then it works well.

Has it been a transient error or it's happening all the time?

stale[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] commented 5 months ago

This issue has been automatically closed due to inactivity.

guymers commented 3 months ago

I had the same issue in 2.14.0, removing defaultMode made it work.

I also had to mount certificates at /cert in addition to /certs for keda-metrics-apiserver to start.