kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.46k stars 1.07k forks source link

AWS Elasticache TLS connection fails with EOF error #5745

Closed dmitrii-didenko closed 6 months ago

dmitrii-didenko commented 6 months ago

Report

I'm not able to get scaledobject working with AWS Elasticache with tls. I have the following scaledobject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
...
...
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          policies:
          - periodSeconds: 30
            type: Percent
            value: 20
          stabilizationWindowSeconds: 300
  cooldownPeriod: 300
  maxReplicaCount: 5
  minReplicaCount: 2
  pollingInterval: 30
  scaleTargetRef:
    apiVersion: apps/v1
    envSourceContainerName: app
    kind: Deployment
    name: foo
  triggers:
  - metadata:
      activationListLength: "0"
      addressFromEnv: SOME_ENV_VAR_1
      databaseIndex: "7"
      enableTLS: "true"
      unsafeSsl: "true"
      listLength: "1"
      listName: queues:export
      passwordFromEnv: SOME_ENV_VAR_2
    type: redis

Note, both option enableTLS and unsafeSsl are set to expected values. I've tested additionally with redis-cli on the same node and it works perfectly fine, so it's not the networking issue. The command I used

redis-cli -h host --tls --insecure

Elasticache uses TLS 1.2 https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/infrastructure-security.html so the KEDA as well https://github.com/kedacore/charts/blob/207bab7e7639729daf131c16b2a17deb6e888adc/keda/values.yaml#L545

What else can be done? Is it a bug or I'm missing something.

Expected Behavior

No errors with scaled object

Actual Behavior

Error with scaled object

Steps to Reproduce the Problem

  1. Prepare AWS infra and create scaledobject

Logs from KEDA operator

2024-04-25T10:27:42Z    ERROR    failed to ensure HPA is correctly created for ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"****","namespace":"staging"}, "namespace": "staging", "name": "******", "reconcileID": "0e1ee205-daf3-42c4-993e-34077a1ce6fc", "error": "connection to redis failed: EOF"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
    /workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

KEDA Version

2.13.1

Kubernetes Version

1.29

Platform

Amazon Web Services

Scaler Details

Redis

Anything else?

No response

dmitrii-didenko commented 6 months ago

The issue appairs to be on AWS side. Somehow, Elasticache doesn't work properly with TLS1.3. We had the following options configured on KEDA helm chart:

    http:
      minTlsVersion: TLS13

When we changed to TLS12 everything started to working as expected.