search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.46k stars 1.07k forks source link

I/O timeout unable to connect to Elasticache #4880

Closed madyarini closed 1 year ago

madyarini commented 1 year ago

Report

I'm facing similar issue as https://github.com/kedacore/keda/issues/2903 but I'm not sure how to reopen that issue, hence creating new issue I'm unable to connect to Elasticache due to I/O timeout my Elasticache has no Redis AUTH, it's secured by security groups. im ensured that my security groups are properly configured as in the kubernetes namespace where i placed my scaledobject & triggerauthentication, the pods are able to connect to Redis. 

kubectl describe scaledobject -n mynamespace
Name:         ..
Namespace:   ..
Labels:       app.kubernetes.io/managed-by=Helm
              scaledobject.keda.sh/name=..
Annotations:  meta.helm.sh/release-name: ..
              meta.helm.sh/release-namespace: ..
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2023-08-15T12:23:19Z
  Finalizers:
    finalizer.keda.sh
  Generation:        1
  Resource Version:  515637847
  UID:               1368e2b6-075c-4b28-9843-b1594489c6dd
Spec:
  Advanced:
    Horizontal Pod Autoscaler Config:
      Behavior:
        Scale Down:
          Policies:
            Period Seconds:              60
            Type:                        Percent
            Value:                       50
          Stabilization Window Seconds:  300
        Scale Up:
          Policies:
            Period Seconds:              30
            Type:                        Percent
            Value:                       100
          Stabilization Window Seconds:  30
    Restore To Original Replica Count:   true
  Max Replica Count:                     5
  Min Replica Count:                     1
  Scale Target Ref:
    Name: ..
  Triggers:
    Authentication Ref:
      Name:  my-trigger-auth
    Metadata:
      List Length:  15
      List Name:    ..
    Type:           redis
Status:
  Conditions:
    Message:               Failed to ensure HPA is correctly created for ScaledObject
    Reason:                ScaledObjectCheckFailed
    Status:                False
    Type:                  Ready
    Message:               ScaledObject check failed
    Reason:                UnkownState
    Status:                Unknown
    Type:                  Active
    Status:                Unknown
    Type:                  Fallback
  Original Replica Count:  1
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason                   Age                   From           Message
  ----     ------                   ----                  ----           -------
  Warning  KEDAScalerFailed         46m                   keda-operator  error parsing redis metadata: no address or host given. address should be in the format of host:port or you should set the host/port values
  Warning  KEDAScalerFailed         45m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:33830->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         45m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:45232->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         45m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:57852->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         45m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:38124->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         45m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:48790->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         44m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:48690->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         44m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:39036->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         44m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:57424->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         31m (x9 over 44m)     keda-operator  (combined from similar events): connection to redis failed: read tcp 172.31.103.20:43306->10.10.138.178:6379: i/o timeout
  Warning  KEDAScalerFailed         20m                   keda-operator  connection to redis failed: read tcp 172.31.103.20:42138->10.10.138.178:6379: i/o timeout
  Warning  ScaledObjectCheckFailed  3m48s (x20 over 46m)  keda-operator  Failed to ensure HPA is correctly created for ScaledObject
  Warning  KEDAScalerFailed         3m48s                 keda-operator  connection to redis failed: read tcp 172.31.103.20:43832->10.10.138.178:6379: i/o timeout

triggerauthentication

kubectl describe triggerauthentication -n mynamespace
Name:         my-trigger-auth
Namespace:    ..
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: ..
              meta.helm.sh/release-namespace: ..
API Version:  keda.sh/v1alpha1
Kind:         TriggerAuthentication
Metadata:
  Creation Timestamp:  2023-08-15T09:38:43Z
  Generation:          1
  Resource Version:    515566941
  UID:                 d52855e8-565f-4f96-8719-138a2d148f6b
Spec:
  Secret Target Ref:
    Key:        KEDA_REDIS_ADDRESS
    Name:       ..
    Parameter:  address
Events:
  Type    Reason                      Age   From           Message
  ----    ------                      ----  ----           -------
  Normal  TriggerAuthenticationAdded  27s   keda-operator  New TriggerAuthentication configured

I have also tried adding parameter password on the trigger auth with blank string, but it's the same issue. Thanks

Expected Behavior

Able to connect to elasticache

Actual Behavior

there are error events:

Warning KEDAScalerFailed 45m keda-operator connection to redis failed: read tcp 172.31.103.20:33830->10.10.138.178:6379: i/o timeout

Steps to Reproduce the Problem

keda version: 2.8.4

Logs from KEDA operator

example

KEDA Version

Other

Kubernetes Version

1.25

Platform

Amazon Web Services

Scaler Details

Redis List

Anything else?

No response

JorTurFer commented 1 year ago

Hi I have not experience with "security groups" but I guess that it's something like, if you are in this namespace, you can call to here, if you aren't you can't. Is something like that? If it's that, you need to ensure that the namespace where KEDA is, has access too. KEDA executes all the requests from the namespace where KEDA is (even if the resources are in other namespaces).

madyarini commented 1 year ago

sorry, nvm, turns out i typo enableTls. should be enableTLS