search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.56k stars 1.08k forks source link

ActiveMQ Artemis Scaler - connection refused using exact same scaled object working on 2.13.1 but not on 2.14 or higher #6333

Open GrayBlueJay opened 1 week ago

GrayBlueJay commented 1 week ago

Report

After upgrading from 2.13.1 to 2.14 (also tried on 2.16) existing scaled objects that worked on 2.13.1 now get a connection refused error in 2.14 and 2.16 (have not tested 2.15).

When rolling back to 2.13.1 the scaled objects are able to connect without issue.

Trigger definition:

  Triggers:
    Authentication Ref:
      Name:  trigger-auth-kedartemis
    Metadata:
      Broker Address:       DEV_File_Notify
      Broker Name:          tscore-artemis
      Management Endpoint:  192.168.62.224:8161
      Queue Length:         100
      Queue Name:           DEV_File_Notify
    Type:                   artemis-queue

keda-operator logs for scaled object in version 2.16:

2024-11-14T20:17:43Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:17:43Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:17:43Z    ERROR   scale_handler   error getting scale decision    {"scaledObject.Namespace": "dev", "scaledObject.Name": "tscorefilenotify", "scaler": "artemisScaler", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:13Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:13Z    ERROR   artemis_queue_scaler    Unable to access the artemis management endpoint        {"type": "ScaledObject", "namespace": "dev", "name": "tscorefilenotify", "managementEndpoint": "192.168.62.224:8161", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:13Z    ERROR   scale_handler   error getting scale decision    {"scaledObject.Namespace": "dev", "scaledObject.Name": "tscorefilenotify", "scaler": "artemisScaler", "error": "Get \"http://192.168.62.224:8161/console/jolokia/read/org.apache.activemq.artemis:broker=%22tscore-artemis%22,component=addresses,address=%22DEV_File_Notify%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22DEV_File_Notify%22/MessageCount\": dial tcp 192.168.62.224:8161: connect: connection refused"}
2024-11-14T20:18:43Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "bffbdbf3-9c3a-4cc0-84be-e15b442a16a2"}
2024-11-14T20:18:43Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "bffbdbf3-9c3a-4cc0-84be-e15b442a16a2", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}

keda-operator logs after rolling back to 2.13.1:

2024-11-14T20:33:32Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Updated HPA according to ScaledObject   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "HPA.Namespace": "dev", "HPA.Name": "keda-hpa-tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Initializing Scaling logic according to ScaledObject Specification      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    scaleexecutor   Successfully updated ScaleTarget        {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 0, "New Replicas Count": 1}
2024-11-14T20:33:35Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3"}
2024-11-14T20:33:36Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:39:32Z    INFO    scaleexecutor   Successfully set ScaleTarget replicas count to ScaledObject minReplicaCount     {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 1, "New Replicas Count": 0}

Please let me know if you need additional information

Expected Behavior

I do not see any changes documented in the scaler for Artemis or note any breaking changes in the 2.14 release notes and therefore expect that the scaler would continue to work as it did in 2.13.1

Actual Behavior

Connection refused error being thrown when using an artemis scaler that worked in 2.13.1 but now no longer works in version >= 2.14

Steps to Reproduce the Problem

  1. Create artemis scaler in version < 2.14
  2. Confirm artemis scaler works
  3. Upgrade Keda to version >= 2.14
  4. Observer connection refused error

Logs from KEDA operator

2024-11-14T20:33:32Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Updated HPA according to ScaledObject   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636", "HPA.Namespace": "dev", "HPA.Name": "keda-hpa-tscorefilenotify"}
2024-11-14T20:33:32Z    INFO    Initializing Scaling logic according to ScaledObject Specification      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "ea9191c1-df3a-4d7d-a5e1-94e040b0e636"}
2024-11-14T20:33:32Z    INFO    scaleexecutor   Successfully updated ScaleTarget        {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 0, "New Replicas Count": 1}
2024-11-14T20:33:35Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3"}
2024-11-14T20:33:36Z    INFO    Detected resource targeted for scaling  {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"tscorefilenotify","namespace":"dev"}, "namespace": "dev", "name": "tscorefilenotify", "reconcileID": "071d2235-44cd-4608-8cc9-337d7651a6c3", "resource": "apps/v1.Deployment", "name": "tscorefilenotify"}
2024-11-14T20:39:32Z    INFO    scaleexecutor   Successfully set ScaleTarget replicas count to ScaledObject minReplicaCount     {"scaledobject.Name": "tscorefilenotify", "scaledObject.Namespace": "dev", "scaleTarget.Name": "tscorefilenotify", "Original Replicas Count": 1, "New Replicas Count": 0}

KEDA Version

2.13.1

Kubernetes Version

1.30

Platform

Amazon Web Services

Scaler Details

ActiveMq Artemis

Anything else?

No response