[bitnami/redis-cluster] Add HorizontalPodAutoscaler to chart Redis-Cluster

Praveen-98cs commented 4 months ago

Name and Version

bitnami/redi-cluster

What is the problem this feature will solve?

don't have to add the HPA manually to the redis-cluster deployment.

What is the feature you are proposing to solve the problem?

I saw that a HPA has been added to the redis chart. just wanted to know, if it is possible to add a HPA to redis-cluster chart as well?

What alternatives have you considered?

Cause currently I am using a HPA created by me but it would be idea if this can be added to the chart it self.

javsalgar commented 4 months ago

Hi!

It is not clear to me that the automations would work as expected by adding-removing nodes automatically. Currently there is a job that perform this kind of update operations. Did you try adding a HPA and see what happens?

Praveen-98cs commented 4 months ago

Hi! Yes, I tried adding an HPA (Horizontal Pod Autoscaler) and observed that it did scale up and down when putting a load on the Redis cluster using a benchmark tool. The autoscaling worked, but I noticed an issue: when scaling down, the PVCs (Persistent Volume Claims) created for the additional pods were not deleted. I am also concerned about potential data loss during the scaling down process.

Could you please provide more details about the job that performs these update operations? Specifically, I would like to understand if there is an alternative method to manage scaling that addresses the issues with PVCs and ensures data integrity.

Currently there is a job that perform this kind of update operations. can you explain a bit more?

This is the HPA that I used

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: redis-hpa
  namespace: redis
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: redis-cluster
  minReplicas: 6
  maxReplicas: 12
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

Thank you!

javsalgar commented 4 months ago

Hi!

You can see the cluster update logic in this script.

https://github.com/bitnami/charts/blob/f716e12ccc46a838fee203a137e15ab5a1791b32/bitnami/redis-cluster/templates/update-cluster.yaml#L80

As you can see it performs redis cluster rebalance operations. And that would be missed with the HPA

Praveen-98cs commented 3 months ago

Hi! Thanks for the explanation. I understand your point, but in that case what can be the solution we can get if we face a resource spike and need to scale up. cause it's not possible to look at the cluster add nodes manually every time.

any suggestion? thank you in advance

javsalgar commented 3 months ago

Something that comes to mind would be triggering a bash script on a certain prometheus metrics (in case you have a prometheus instance connected). https://stackoverflow.com/questions/71419928/trigger-an-action-based-on-prometheus-metrics

Praveen-98cs commented 3 months ago

Thank you will look into this.

jayqwal commented 1 month ago

@Praveen-98cs , would you mind sharing your working solution here? Did you add some logic in your HPA to trigger script?

jayqwal commented 1 month ago

@javsalgar ,

triggering a bash script on a certain prometheus metrics

So would you suggest a new separate pod/process that monitors the metrics and call the helm upgrade ... command as suggested in the [README](triggering a bash script on a certain prometheus metrics)?

Praveen-98cs commented 1 month ago

@jayqwal for now we are going with the approach of using the helm upgrade and adding a node, but still there is an issue as we are using flux so we can't pass any commands with the helm upgrade, will update if we have found a solution.

jayqwal commented 1 month ago

@Praveen-98cs thanks for the updates! @javsalgar so this feature request is still valid as autoscaling is a very useful feature for many peoples. Can we re-open this ticket?

github-actions[bot] commented 1 month ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 1 month ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

bitnami / charts