search

kedacore / keda

KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
https://keda.sh
Apache License 2.0
8.5k stars 1.08k forks source link

Octopus Worker Scaler #3808

Open Eldarrin opened 2 years ago

Eldarrin commented 2 years ago

Proposal

A docker container running an Octopus Tentacle which can be scaled according to queue requirements; focusing on using kubernetes to scale Octopus Workers

Scaler Source

Octopus Task Queue length

Scaling Mechanics

Queue Length > metric initiate a new worker in a pool

Authentication Source

Octopus API Key

Anything else?

Happy to write it if it gets enough appreciation. :)

tomkerkhove commented 2 years ago

Do you have a URL to that product? Never heard of it, but thanks for the suggestion!

Are you interested in contributing it?

Eldarrin commented 2 years ago

Sure: https://octopus.com/

its used a lot in regulated industries as it has very good audit logs for deployments. I've been playing with it at my current contract and they don't really get scaling and I thought it'd be a good fit.

I'm happy to write the initial code and test; as long as someone reviews my dirty hack skills :D

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

tomkerkhove commented 1 year ago

@Eldarrin Are you still interested in contributing?

Eldarrin commented 1 year ago

Picking this up again :)

Eldarrin commented 1 year ago

@JorTurFer @tomkerkhove I have an interesting issue here which I suspect you have hit before.

The worker will come up with the scaler as decided by the queue length, but the worker may or may not be actually executing a task as that may be a different worker. Is there a simple way already in KEDA for detecting the running vs executing state (ready/live probe etc).

Any thoughts? I'm actually thinking about the same for the azure pipeline scalers as they exhibit the same behaviours.

Basically I don't want to SIGTERM an executing process

JorTurFer commented 1 year ago

@JorTurFer @tomkerkhove I have an interesting issue here which I suspect you have hit before.

The worker will come up with the scaler as decided by the queue length, but the worker may or may not be actually executing a task as that may be a different worker. Is there a simple way already in KEDA for detecting the running vs executing state (ready/live probe etc).

Any thoughts? I'm actually thinking about the same for the azure pipeline scalers as they exhibit the same behaviours.

Basically I don't want to SIGTERM an executing process

It's an excellent point, but I think that we should follow the same approach here as we follow in Azure Pipelines or GH Actions scalers: The scaler should be used with ScaledJob to ensure that an active worker isn't killed.

KEDA shouldn't know anything from the scaled workload because KEDA doesn't manage them. Apart from that design perspective, KEDA can't decide which instance is removed because KEDA exposes the metric, and it's the k8s control plane who decides the killed instances

Eldarrin commented 1 year ago

Ok, linking the octopus issue. https://github.com/OctopusDeploy/OctopusTentacle/issues/458

Not sure what to do till I get that resolved.

JorTurFer commented 1 year ago

Maybe using lifecycle you can improve the things https://keda.sh/docs/2.10/concepts/scaling-deployments/#leverage-the-container-lifecycle I mean, if you add some stop command(to "drain" the tentacle) with a huge timeout, maybe you can solve or mitigate the issue

Exnadella commented 1 year ago

Hi @tomkerkhove @JorTurFer

I am using the octopus deploy and would like to auto-scale the tentacle based on the KEDA and I am interested in testing this. I am using OCI-OKE cluster and tentacle is deployed, but the workers are inactive and interested to get this up and running and use KEDA to scale the workers inside the workerpool.

JorTurFer commented 1 year ago

@Eldarrin , are you still interested in contribute with this? If not, maybe @Exnadella can helps with it

Eldarrin commented 1 year ago

@JorTurFer @Exnadella The problem still exists that the scaler can kill itself without warning (possibly mid job). There is a bit of a dirty hack that's possible is you have a kill blocker that detects ps ax | grep calamari but nothing neat. I have raised it with Octopus but haven't seen anything on their roadmap to run tentacles as jobs.

JorTurFer commented 1 year ago

Thanks for the update :heart: