adding scale subresource so that HPA can be used with the cluster

GoogleCloudPlatform / flink-on-k8s-operator

[DEPRECATED] Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.

Apache License 2.0

657 stars 265 forks source link

adding scale subresource so that HPA can be used with the cluster #394

Closed Padarn closed 3 years ago

Padarn commented 3 years ago

WIP - addresses https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/issues/389

functicons commented 3 years ago

/gcbrun

functicons commented 3 years ago

/gcbrun

Padarn commented 3 years ago

Thank for the review @functicons. Yep if we're happy with the approach I'll modify the user guide and see if I can put some tests together 👍

functicons commented 3 years ago

@Padarn go ahead, thanks!

Padarn commented 3 years ago

@functicons while I was working on this I realised something (I should have seen this before): We will also need to scale the jobs parallelism for it to be useful.

This gets a bit confusing because parallelism doesn't make sense unless you are using a job right now. I'd suggest for this version:

HPA required you to be using job mode
HPA will make you set parallelism per task manager

this feels a bit ugly, but without deeper thought, I'm not sure how it would be best to work. Thouhts?

functicons commented 3 years ago

@Padarn parallelism per task manager sounds like a good idea, I don't feel it is ugly, we just need to document it well.

Padarn commented 3 years ago

after battling with this for some time, I realised it was simpler to separate this PR between the HPA and the scale subresource

I've made a new PR here: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/pull/425