GoogleCloudPlatform / flink-on-k8s-operator

[DEPRECATED] Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.
Apache License 2.0
658 stars 266 forks source link

Use Update instead of deleting component When updating a flinkcluster #360

Closed DeliangFan closed 3 years ago

DeliangFan commented 3 years ago

When update the flinkcluster spec, the operator will delete all of the old components and create the new components.

For a session flink cluster, it's a common use case to adjust the task manager replicas, and the deleting old components may result in some bad impacts for a realtime serving.

I'm also wondering why the operator must delete all of the old components rather than apply update actions.

elanv commented 3 years ago

There were plans to improve update feature allowing to choose recreating or updating deployment resources. I still have the plan but don't have enough bandwidth to do it right now, so if someone can improve it, that would be good too.

On the one hand, it seems necessary to introduce native Kubernetes support feature from Flink 1.10 or later. Flink native kubernetes support feature enables that dynamically manages cluster size according to the resources allocated to the submitted job.

DeliangFan commented 3 years ago

There were plans to improve update feature allowing to choose recreating or updating deployment resources. I still have the plan but don't have enough bandwidth to do it right now, so if someone can improve it, that would be good too.

On the one hand, it seems necessary to introduce native Kubernetes support feature from Flink 1.10 or later. Flink native kubernetes support feature enables that dynamically manages cluster size according to the resources allocated to the submitted job.

Would you please share your plans or blueprint. I'm doing the improvement for private cluster, and I would like to make a little contribution to our community.

elanv commented 3 years ago

The plan is not detailed enough to share. In my opinion, if you have any good ideas or work in progress, it would be nice to share them here.