search

kubeflow / spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Apache License 2.0
2.79k stars 1.37k forks source link

Multi-version support in master #610

Open akhurana001 opened 5 years ago

akhurana001 commented 5 years ago

Hi @liyinan926, I am creating this issue to track any open items blocking merging the multi-version-support branch back into master.

The unit-tests were fixed as part of https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/526 and let's use this issue to track if there is any other work required for this. Happy to pick-up if there are any open items.

liyinan926 commented 5 years ago

@pmkc we plan to bump the version to v1beta2 soon with #578. The multi-version-support branch contains backward-incompatible changes so likely will go into v1beta3.

akhurana001 commented 5 years ago

Thanks @liyinan926 . https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/578 is merged. Let me know if there is anything else which needs to be done before we can merge the multi-version as well.

liyinan926 commented 5 years ago

I would like to see more users testing this before making it the default in master. We are also doing some testing on our end. I suggest baking it a bit more before merging it back to master. I think we can target mid of Q4 to get it in and released in v1beta3. Does that make sense?

liyinan926 commented 5 years ago

@akhurana001 I rebased the branch multi-version-support onto the latest master. If possible, please test using the latest version, which uses the new v1beta2 version of the API.

akhurana001 commented 5 years ago

@liyinan926 Thanks for merging. I can definitely run some tests on the latest version. I do want to minimize the number of version migrations for our prod as it becomes a little tricky to migrate to a new version with live workloads due to potential downtime etc. I was also looking at https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/ to see if it makes sense for us to support version conversions.

Overall, We have been running the multi-version-support branch for 6+ months now in production and it has been fairly stable for us.

urihoenig commented 4 years ago

@akhurana001 / @liyinan926 - Is there any corresponding helm chart issue/PR to add job permissions to the clusterrole? (I couldn't find one and thought I might be missing something)

akhurana001 commented 4 years ago

@urihoenig That's correct, The helm chart currently is based on the master branch/image of the operator. @liyinan926 Let's sync-up (maybe a meeting ?) to see how we can make progress on merging multi-version into master. I can also create a PR to update the helm chart as part of that.

akhurana001 commented 4 years ago

@liyinan926 circling back on this to see if we can get this done . Happy to chat/help if there are any open issues/tasks blocking this.

liyinan926 commented 4 years ago

@akhurana001 thanks for checking on this. We will start creating PRs against the master soon after we get the next release out, which is blocked by the release of Spark 2.4.5.

akhurana001 commented 4 years ago

Thanks @liyinan926 . Let me know if there is anything i can help with.

urihoenig commented 4 years ago

Hi @liyinan926 - when is this expected? Is the current multi-version-support rebased on 1.1.0? (which spark version does it require)

kumare3 commented 4 years ago

@liyinan926 how should we make it such that we can merge this into master? The problem is we are running this in production and we want to continue adding features. Please help us understand what is required.

liyinan926 commented 4 years ago

Nothing major required for this AFAIK. Will start introducing a new API version v1 for this.

urihoenig commented 4 years ago

@liyinan926 - Is this branch still planned to be merged into master? Will it support spark 2 & 3 in parallel?

github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.