Action items for adoption of Spark Kubernetes Operator in Kubeflow

andreyvelich commented 10 months ago

We have to follow these action items to transfer Spark Operator to Kubeflow.

[x] Get approval from Kubeflow Steering Committee to accept Spark Operator as a Kubeflow core component.
[x] Transfer https://github.com/GoogleCloudPlatform/spark-on-k8s-operator repository to https://github.com/kubeflow/spark-operator.
[x] Configure tide for the new repository.
[x] Add Spark Operator maintainers: @vara-bonthu @yuchaoran2011 @mwielgus to the Kubeflow Org: https://github.com/kubeflow/internal-acls?tab=readme-ov-file#joining-kubeflow-github-organization
[x] Create OWNERS file with the maintainers: @vara-bonthu @yuchaoran2011 @mwielgus in the new Kubeflow repository.
[x] Modify the contributing doc to follow Kubeflow contributing guidelines (e.g. licence agreement if that is required).
[x] Attend to repo migration side effect in https://github.com/kubeflow/spark-operator/issues/1926.
[x] Update docs with new helm charts link. (For example, https://github.com/kubeflow/spark-operator/tree/master?tab=readme-ov-file#installation)
[x] Fix GitHub actions for helm charts.
[x] Add Docker Hub Token to Spark Operator repo
[x] Fix GitHub actions for container registry (Kubeflow DockerHub).
[x] Migrate Spark Operator docs to Kubeflow Website..
[x] Improve the release process: https://github.com/kubeflow/spark-operator/issues/1975

Please let me know if I missed something @jbottum @james-jwu @zijianjoy. Excited to see this moving forward 🎉

jbottum commented 10 months ago

this looks complete, let's move this forward

zijianjoy commented 10 months ago

To share more information about repo config and tide config: We will follow the instruction in https://github.com/kubeflow/community/blob/master/repository-setup.md#repository-configuration to set up this new repo. Feel free to let me know if you need my help on the admin side of steps.

andreyvelich commented 10 months ago

Great, thank you, James!

nttq1sub commented 9 months ago

@andreyvelich any update on this

andreyvelich commented 9 months ago

@andreyvelich any update on this

We are working with @zijianjoy and Google team to transfer Spark Operator repository to Kubeflow GitHub. I will share updates when we have more information. Feel free to join Kubeflow Community Call today to hear about the latest updates.

vikas-saxena02 commented 9 months ago

@andreyvelich do we know when will the work start on this?

andreyvelich commented 9 months ago

@andreyvelich do we know when will the work start on this?

Once we transfer the Spark Operator repository to Kubeflow the team will start work on the items to make a release. In the meantime, it would be nice if you @vikas-saxena02 could help to answer some user questions in the #kubeflow-spark-operator Slack Channel: https://kubeflow.slack.com/archives/C06627U3XU3

gangahiremath commented 9 months ago

I am not able to sign in to kubeflow slack. Please share the steps. Thank you

andreyvelich commented 9 months ago

I am not able to sign in to kubeflow slack. Please share the steps. Thank you

@gangahiremath Please follow this link to join Kubeflow Slack: https://www.kubeflow.org/docs/about/community/#kubeflow-slack

vikas-saxena02 commented 8 months ago

sure @andreyvelich Quick question is there work planned to run spark jobs from kubeflow notebooks (jupyter notebooks)?

andreyvelich commented 8 months ago

sure @andreyvelich Quick question is there work planned to run spark jobs from kubeflow notebooks (jupyter notebooks)?

We have this item in our design doc: https://docs.google.com/document/d/1rCPEBQZPKnk0m7kcA5aHPf0fISl0MTAzsa4Wg3dfs5M/edit#heading=h.sb3itb5lb4jl. So if you want to start this effort that would be nice.

zijianjoy commented 7 months ago

The repo is migrated and renamed to https://github.com/kubeflow/spark-operator.

zijianjoy commented 7 months ago

This link is helpful for setting up tide and prow: https://github.com/kubeflow/community/blob/master/repository-setup.md#setup-prow

sid-habu commented 7 months ago

With this move what's the name of the helm repo? The existing repo doesn't work

helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator

Error: looks like "https://googlecloudplatform.github.io/spark-on-k8s-operator" is not a valid chart repository or cannot be reached: failed to fetch https://googlecloudplatform.github.io/spark-on-k8s-operator/index.yaml : 404 Not Found

sid-habu commented 7 months ago

With this move what's the name of the helm repo? The existing repo doesn't work

helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator

Error: looks like "https://googlecloudplatform.github.io/spark-on-k8s-operator" is not a valid chart repository or cannot be reached: failed to fetch https://googlecloudplatform.github.io/spark-on-k8s-operator/index.yaml : 404 Not Found

Nvm, I found the new repo link

helm repo add spark-operator https://kubeflow.github.io/spark-operator/

zijianjoy commented 7 months ago

Hello team, you might want to attend to repo migration side effect in https://github.com/kubeflow/spark-operator/issues/1926.

zevisert commented 2 months ago

Update to the spark-website repo: https://github.com/apache/spark-website/pull/553

andreyvelich commented 2 months ago

Since all of the items have been completed, we can close it. Thanks everyone for your contributions 🎉

kubeflow / spark-operator

Action items for adoption of Spark Kubernetes Operator in Kubeflow #1928