search

kubeflow / spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Apache License 2.0
2.79k stars 1.38k forks source link

Add support for arm64 #915

Closed holdenk closed 1 month ago

holdenk commented 4 years ago

Given the proliferation of arm based cloud instances and the nvidia arm platforms it would make sense to support running on arm64.

lucaskjaero commented 4 years ago

I also have this issue. Thanks for raising it @holdenk .

vl-shopback commented 2 years ago

any news?

ljy2010a commented 2 years ago

any news?

shechostr commented 2 years ago

any news?

gonzalezzfelipe commented 2 years ago

Hey there, adding my 2 cents here, because I just got it working. It looks like this has no more support so I'll tell you the steps I did to get it working

. Change this line from GOARCH=amd64 to GOARCH=arm64.

. Run docker buildx build --platform linux/arm64 --build-arg SPARK_IMAGE=apache/spark:v3.2.2 -t your-repo/your-image:v3.2.2-arm .

I'm currently testing that no problems arise from this, but it seems like these are the only change requirements. Then update the values of the helmchart to use this image and that should be it.

mfidemraizer commented 1 year ago

@gonzalezzfelipe I've done so, and I'm getting the following error once container is started:

exec: "/usr/bin/entrypoint.sh": permission denied: unknown

ZachThill commented 1 year ago

@mfidemraizer, I was just working on this trying to deploy spark operator on an AWS Graviton cluster. After successfully building the image and attempting to run the container on docker desktop, I got the same error as you: Screenshot 2023-06-22 at 8 24 43 AM To fix this issue, I added a new RUN line to my dockerfile that changes the permissions of the entrypoint.sh file and was able to initialize the container successfully:

COPY spark-operator/entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["/usr/bin/entrypoint.sh"]
manojmukkamala commented 10 months ago

tested spark 3.5.0 on RPi microk8s today using docker image and can confirm it is working fine by following the instructions above.

stefanbethke commented 1 month ago

Given that #1735 would likely be solved by an arm64 image, is there any reason the project does not publish arm64 images?

missedone commented 1 month ago

I'm new to this community, but it looks like the arm64 image of spark-operator has been available on Docker Hub for a while. I have successfully run the operator, webhook, and spark driver/exec pods on AWS graviton instances.

image
jacobsalway commented 1 month ago

/close

The images are built for both amd64 and arm64 in the release workflow.

google-oss-prow[bot] commented 1 month ago

@jacobsalway: Closing this issue.

In response to [this](https://github.com/kubeflow/spark-operator/issues/915#issuecomment-2389748654): >/close > >The images are built for both amd64 and arm64 [in the workflow](https://github.com/kubeflow/spark-operator/blob/master/.github/workflows/release.yaml#L77-L79). Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.