search

dask / dask-kubernetes

Native Kubernetes integration for Dask
https://kubernetes.dask.org
BSD 3-Clause "New" or "Revised" License
312 stars 148 forks source link

Add ARM64 support for both dask and dask operator docker images #919

Open VanWolf29 opened 2 weeks ago

VanWolf29 commented 2 weeks ago

Hello,

I've been trying to implement this in a Kubernetes cluster in AWS, but I prefer to use Graviton (ARM64) instances due to its benefits and optimal cost. However, I've noticed that there are not images supporting this architecture.

I could also volunteer for doing it (if possible) but I'd need a bit of orientation for the CI building the images.

Thanks!

bstadlbauer commented 2 weeks ago

@jacobtomlinson Please correct me if I'm wrong here, but images are built using chartpress (here) and chartpress should support ARM builds (https://github.com/jupyterhub/binderhub/pull/1422/) with using 

chartpress \
    --builder docker-buildx \
    --platform linux/amd64 --platform linux/arm64

The question would be more around whether we'd want to maintain another image. Happy to help testing here as I've got an ARM machine in case that's helpful.

jacobtomlinson commented 2 weeks ago

Yep no issues from me to add the other build.

ryanolee commented 3 days ago

There are some minor issues building on QEMU specifically @bstadlbauer @jacobtomlinson (Currently used in CI) https://github.com/dask/dask-docker/commit/f82d231e95b4886b93dc1104815a36312353f708

Mainly that builds will currently hang forever if you do not build on a native hardware without the changes from that commit. Beyond that can confirm arm64 docker image work as expected. (see: https://mamba.readthedocs.io/en/latest/user_guide/troubleshooting.html#hangs-during-install-in-qemu)

(Falls inline with https://github.com/dask/dask-docker/pull/333)