search

romilbhardwaj / nomad

Nomad - Hierarchical Computation Framework for IoT applications
MIT License
2 stars 0 forks source link

Cross building ARM container images #1

Open romilbhardwaj opened 5 years ago

romilbhardwaj commented 5 years ago

Currently the nomad master builds only x86 images. We should also be building ARMv7 (RPi) images and use the following naming scheme:

lab11nomad/operators:<pipeline_id>_op_<operator_sequence>_<arch>

Where arch is either of "x86" or "armv8".

Since we assume the Nomad master runs on x86, we would need to cross build the ARM images from an x86 machine. This post has good directions on how we can do that: https://ownyourbits.com/2018/06/27/running-and-building-arm-docker-containers-in-x86/

This would probably require one line change change to the RPi image at docker/images/client/Dockerfile-pi and updating the build routine in the nomad master.

romilbhardwaj commented 5 years ago

Digging in a little more, seems like this works out of the box on MacOS (it has had binfmt_misc support built-in since the beta). However, it still needs to be setup on linux using qemu. That's a complicated process, but you could instead use static compiled binaries from multiarch to add this support by running one line: docker run --rm --privileged multiarch/qemu-user-static:register

We just need to run this in the nomad master when it's launched - maybe in core/master/master_launcher.py?

alvinalexander commented 5 years ago

Right now the client base image assumes x86, right? should we create another base image for ARM? if so, what image should we inherit from? I found this image on docker hub: arm64v8/ubuntu/

romilbhardwaj commented 5 years ago

I think Dockerfile-pi should be our base image, which inherits from the ARM image resin/rpi-raspbian:jessie. https://github.com/romilbhardwaj/nomad/blob/f9477fff44581c3a1c675166495fa3aa49402027/docker/images/client/Dockerfile-pi#L1

alvinalexander commented 5 years ago

okay sounds good. I am tagging the image lab11nomad/client-pi and pushing it to docker hub. Then I'll create a dockerfile 'Dockerfile.pi.operator' that inherits from cilent-pi

alvinalexander commented 5 years ago

Hm. After running the above docker container I still get this error when I try to build the rpi image:

qemu: uncaught target signal 4 (Illegal instruction) - core dumped Illegal instruction The command '/bin/sh -c conda update conda -y' returned a non-zero code: 132

I assume that is because the conda binary (berryconda in this case) is compiled for arm and is trying to execute a command that is not supported by x86.

I will debug this. Let me know if you have any thoughts on how to fix this.

alvinalexander commented 5 years ago

I think I found a solution. It turned out that the base image resin/rpi-raspbian:jessie was deprecated. resin is now balena and they moved all their images to the new namespace balenalib So I changed the base image to balenalib/raspberrypi3:jessie. This image already supports mult-arch builds (it includes qemu-static) so by first running docker run --rm --privileged multiarch/qemu-user-static:register , I was able to build the rpi image from within the ubuntu vm I am running locally.

So in theory we should be able to build the rpi image from any x86 machine running linux. Still need to check if it works with Mac OS.