Docker for Mac Edge release 2.3.2.0 re-adds device-mapper, update README when released

[rossmckelvie]

In this comment on docker/for-mac#4549 mac issue related to device-mapper not being available, I noticed that the Edge build 2.3.2.0 of Docker for Mac contains the dependencies necessary to run the docker workflow on Mac again! 🙌

I have installed the Docker for Mac beta and confirmed that I am able to build successfully again and produce images. There are intermittent failures on the kpartx command related to device-mapper, but I have only needed to restart the build at most two times for it to work, and the majority of the time I am able to build images without error.

When these changes make it into the stable release channel, I plan to open a PR to correct the README changes I made a few months back regarding the device-mapper dependency.

For reference, here is my Makefile that I'm using in my project workspace, which doesn't require cloning this repo at all either:

clean:
    rm -f output-arm-image/image

docker:
    docker pull docker.pkg.github.com/solo-io/packer-builder-arm-image/packer-builder-arm

img-base: docker clean
    docker run \
        --rm \
        --privileged \
        -v ${PWD}:/build:ro \
        -v ${PWD}/packer_cache:/build/packer_cache \
        -v ${PWD}/output-arm-image:/build/output-arm-image \
        -e PACKER_CACHE_DIR=/build/packer_cache \
    docker.pkg.github.com/solo-io/packer-builder-arm-image/packer-builder-arm build packer/base.json
    mv output-arm-image/image output-arm-image/base.img

img-sdr: docker clean
    docker run \
        --rm \
        --privileged \
        -v ${PWD}:/build:ro \
        -v ${PWD}/packer_cache:/build/packer_cache \
        -v ${PWD}/output-arm-image:/build/output-arm-image \
        -e PACKER_CACHE_DIR=/build/packer_cache \
    docker.pkg.github.com/solo-io/packer-builder-arm-image/packer-builder-arm build packer/rtl-sdr.json
    mv output-arm-image/image output-arm-image/sdr.img

I have a few other images that I build on top of the base, but implementing it this way has allowed me to save a lot of time not having to configure the locale or updating all of the apt dependencies before each unique build image. Similar to docker's layers during build, this saves me a lot of time when the application-specific images fail due to the kpartx or loop errors.

[rossmckelvie]

Here is one of the intermittent errors I get related to installing qemu-user-static, rerunning the same build command and/or restarting docker is resolving the issue for me. Running on Docker for Mac (Edge) 2.3.2.0.

running /bin/packer
arm-image: output will be in this color.

==> arm-image: Retrieving Image
==> arm-image: Trying http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-05-28/2020-05-27-raspios-buster-lite-armhf.zip
==> arm-image: Trying http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-05-28/2020-05-27-raspios-buster-lite-armhf.zip?checksum=sha256%3Af5786604be4b41e292c5b3c711e2efa64b25a5b51869ea8313d58da0b46afc64
==> arm-image: http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-05-28/2020-05-27-raspios-buster-lite-armhf.zip?checksum=sha256%3Af5786604be4b41e292c5b3c711e2efa64b25a5b51869ea8313d58da0b46afc64 => /build/packer_cache/c193b0ba86d10e5ce653a34e195a72fd1e36fd79.iso
==> arm-image: Copying source image.
==> arm-image: Image is a zip file.
==> arm-image: Unzipping 2020-05-27-raspios-buster-lite-armhf.img
    arm-image: Speed:   63.14 MB/s
    arm-image: Progress: 17.03%
    arm-image: Speed:   42.46 MB/s
    arm-image: Progress: 28.49%
    arm-image: Speed:   39.16 MB/s
    arm-image: Progress: 39.06%
    arm-image: Speed:   42.34 MB/s
    arm-image: Progress: 50.46%
    arm-image: Speed:   39.15 MB/s
    arm-image: Progress: 61.02%
    arm-image: Speed:   46.73 MB/s
    arm-image: Progress: 73.63%
    arm-image: Speed:   49.05 MB/s
    arm-image: Progress: 86.86%
    arm-image: mapping output-arm-image/image
==> arm-image: kpartx -s -a -v output-arm-image/image
==> arm-image: partitions: [/dev/mapper/loop4p1 /dev/mapper/loop4p2]
    arm-image: Mounting: /dev/mapper/loop4p2
    arm-image: Mounting: /dev/mapper/loop4p1
==> arm-image: Mounting additional paths within the chroot...
    arm-image: Mounting: /proc
    arm-image: Mounting: /sys
    arm-image: Mounting: /dev
    arm-image: Mounting: /dev/pts
    arm-image: Mounting: /proc/sys/fs/binfmt_misc
==> arm-image: Installing qemu-user-static in the chroot
==> arm-image: write /proc/sys/fs/binfmt_misc/register: file exists
==> arm-image: fuser -k /tmp/455159232
Build 'arm-image' errored: step canceled or halted

[rossmckelvie]

kpartx intermittent error for reference, same solution of retrying to fix while on the docker for mac beta.

running /bin/packer
Warning: Warning when preparing build: "arm-image"

last_partition_extra_size is deprecated, use target_image_size to grow your
image

arm-image: output will be in this color.

==> arm-image: Retrieving Image
==> arm-image: Trying http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-05-28/2020-05-27-raspios-buster-lite-armhf.zip
==> arm-image: Trying http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-05-28/2020-05-27-raspios-buster-lite-armhf.zip?checksum=sha256%3Af5786604be4b41e292c5b3c711e2efa64b25a5b51869ea8313d58da0b46afc64
==> arm-image: http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-05-28/2020-05-27-raspios-buster-lite-armhf.zip?checksum=sha256%3Af5786604be4b41e292c5b3c711e2efa64b25a5b51869ea8313d58da0b46afc64 => /build/packer_cache/c193b0ba86d10e5ce653a34e195a72fd1e36fd79.iso
==> arm-image: Copying source image.
==> arm-image: Image is a zip file.
==> arm-image: Unzipping 2020-05-27-raspios-buster-lite-armhf.img
    arm-image: Speed:   66.20 MB/s
    arm-image: Progress: 17.86%
    arm-image: Speed:   47.33 MB/s
    arm-image: Progress: 30.62%
    arm-image: Speed:   47.53 MB/s
    arm-image: Progress: 43.44%
    arm-image: Speed:   44.75 MB/s
    arm-image: Progress: 55.51%
    arm-image: Speed:   41.95 MB/s
    arm-image: Progress: 66.82%
    arm-image: Speed:   50.00 MB/s
    arm-image: Progress: 80.31%
    arm-image: Speed:   62.19 MB/s
    arm-image: Progress: 97.08%
==> arm-image: Growing partition with 1024 M (1073741824 bytes)
    arm-image: mapping output-arm-image/image
==> arm-image: kpartx -s -a -v output-arm-image/image
==> arm-image: error kaprts -l exit status 1: device-mapper: reload ioctl on loop6p2  failed: Invalid argument
==> arm-image: create/reload failed on loop6p2
==> arm-image: add map loop6p1 (253:12): 0 524288 linear 7:6 8192
==> arm-image:
Build 'arm-image' errored: step canceled or halted

[yuval-k]

thanks you for the info! I don't have a mac, so unfortunately can't help with testing. will be happy to update the readme once it's released and confirmed to work!

[rossmckelvie]

@yuval-k I already have some of the docs updated locally, waiting for this to make it into docker for mac stable channel before I open a PR for your review. Hopefully they work out some of the kinks that cause my intermittent failures, but otherwise I am able to produce fully working images on my Mac again!

If anyone out there reading this has a Windows machine, I'm curious to know if any Docker for Windows users are having any success. When I updated the README a few months ago, I included Windows in the same bucket as Mac as not having device-mapper support, but I haven't been keeping up on updates.

[yuval-k]

not sure about windows (I'm running linux) but a different user reported success on a mac here https://github.com/solo-io/packer-builder-arm-image/issues/78