qemuimg2disk action

[nshalman]

Description

This creates a new action that uses qemu-img to stream any compatible image (particularly QCOW2) unmodified straight to disk.

Why is this needed

QCOW2 is a common image format across the internet. We should use it directly.

How Has This Been Tested?

make test-https will exercise the libcurl https support, but it hits a URL on the internet and takes a while to run so it should be run manually when making changes to this action.

The partprobe functionality was tested on bare metal. An example workflow template has been added in the comments of this PR.

How are existing users impacted? What migration steps/scripts do we need?

None. This is a new action.

Why static compilation? Isn't there a simpler way?

The qemu-img package from Alpine isn't linked against libcurl and so can't handle being passed a URL. This statically compiled binary, while more complicated to produce (e.g. sadly this build doesn't work against Alpine 3.14.2...) is guaranteed to always have the features we want.

Checklist:

I have:

• [ ] updated the documentation and/or roadmap (if required)
• [x] added unit or e2e tests
• [ ] provided instructions on how to upgrade

[gianarb]

@thebsdbox can you have a look? @nshalman do you mind to write a chapter about why this action is useful as part of the readme? maybe compared with the other images we have like archive2disk or image2disk? Thanks

[thebsdbox]

This is EXCELLENT!

[nshalman]

Well, that's a lot of feedback, thanks everyone! I intend to squash this all down and add a sign-off line after I'm done with all necessary cleanups.

[thebsdbox]

What is left for this to be added?

[nshalman]

What is left for this to be added?

A little bit of cleanup including naming and versioning along with figuring out how it should be pushed to Quay.

[rgl]

@nshalman, can this also stream from a compressed file (e.g. .qcow2.gz)?

[nshalman]

@nshalman, can this also stream from a compressed file (e.g. .qcow2.gz)?

I wouldn't expect it to, but qcow2 has a native compression format. It would be better to use that when preparing a qcow file for distribution.

E.g. (via https://serverfault.com/a/446044)

qemu-img convert -c -O qcow2 source.qcow2 shrunk.qcow2

[tstromberg]

@nshalman - would you like any help on how to get this pushed to quay?

[nshalman]

Example template used for testing in a bare metal sandbox. (The "nocloud" action is still under development. As of writing, the code can be found at https://github.com/nshalman/hub/tree/74091a78872b5d29c345d622a5a1831ff969008b/actions/nocloud/v1)

version: "0.1"
name: Focal_NoCloud-v2
global_timeout: 1800
tasks:
  - name: "os-installation"
    worker: "{{.device_1}}"
    volumes:
      - /dev:/dev
      - /dev/console:/dev/console
      - /lib/firmware:/lib/firmware:ro
    actions:
      - name: "stream-ubuntu-image"
        image: nshalman/qemuimg2disk:1.0
        timeout: 600
        environment:
          DEST_DISK: /dev/sda
          IMG_URL: "http://192.168.1.1:8080/focal-server-cloudimg-amd64.img"
      - name: "nocloud"
        image: nshalman/nocloud:1.0
        timeout: 30
        environment:
          DEST_DISK: /dev/sda
          METADATA_URL: http://192.168.1.1:50061
      - name: "fix-serial"
        image: quay.io/tinkerbell-actions/cexec:v1.0.0
        timeout: 90
        pid: host
        environment:
          BLOCK_DEVICE: /dev/sda1
          FS_TYPE: ext4
          CHROOT: y
          DEFAULT_INTERPRETER: "/bin/sh -c"
          CMD_LINE: "sed -e 's|ttyS0|ttyS1,115200|g' -i /etc/default/grub.d/50-cloudimg-settings.cfg ; update-grub"
      - name: "kexec-ubuntu"
        image: quay.io/tinkerbell-actions/kexec:v1.0.0
        timeout: 90
        pid: host
        environment:
          BLOCK_DEVICE: /dev/sda1
          FS_TYPE: ext4

[nshalman]

Description has been updated to clarify why we're building from scratch rather than using e.g. the Alpine package. A manual test that streams a QCOW2 image from an HTTPS url has been added to the Makefile for anyone who wants to make improvements to this action in the future.

I've learned that the CD process that builds and pushes actions reads the version information from the README.md file, so when making changes to an action in the future that are deserving of a version bump, the place to change it is in the README.md

My Makefile now reads those values from the README.md file rather than containing something else. The upper sections should be reusable by other actions in the future.

[tobert]

Approach looks ok to me. The main thing I'd like to see is a document added with basic instructions for how to update the build distro and resulting binary. Qemu does get CVEs every once in a while.

[nshalman]

Approach looks ok to me. The main thing I'd like to see is a document added with basic instructions for how to update the build distro and resulting binary. Qemu does get CVEs every once in a while.

@tobert Please clarify if you want those changes before this is shipped. We only ship qemu-img which is probably not the target of most of the QEMU CVEs.

I can certainly update the README.md with notes about bumping the versions in the Dockerfile, but as mentioned, I was unable to bump from the Alpine 3.13 series to 3.14 series as a build environment.

[tobert]

Approach looks ok to me. The main thing I'd like to see is a document added with basic instructions for how to update the build distro and resulting binary. Qemu does get CVEs every once in a while.

@tobert Please clarify if you want those changes before this is shipped. We only ship qemu-img which is probably not the target of most of the QEMU CVEs.

I can certainly update the README.md with notes about bumping the versions in the Dockerfile, but as mentioned, I was unable to bump from the Alpine 3.13 series to 3.14 series as a build environment.

I wouldn't hold up this PR. Just trying to help set us up for long-term maintenance :)

[nshalman]

If we don't care about image size, I did confirm that qemu-img shipped in Ubuntu Focal does have libcurl support.

FROM ubuntu:20.04 as builder
RUN apt-get -y update && apt-get -y upgrade
RUN apt-get -y install \
    qemu-utils parted
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT /entrypoint.sh

This creates a 205MB docker image and the entrypoint script would need light modifications.