Bare Metal Linux ISO (not PXE) Redux

[timblaktu]

Like many others before me, I'd like to use Packer to create Linux .iso images enabling the installation of my Packer-built Linux virtual environments on bare metal machines. As the three past requests about creating images for bare metal machines were closed without resolution, I'd like to re-open a feature request wherein we can:

1. Summarize and hopefully complete the solution fragments that have been presented in past issues
2. Discuss how and whether it makes sense to:
   a. integrate into an official Packer extension (builder, post-processor, provisioner)
   b. leave as an informal solution for users to implement in their Packer templates
   c. document these instructions in the official Packer documentation

Use Case Discussion/Limits

There are still many use cases for bare metal Linux installation, and here I am specifically targeting the use case of a (technically-proficient) development team that uses a Packer-built Linux virtual environment to develop a product with a bare metal deployment requirement, i.e. that the product be deployed as an .iso that can be installed, unattended, by (technically-non-proficient) customers onto arbitrary physical amd64 machines. Many past discussions focused on PXE-boot solutions, and although most of the underlying technology and mechanics are applicable here, please note that the PXE-boot use case is different enough to derail this issue like the rest, so please help me keep the discussion limited to creating installable iso media with Packer. Also, note that I'm specifically interested in Linux targets here; I know Packer supports building more OSes than Linux ones, but I don't want this discussion to be derailed by trying to find the perfect and general solution.

My use case has been building primarily Debian-based systems, and we're currently building virtualbox VM and docker containers with the same (mostly ansible) provisioning. Now add to this use case the need to create an iso that can be installed on bare metal that will be identical to the corresponding virtualbox VM.

Proposed Solution

For our needs, we believe "the Squashfs approach," originally hinted by @vtolstov, will work best as we're building primarily Debian machines. The Debian Live Project started including the Calamares Installer, which is a distro-independent linux installer infrastructure that enables the inclusion of an arbitrary squashed filesystem (squashfs) into the iso media, to boot into this squashfs "live linux" in non-persistent RAM, and notably, to conditionally copy the same squashfs into a device on the target system. All of this can be greatly customized, and debconf can be preseeded during installation just like a normal d-i install.

The Debian-Live Documentation describes the operation of "a special udeb called live-installer" in which "Installation will proceed in an identical fashion to the "normal" installation described above, but at the actual package installation stage, instead of using debootstrap to fetch and install packages, the live filesystem image is copied to the target."

From the above I hope it's obvious that this would provide an ideal platform for creating a Debian-based distributable appliance as a product which is identical to virtual images previously created by Packer, but as a slight aside, it would also seem to me that this approach could install any Linux.

It would work like this:

1. A Packer VM builder (in our case, virtualbox-iso) creates a normal virtual machine per the packer.json and our various provisioners (in our case, mostly ansible)
2. NEW provisioner steps are appended to the end, which will:
   • shell provisioner runs a script that:
     • install the squashfs tools
     • creates a squashfs file representing the machine's entire root partition, and store this in a known location in the VM
     • copies the kernel from the /boot to same location in VM
     • copies the initrd from the /boot partition to same location in VM
   • file provisioners then download the kernel, initrd, and root squashfs to the Packer host where they can be available for the subsequent post-processor step
3. A NEW final Shell post-processor step will then run the live-build commands required to build the iso, per the debian live project documentation. The output from this step will also be placed in the builds/ dir where it is automatically archived by the jenkins job.

To further illustrate, 2. above is implemented pretty much exactly as what @vtolstov originally proposed:

    {
      "type": "shell",
      "execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
      "script": "scripts/squash-file-system.sh",
      "only": ["virtualbox-iso"]
    },
    {
      "type": "file",
      "direction": "download",
      "sources": ["/squashfs/vmlinuz"],
      "destination": "builds/squashfs/debian.vmlinuz",
      "only": ["virtualbox-iso"]
    },
    {
      "type": "file",
      "direction": "download",
      "sources": ["/squashfs/initrd.img"],
      "destination": "builds/squashfs/debian.initrd",
      "only": ["virtualbox-iso"]
    },
    {
      "type": "file",
      "direction": "download",
      "sources": ["/squashfs/filesystem.squashfs"],
      "destination": "builds/squashfs/root.squashfs",
      "only": ["virtualbox-iso"]
    }

where scripts/squash-file-system.sh is:

#!/bin/sh -ex

apt-get -y install squashfs-tools

mkdir -p /mnt/squashfs /squashfs
mount -o bind / /mnt/squashfs

mksquashfs /mnt/squashfs /squashfs/filesystem.squashfs -comp gzip -no-exports -xattrs -noappend -no-recovery -e /mnt/squashfs/squashfs/filesystem.squashfs
find /boot -name 'vmlinuz-*' -type f -exec cp {} /squashfs/vmlinuz \;
find /boot -name 'init*' -type f -exec cp {} /squashfs/initrd.img \;

We can soon post what we're doing for step 3, which is the Debian live-build step, but for now this is all we got.

Is anyone else using the new Debian Live (or another Calamares Installer) project as a Packer post-processor step to make isos?

Do you think this approach could / should be adopted as an official "Linux ISO" Packer post-processor?

Is there some way for the post-processor to ensure provisioning step 2 above is done, so that its inputs (root squashfs, kernel, and initrd) are made available for the post-processor? Or is the only way to create a new specialized builder that just does it all (builds and provisions the VM, generates squashfs, kernel, and initrd, AND builds the installer iso.)?

In this approach, the final live-build post-processor step which builds the iso, can't happen until after the VM is built. So we are trading time (it's not very time efficient this way) for versatility (can use this approach for ANY Packer machine that supports squashfs, regardless of what provisioners were used).

An alternative approach, which I was considering before discovering Debian's Live Installer/Calamares, would enable the bare metal iso to be built concurrently with the virtual machines. This uses the standard debian preseeded installer approach, modifying the initrd of an existing iso, embedding your preseed file, and the provisioning scripts referenced by the preseed file, inside it. In this approach, you just use the same provisioning scripts for the iso build process that you use for the packer template building the VM. But for this there's much more orchestration and management of scripts and processes than the former approach. So we choose to do the former for now.

Sorry for the dissertation, but it's not a simple topic, and I wanted to take care to thoroughly capture all the aspects of this specific problem. Thanks for your time!

[MarkusTeufelberger]

Personally I suspect that an even easier(?) approach would be the "boot + dd + sha3sum + reboot approach". Build a VM image that should end up on your bare metal host, add this large image + a small Linux distro with good hardware support to an ISO file. Add a script at boot time that just copies the VM image 1:1 to a pre-specified physical disk, verifies that everything was written correctly and reboots after ejecting the ISO again into the newly provisioned host. No need for OS specific installers, Linux would just be used to provide the ability to copy and verify files. The Linux distro and script used for installation would be mostly static, the only difference between ISO files would be the VM image added to it.

Maybe I'm overlooking something, but VMs also have bootloaders installed after all, their disk images are also partitioned etc. - why not just dump them out to a physical disk and boot into them? I'm sure there's some massaging to be done to make the OS not freak out as hardware IDs change etc., but that hardly seems to be taken care of by copying vmlinuz files around either.

[SwampDragons]

I don't think the core maintainers are going to have a chance to implement something like this for a long time, but I'd love to see a community builder get created. We're in the process of trying to make integrating third party builders much more seamless, and allowing third party builders to be documented on our site, so if someone in the community wants to create a bare metal builder I'd be thrilled to see it and would provide guidance for getting it integrated nicely with the packer core.

[SwampDragons]

Here are some good getting started points for a community member who would like to create a builder:

https://www.packer.io/docs/plugins/creation https://www.packer.io/docs/plugins/creation/custom-builders

[mikeschinkel]

As a Go developer I would be interested in building this if there were someone with the Linux expertise to fill in the knowledge gaps I have?

[jacobsalmela]

As a Go developer I would be interested in building this if there were someone with the Linux expertise to fill in the knowledge gaps I have?

I can fill in those gaps (as well as writing go).

[mikeschinkel]

@jacobsalmela — That is very cool to hear. How would you like to collaborate on this? Want to email me?
mike -at- newclarity -dot- net.

[dgengtek]

I had the same issue when I found the suggestion in #955

live-boot and the squashfs is only required if you want to run the image live either over pxe or by setting the required kernel parameters in your bootloader

Your image only needs to be in the raw image format and have a bootloader and then write the raw image to the bare-metal disk if you want to "install images to bare-metal"

Here are examples on how I build debian live-boot images with packer and qemu for my use case

https://github.com/dgengtek/packer-templates/blob/1.0/terminal/main.json https://github.com/dgengtek/packer-templates/blob/1.0/scripts/debian/build_live_boot.sh

[zero-pytagoras]

any update on this ? I would love to have this possibility as well.

[CollinLeishman]

Hey @mikeschinkel, It seemed that @jacobsalmela was willing and able. May we know if that's still the case?

[jacobsalmela]

We connected offline, but I haven't done anything with it yet. I still plan to, but work and life are coming up first.

[mikeschinkel]

@CollinLeishman — I have moved on to other projects, but if @jacobsalmela can help I will try to work on it. @dgengtek comment is helpful, but not sufficient for me to be able to figure it out.

@jacobsalmela let me know if and when you want to tackle this.

[CollinLeishman]

Thanks for the update guys. Best wishes.

[xeor]

Another bump.. How is it going? :)

[mikeschinkel]

I would still love to work on it, but I do need to Linux help to get there...

[SuperPauly]

Any updates?

[jacobsalmela]

I still think it would be fun, but it is not on my radar right now

[cilki]

This isn't a solution since it's not finished (yet), but I started a new project called goldboot dedicated to building and deploying bare-metal machine images similar to Packer. Only works with UEFI systems, though.