Investigate direct serving/booting of ISO images

[danderson]

It would be nice to be able to magically boot directly from ISOs. This is very hard/impossible to achieve in the general case, but it might be feasible to make it work for certain specific cases.

The closest we can easily get is with MEMDISK, which adds a chainloading step that works with linux kernels/distros that are built with MEMDISK support. I have no experience with running such a setup, but the first step would be to create a Booter that constructs memdisk-based Specs, and start experimenting with that.

Alternatively, the "quick" booting templates might just replace this, in terms of ease of use.

[masaeedu]

ipxe already has good support for booting ISO images, but the ability to just directly set an ipxe script isn't exposed via pixiecore. Right now I compile another undionly.kpxe on the fly whenever my api is hit by pixiecore, embed an ipxe script which ultimately boots the ISO I intended to boot, and hand that to pixiecore to be chainloaded by the first ipxe instance.

Works okay, it's just kind of awkward.

[danderson]

Huh, TIL the sanboot command can boot straight to ISOs, apparently. I have many questions.

The main one is how portable this is going to be. What ISO are you booting? I'm assuming the booting kernel needs some kind of SAN support to successfully continue after ipxe hands off. If it requires a lot of fiddling and weird ISOs to work correctly, this may be a case where I say that it's just outside the scope of what Pixiecore does. I think it's okay to say that if you want really esoteric booting, you need to fall back to a homegrown solution.

But, if it works reasonably universally across most BIOS/UEFI... Then we're in business.

Implementing support is reasonably simple, we just need to tweak the ipxe script we send during the boot sequence. The catch is that this will probably bubble into changes in the various internal and external APIs, so that servers can select either a kernel+initrd or an ISO boot. We could get hacky compat by saying "if you provide a kernel that ends in .iso we'll assume stuff"... But I think I'd rather just update the API.

[masaeedu]

I'm booting all kinds of wacky stuff, ESXi, systemrestore, probably going to try a library of things in the future. It Just Works TM, without any fiddling. The stuff I'm working with is all HP iLO, but I'm pretty sure if ipxe works as advertised it's going to work across different server types.

Anyway, as a more general request it'd be nice if the cmdline and kernel bits in the API could be swapped/supplemented for ipxeScript, whether or not my ipxe script works across all the situations I need to work in is then my problem.

[masaeedu]

Here's my ipxe script FWIW:

  const pxeScript = `#!ipxe
dhcp && sanboot http://${selfIP}:${port}/images/${isoName}
`;

There's some js bits getting interpolated there, but that's it. Just dhcp, then sanboot.

[danderson]

Yeah, not supporting ipxe scripts was a deliberate design decision. In fact, pixiecore started out using syslinux as the intermediate bootloader, and was later able to switch to ipxe transparently, because the details of the boot process were not exposed to the higher level systems.

It also leaves the door open for supporting non-PXE boot methods in future, because all the higher-level system specifies is what it wants the machine to be running at the end of the boot sequence.

With that said... I've mellowed over the years, and as long as you'd be okay with a massive "This might break at any point in the future, and also not work correctly in all cases, and it's your problem to make it work correctly" disclaimer, I think I'd be okay with just adding an option to just pass a custom ipxe script.

But, I'd also like to add native ISO booting using the "high level" API if I can, and you've given me a potential solution to do that, so thank you :)

If you're interested in a raw ipxe script option, I'll file a separate bug for that and start hacking.

[masaeedu]

If it makes it "cleaner", I'd be fine with any approach that lets me inject my ipxe script into a second chainloaded instance of ipxe. I just couldn't find any way to do it reliably that didn't involve compiling the script into the kernel image when my API is called, which is a really annoying hack, and involves awkward lockfile and timeout juggling.

[danderson]

Do you want a single ipxe script for all machines you're booting? i.e. if sudo pixiecore --ipxe-script=my/ipxe.script works, is that enough for your needs?

[masaeedu]

No, the script I want to load is dynamic. If you go with the static approach, I'd want a mac filter option as well, so I can start one pixiecore instance per target server with an especially prepared script.

[danderson]

Gotcha, we'll do it via API then.

[masaeedu]

@danderson As a slightly off topic question, did you recently rebase some code around? I'm trying to get some code that used to work properly with undionly.kpxe and now it's broken. Having a hard time understanding the history.

[danderson]

Yeah, I screwed up a push at one point and mangled the history. The branches are now protected so I don't make that mistake again. Sorry about that.

The issues you're having probably come from the changes from #51 or #52. Specifically, pixiecore unconditionally chainloads to an embedded build of ipxe, to guarantee a known featureset for the rest of the boot. This chainload unloads the UNDI stack, because ipxe has documented bugginess if you try chainload from ipxe into undionly.kpxe (some firmwares bork the network stack, so ipxe can never get back online). So, when you chainload to undionly.kpxe... There's no UNDI stack left, and the boot fails.

(See why I'm worried about exposing the boot pipeline externally? The more buggy firmwares I discover, the more terrible workarounds I have to add - but they're all hidden inside the machinery of pixiecore, nobody has to know...)

I believe you can fix this by chainloading to ipxe.pxe instead (make bin/ipxe.pxe in the ipxe source tree). In my experiments, I was able to chainload ipxe.pxe -> ipxe.pxe indefinitely without problems.

[AndreyNikiforov]

bumping the thread. Any chance to get iso booting capability in the pixiecore (including docker image), @danderson ?

[danderson]

From the readme:

This project is no longer actively developed. I'm glad if you find it useful, but don't expect any significant changes.

I may come back to it at some point, but if you have a burning need to boot ISOs, you may need to look elsewhere.

[AndreyNikiforov]

From the readme:

This project is no longer actively developed. I'm glad if you find it useful, but don't expect any significant changes.

I may come back to it at some point, but if you have a burning need to boot ISOs, you may need to look elsewhere.

I understand, no problem. IIUC I can replace chain with sanboot in boot.ipxe file and then supply iso instead of kernel image to the cli -- hacky way to make it work for me. Figuring out how to compile project is taking longer than I expected though...