search

warewulf / warewulf3

Warewulf is a scalable systems management suite originally developed to manage large high-performance Linux clusters.
107 stars 45 forks source link

ARM64 Boot #35

Closed ericvh closed 7 years ago

ericvh commented 7 years ago

So, as part of OpenHPC we've been trying to use warewulf to provision ARM64 servers which are UEFI based. Right now the work-around is to netboot grub and have a grub.cfg contain the provision and boot information for the nodes. The current solution is documented (somewhat) here: https://github.com/openhpc/ohpc/wiki/ARM-Tech-Preview

I was somewhat hoping changes to grub2 and the UEFI boot would magically fix my problems, but it doesn't seem there is a clear path to fixing the work around (ie. no ARM support on the horizon for SYSLINUX and grub2 syslinux_confligfile doesn't seem to "do the right thing").

The problems I am currently running into:

a) No syslinux equivilent for ARM or ARM64 b) Current ARM server systems need some extra bootloader support for UEFI/FDT embedding so you need to boot Image files versus raw kernel. c) grub appears necessary as a step before (b) to make sure FDT is loaded appropriately

I'm willing to accept that I'm doing something stupid and there is a solution to one or all of the above.

Here's the alternatives I'm currently considering:

a) Modify warewulf to provision ARM nodes purely through grub netboot and grub.cfg instead of using syslinux formats. This is the most straightforward IMHO, but may require a slightly different set of parameters for ARM versus x86 and would require embedded grub2 in a similar fashion to the current embedding of syslinux.

b) Modify grub2 to fix syslinux_configfile to work as expected - not sure how well this will go over in the grub2 folks. The other problem of needing to load Image versus "kernel" will remain, but maybe that's a more straightforward fix? In this mode, grub2 could be used as a "drop-in" for syslinux.

c) Further modification of grub2 to be able to more efficiently handle raw kernel files versus Image files.

Ideas? Happy to jump on slack/irc/whatever to discuss if this community uses that sort of thing.

bensallen commented 7 years ago

Hi Eric,

This is a topic I'm also actively looking into. Grub2 does appear to be the only UEFI aarch64 network capable boot loader available today.

One other possible item is using an UEFI implementation that has a ramdisk and TFTP or HTTP clients. Copying the kernel and initrd into a ramdisk, and booting the it via EFI stub. This approach provides a number of challenges. Relying on the vendor to implement said functionality in their UEFI. These functions don't appear to be standard in EDKII. We would also need to generate a EFI shell script to be the pxelinux.cfg equivalent. HPE is the only vendor that I know of that can pull this off today, and only with x86_64 hardware [1].

a) I think this option is the most attainable, but the most work in Warewulf. I don't see any significant blockers for this approach however.

b) Can you point me to any docs on this existing functionality? This would be the nicest option for Warewulf sans syslinux being ported to aarch64. The only work needed would be to create the boot image, which means we'll still need a flag to call out a node as aarch64.

[1] http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=c04565930

ericvh commented 7 years ago

(b) documentation is scant (I think the feature is relatively new - I'm operating off Beta builds) - relevant source is: http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/lib/syslinux_parse.c?h=next and http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/commands/syslinuxcfg.c?h=next

It trips over something in the configs though and says you need to load kernel first. I haven't tried fault isolating yet, probably an ordering issue. Also, its mechanisms seem to be convert syslinux to grub.cfg and then parse grub.cfg which could be part of the problem.

bensallen commented 7 years ago

@ericvh Have you tested iPXE on aarch64?

ericvh commented 7 years ago

AFAIK there isn't iPXE support for any of the server class hardware -- looking at the source there is arm64 -- so I can try this path.

bensallen commented 7 years ago

@ericvh Yea I don't expect explicit network driver support to be included for most platforms outside of SNP. The main reference I'm going on: http://forum.ipxe.org/showthread.php?tid=8526. Hoping to get our Mustang boards firmware updated so I can test this as well.

bensallen commented 7 years ago

Compiling ipxe snp target make ARCH=arm64 bin-arm64-efi/snp.efi using GCC 5.4.0, and booting the resulting snp.efi on a APM Mustang board appears to work. iPXE finds all the network interfaces including the ConnectX-3 cards we have installed. Next step is to actually have iPXE launch a kernel and initrd.

Testing with the latest TianoCore and SlimPro (3.06.25) firmware installed.

Note, CentOS 7.3 AltArch ships with GCC 4.8, which doesn't appear to compile arm64 iPXE. This will make our build process of warewulf-provision-server on aarch64 quite bit more complicated.

bensallen commented 7 years ago

iPXE boots the CentOS 7.3 AltArch 4.5.0 kernel and initramfs fine. The kernel needs to be uncompressed for iPXE to load it. There wasn't anything special I had todo with the DTB and kernel.

Thinking now of the various tasks within Warewulf that are needed. I don't think a cross-compile toolchain will be needed for the first pass, but eventually it would be very nice to have. For the first pass I would expect to have an aarch64 Warewulf server. It should be able to share the same database with an x86_64 server since both are little endian.

Tasks

Bootstrap

iPXE

bensallen commented 7 years ago

Started a project https://github.com/warewulf/warewulf3/projects/1 to track tasks.

ericvh commented 7 years ago

Can you share your general config for the CentOS test. I've finally gotten around to build iPXE myself and tried slotting it in to replace grub2 as the syslinux equivelent on a cavium (with USB attached network). It does seem to startup but then it proceeds to try and load lpxelinux.0.

I'll admit I just did the naive thing when configuring so its possible I just need to RTFM, but want to make sure I have a similar setup w.r.t. how warewulf will eventually incorporate it.

bensallen commented 7 years ago

Update /etc/warewulf/dhcpd-template.conf and add/update:

...
if substring (option vendor-class-identifier, 15, 5) = "00011" {
    if exists user-class and option user-class = "iPXE" {
        filename "/warewulf/aarch64/test.cfg";
    } else {
    filename "/warewulf/aarch64/snp.efi";
    }
}
# IB Clients
elsif substring(hardware,0,1) = 20 {
...

Run wwsh dhcp update.

The final form of dhcpd-template.conf will be cleaned up a bit from this, but this only uses iPXE for aarch64 clients.

Create /var/lib/tftpboot/warewulf/aarch64/. Install snp.efi there, and create a static test.cfg for now. Testing with this:

#!ipxe

set base tftp://${next-server}/warewulf
kernel ${base}/aarch64/bootstrap/0/kernel ro initrd=initfs.gz wwhostname=mustang00 net.ifnames=0 biosdevname=0 nosplash wwmaster=10.17.6.9 wwnetmask=255.255.0.0 wwnetdev=eth0 wwhwaddr=00:01:73:02:12:e8

initrd ${base}/aarch64/bootstrap/0/initfs.gz
boot

Place your kernel (gunzip'ed) and initfs.gz in /var/lib/tftpboot/warewulf/aarch64/bootstrap/0. Double check permissions (and Selinux labeling if you have it enabled).

Note, USB attached networking almost certainly will require you specify additional kargs for the host to force the correct kernel modules to be loaded, i.e. wwsh provision set --kargs="nosplash wwkmods=virtio_net,virtio_pci" n01. Likely you need to update /etc/warewulf/bootstrap.conf to include said kernel modules as well.

Lastly, Warewulf currently builds bootstraps based on the current system's files. Even you use wwbootstrap to create a bootstrap file on a aarch64 system, when you go import it on a x86_64 server it'll use the initfs system files locally on the system. The only way around this currently would be to run a Warewulf server on an aarch64 system.

PS. I haven't actually booted the Warewulf's initfs.gz on aarch64 yet, only the initramfs.gz from CentOS 7.3's rootfs tar.gz. Warewulf's initfs.gz provisioning environment does at least compile cleanly on aarch64.

ericvh commented 7 years ago

I've worked through the kernel side of things with my existing grub2 based workaround, I can document the specifics for USB and other platforms -- we'll likely tie this into the generic OHPC ARM variant of the build instructions for completeness. Right now we are working with aarch64 warewulf servers, but for heterogenous environments we might be able to use a cross-chroot or cross-docker env to build the bootstrap.

It's probably worth tracking on another issue, but has there been any thought about Docker deployment using warewulf (w/swarm or kubernetes, or whatever)?

    -eric

On Thu, Jul 6, 2017 at 12:16 PM Ben Allen notifications@github.com wrote:

Update /etc/warewulf/dhcpd-template.conf and add/update:

... if substring (option vendor-class-identifier, 15, 5) = "00011" { if exists user-class and option user-class = "iPXE" { filename "/warewulf/aarch64/test.cfg"; } else { filename "/warewulf/aarch64/snp.efi"; } }

IB Clients

elsif substring(hardware,0,1) = 20 { ...

The final form of dhcpd-template.conf will be cleaned up a bit from this, but this only uses iPXE for aarch64 clients.

Create /var/lib/tftpboot/warewulf/aarch64/. Install snp.efi there, and create a static test.cfg for now. Testing with this:

!ipxe

set base tftp://${next-server}/warewulf kernel ${base}/aarch64/bootstrap/0/kernel ro initrd=initfs.gz wwhostname=mustang00 net.ifnames=0 biosdevname=0 nosplash wwmaster=10.17.6.9 wwnetmask=255.255.0.0 wwnetdev=eth0 wwhwaddr=00:01:73:02:12:e8

initrd ${base}/aarch64/bootstrap/0/initfs.gz boot

Place your kernel (gunzip'ed) and initfs.gz in /var/lib/tftpboot/warewulf/aarch64/bootstrap/0. Double check permissions (and Selinux labeling if you have it enabled).

Note, USB attached networking almost certainly will require you specify additional kargs for the host to force the correct kernel modules to be loaded, i.e. wwsh provision set --kargs="nosplash wwkmods=virtio_net,virtio_pci" n01. Likely you need to update /etc/warewulf/bootstrap.conf to include said kernel modules as well.

Lastly, Warewulf currently builds bootstraps based on the current system's files. Even you use wwbootstrap to create a bootstrap file on a aarch64 system, when you go import it on a x86_64 server it'll use the initfs system files locally on the system. The only way around this currently would be to run a Warewulf server on an aarch64 system.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/warewulf/warewulf3/issues/35#issuecomment-313460753, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEP2KSP2P9txvVs_WPRTqAXTHPZS76lks5sLRYegaJpZM4N0Rvb .

bensallen commented 7 years ago

Running into a bit of a conundrum with integrating the aarch64 build of iPXE. Looks like iPXE for arm64 uses the gcc option -mabi=lp64, which was added in GCC 4.9. RHEL/CentOS 7 ships with GCC 4.8.

Ubuntu 14.04, SLES 12 should be fine as they have later GCC versions available.

Anyone know of a clever way of dealing with this on CentOS/RHEL 7 short of checking the compiled iPXE binaries into the repo, or asking users to roll their own GCC install...?

bensallen commented 7 years ago

Replying to myself, the iPXE team fixed the 4.8.5 GCC build issues.

rengolin commented 7 years ago

Hi, has the merging of pull request #46 fixed this issue?

I'd like to use this with OpenHPC hopefully before release 1.4.

cheers, --renato

bensallen commented 7 years ago

@rengolin

Almost, we still need to add:

After thats done there will still be some heavy caveats/limitations :

  1. There's no cross-compile support for the Initramfs, so as of today there still needs be an aarch64 master. Although one could manually copy /var/warewulf/initramfs/aarch64 to a x86_64 master and it should work.
  2. There's been very little testing of the development branch, and there's been a lot of changes since 3.7 (master). I would like to see more testing on x86_64 and aarch64 before we start telling people that it is stable. Also there's been zero testing on SLES/OpenSUSE.
  3. The upgrade procedure hasn't been well tested from 3.7 to the development branch's 3.8.
  4. The Debian/Ubuntu package configs haven't been updated to match current RPM specs.

Known to break on upgrade:

Should migrate:

rengolin commented 7 years ago

Right. Some comments...

  1. We're only testing aarch64 master, so that shouldn't be a problem for now. Copying kernel+initramfs is also not a big issue, if necessary. We use modern kernels with CentOS 7 anyway, so we'd have to replace them manually if we want to experiment.
  2. Didn't mean to push, just wanted to understand the time frames. OpenHPC 1.4 will come later this year, so if the validation of warewulf 3.8 would start soon and take weeks to a month or two, we could maybe think it would be ready. If not, I'll have to think about plan B (which I still have no idea what it will be).
  3. I'm (personally) not using Debian/Ubuntu, nor I'm planning migrations. My concern right now is how to setup an OpenHPC [warewulf+slurm] cluster on AArch64 with CentOS from scratch.

I appreciate there are many other sides to this issue, just wanted to clarify my own.

I think best thing now for me to do is to stop the OpenHPC installation by the time I'd install warewulf, install it from master, then continue from there and see what breaks (in addition to what you already said).

bensallen commented 7 years ago

  1. With an aarch64 master it should work without having to copy anything manually.

  2. I don't have much else planned for 3.8 other than what is open right now. The more testing the better. I also have an aarch64 cluster on the way, so we should get a decent amount of in-house testing in the next few months. I'm expecting to run with a x86_64 master however.

A side note, the limitation on cross-compile of the initramfs is EPEL and the distros don't have a cross-compile GLIBC packaged, and we don't want to carry GLIBC in the project. There was some thought if we should goto MUSL libc, but I think we're going to hold off on that for 3.9, or even to a rewrite for 4.0.

  1. Fair, this is more of a concern for our users with existing 3.7 or older installs.

Note, you should install from the development branch not master. Master is our stable branch at 3.7 currently. Development is master + 1. 3.8 at the moment.

rengolin commented 7 years ago

Ok, I'll pull from development. Thanks!

bensallen commented 7 years ago

Closing.

Outstanding but uncertain if/when we'll be able to get to them:

We would need to cross-compile glibc, and the various dependancies of programs in the initrd.

I don't think this will actually be needed in practice, as the package managers make it difficult to build chroot's across architectures. As such the workflow will be to use wwbootstrap and wwvnfs on a native host, either saving the cpio output and importing on another host (eg. wwsh bootstrap import --arch=aarch64, wwsh vnfs import --arch=aarch64 or importing to the database directly.