Support for secure boot

[pbmsys]

Hi.

Is it possible to enable secure boot support for cobbler. Many Linux distros is supporting this feature and would be nice to provision systems directly with secure boot.

/Peter

[philseeley]

NOTE: I’ve only tested this with RHEL7/8 and CentOS7/8.

I've managed to get Secure Boot working with an interim solution with release28.

The complication of secure boot is that all stages of the boot loading need to be distro specific. So the DHCP configuration needs to have a system specific “filename” defined. As I need the flexibility to set this value at each level of the distro/profile/system hierarchy, I can only use the ksmeta facility for this. I’ve submitted a simple pull request #2180 to make ksmeta values available in the DHCP templating.

The ksmeta value for “filename” needs to be set to the first stage loader for the OS. For 64bit RHEL based distros this is the shimx64.efi. This then loads the grubx64.efi which finally loads the host(MAC) specific config file. Because shimx64.efi is hardcoded to fetch grubx64.efi I can’t rename them, so I have to put them in a distro specific sub-directory. Also grubx64.efi will fetch the config files from the same sub-directory.

The steps to get this working are:

1. Apply the patch from PR #2180.
2. Extract the shimx64.efi and grubx64.efi from the distro’s “shim-x64” and “grub2-efi-x64” RPMs.
3. Place these two files in a distro specific sub-directory of /var/lib/cobbler/loaders/, e.g. /var/lib/cobbler/loaders/rhel8-x86_64/shimx64.efi.
4. Set the filename ksmeta at an appropriate level, e.g. “filename=grub/rhel8-x86_64/shimx64.efi”
5. In the dhcp.template add a check for the “filename” in the mgmt_parameters:

   <snip>
   #for dhcp_tag in $dhcp_tags.keys():
   group {
       #for mac in $dhcp_tags[$dhcp_tag].keys():
           #set iface = $dhcp_tags[$dhcp_tag][$mac]
   host $iface.name {
       hardware ethernet $mac;
   <snip>
       #if $iface.mgmt_parameters.get('filename')
       filename "$iface.mgmt_parameters.get('filename')";
       #end if
   }
       #end for
   }
   #end for

6. Update the grubsystem.template to also check for filename. This is required as the file format is different:

   
   default=0
   timeout=0

if $mgmt_parameters.get('filename')

menuentry '$system_name' { linuxefi $kernel_path $kernel_options initrdefi $initrd_path }

else

title $system_name root (nd) kernel $kernel_path $kernel_options initrd $initrd_path

end if

6. Add a post sync trigger /var/lib/cobbler/triggers/sync/post/secure-boot. This copies in the distro specific boot loaders and links in the config files with the name required by grubx64.efi:

!/bin/bash

for loader in /var/lib/cobbler/loaders/* do if [ -d "${loader}" ] then cp -r ${loader} /var/lib/tftpboot/grub/

(
  cd /var/lib/tftpboot/grub/$(basename ${loader})

  for p in ../01-*
  do
    f=$(basename $p)
    f=${f,,}
    ln -sf $p grub.cfg-$f
  done
)

fi done

7. Do a cobbler sync.

[SchoolGuy]

@watologo1 Could you check at work how much of this applies to *SUSE distros (especially SLES). This would be a cool feature if we get this to work!

@philseeley Thanks for the contribution! I will merge your PR anyway because this does not hurt in my eyes. If you have the time I would really much appreciate if you would document this in a blog-style post on cobbler.github.io!

[watologo1]

filename=grub/rhel8-x86_64/shimx64.efi

Either you add a grub.cfg into grub/rhel8-x86_64 which looks like:

rw: work-around for namespace collision on 'shim.efi'

regexp --set origin "(.*)/[^/]+" $prefix echo "resetting prefix from '${prefix}' to '${origin}'" sleep 1 set prefix="${origin}"

we might want to experiment with 'normal' instead of 'configfile ...'!

or, maybe even better, leave 'prefix' alone, and use '${origin}'

(instead of '$prefix') in all those other config-files!?

export origin configfile "${origin}/grub.cfg"

This "should" load the general grub/grub.cfg main config file again by re-setting $prefix from: grub/rhel8-x86_64 to grub

But even nicer would be to put all possible shim loaders into: grub/binaries/rhel8-x86_64/shimx64.efi grub/binaries/sle12-sp1-x86_64/shim.efi ... and link: grub/cobbler_shim.efi -> grub/binaries/sle12-sp1-x86_64/shim.efi

I wonder why this is necessary at all, in the end you only want to have one shim x86 efi binary, right? And this should be placed:

/grub/shim_whatever.efi to get the correct prefix and the general grub.cfg loaded. Overriding filename in dhcp generation is a nice thing to have. I wait for comments first. I hope that makes sense...

[watologo1]

Another comment about using bash/shell trigger after/post sync: This seem to be nice idea to check for boot loaders to add... This is very distro specific and differing this in python code always was error prone and maintenance intensive. I could imaging there could be buildgrub${distro}.sh specific triggers differed at rpm build or make time and copied there. The if "rhel" or "suse" or "debian" code in quite some regions is really ugly.

Thanks for pointing to the post sync triggers. This should make things easier.. What do you think about the distro differing idea there?

[watologo1]

You also may want to have a look at: scripts/mkgrub.sh The idea is to get updated grub code in. Shim is pre-built and different. But the locations are always distro specific (if not yet in /var/lib/cobbler/loaders/* and they may not even be there, but directly taken from the updated/installed rpm files?)

[philseeley]

I wonder why this is necessary at all, in the end you only want to have one shim x86 efi binary, right? And this should be placed:

/grub/shim_whatever.efi to get the correct prefix and the general grub.cfg loaded. Overriding filename in dhcp generation is a nice thing to have. I wait for comments first. I hope that makes sense...

Hi @watologo1, if you're suggesting that only one shim would be required, we tried this... Unfortunately each shim only trusts the grub loader from the same distro, i.e. the shim from RHEL wont load the grub loader from CentOS. So this is why we had to separate them out into distro specific directories.

[philseeley]

@philseeley Thanks for the contribution! I will merge your PR anyway because this does not hurt in my eyes. If you have the time I would really much appreciate if you would document this in a blog-style post on cobbler.github.io!

@SchoolGuy Will do.

[philseeley]

I also have a change for the master branch that add in "mgmt_parameters" for the DHCP templating. This should allow the "autoinstall_meta" to be used in the same way as "ksmeta". I'll submit a pull request for this as I think it's probably worth having this change anyway, but we might want to have a more "officially supported" way for release30. The complication would be trying to cater for all the distro specific differences.

[SchoolGuy]

Okay let's leave this open until we have more than a hack for this feature request! But great to see progress!

[watologo1]

... Unfortunately each shim only trusts the grub loader from the same distro... But you would still only install one shim and the corresponding grub efi executable on the cobbler server to install all distros, right?

@SchoolGuy : Please do not merge this yet. This breaks things heavily for the future and makes it rather complicated if done wrong.

The linuxefi vs linux differing is already there. And much more. Therefore it is important that also shim booted grub makes use of the general /grub/grub.cfg config file.

If you really have to go down into a separate subdirectory with a shim and grub efi boot loader pair, we should try to re-set prefix variable and source /grub/grub.cfg config file asap.

Can you try: Re-setting grub prefix variable via a grub.cfg stub loading grub.cfg one level above in directory hierarchy.

I also tried to override filename with the mgmt approach. Can you please give and example how to pass --autoinstall-meta to get a new filename written into dhcp.conf.

See latest modifications right below where filename is pre-set for different architectures if it's not yet there: if distro is not None and not interface.get("filename"): if distro.arch == "ppc" or distro.arch == "ppc64": interface["filename"] = yaboot ....

In the future I'd like to have both: filename and next-server be overridable per system. Straight forward could be to simply add this to the system as e.g. interface_filename= interface_next-server= variables and by that override any previously set defaults. But the param count for system is rather high already. A general interface param covering all interface properties like: interface_class="mac=ff:ff:ff:ff:ff:ff ipv6-address=xxx netmask=xxx ..." as an alternative to add/modify network interfaces would be nice..., but more intrusive.

[philseeley]

But you would still only install one shim and the corresponding grub efi executable on the cobbler server to install all distros, right?

I think to get this to work, you would need to build a grub.efi which has certs for all possible distros. You would then either need to get this or a shim signed by the Microsoft CA.

This probably isn't practical (or even possible) for us, hence the need to use the shims from each vendor.

See latest modifications right below where filename is pre-set for different architectures if it's not yet there: if distro is not None and not interface.get("filename"): if distro.arch == "ppc" or distro.arch == "ppc64": interface["filename"] = yaboot

As this is only architecture specific, we would also need to select a different shim/filename per vendor.

In the future I'd like to have both: filename and next-server be overridable per system.

I'd agree this would probably be the most flexible. The complexity will always be making things work out-of-the-box for all the different distributions.

I initially thought we could add something to the distro_signatures.json and have some distro specific processing to extract the required shim/grub.efi so we could automatically set the filename. But we would also need the ability to override the filename at the profile and system levels because we would want to have one distro support UEFI and legacy BIOS installs, e.g. in our environment physical servers can do secure boot, but all our VMs can't, so we have rhel7-x86_64 and rhel7-x86_64-efi profiles with different filenames set in their ksmeta, but both based on the same rhel7-x86_64 distro.

[watologo1]

but all our VMs can't JFI: Latest OVMF kvm extensions can do secure boot. But you can still boot via shim, even you do not support secure boot. It's only the other way around which does not work. In fact this is what we or propably all distros do. You never know whether the machine you boot is secure boot enabled. Whether booted via DVD, USB, network.., to get this to work generically, you simply always boot via shim.

Next question is whether the shim can only boot the grub executable signed with the distro specific key or whether still the Microsoft/global key is used. But you said you tested this? Unfortunately the guy in charge is ill for a whole week already and others are sitting somewhere around the world. I try to find out more.

[philseeley]

JFI: Latest OVMF kvm extensions can do secure boot.

True. Unfortunately for some hypervisors we're fixed on RHEL7 and have confirmed with Red Hat that they're not going to support the secure boot OVMF on RHEL7, but do support it on RHEL8.

Next question is whether the shim can only boot the grub executable signed with the distro specific key or whether still the Microsoft/global key is used. But you said you tested this? Unfortunately the guy in charge is ill for a whole week already and others are sitting somewhere around the world. I try to find out more.

We've done some more testing and can confirm that the keys are actually vendor specific. So you can use the shim and grub from RHEL7 to install RHEL8 and vice versa. But you cannot mix shims, grubs and kernels between vendors, e.g. the RHEL shim won't load the CentOS grub. Note: we've only tested RHEL7/8 and CentOS7/8.

I would also think it would be unwise of us to rely on the observed interoperability of shim/grub between distros from the same vendor, e.g. for RHEL9 Red Hat might use a different signing key from RHEL8. So I think it's safest that any solution assumes that each distro has different shims and grubs.

[watologo1]

You would then either need to get this or a shim signed by the Microsoft CA.

This is what shim is for? Every distro provided shim should be Microsoft CA signed. And every server should have a Microsoft key stored in firmware. Then you take the corresponding grub efi executable.

This (shim/grub.efi) pair should automatically come from the distribution your cobbler server is running on, right? I still need to talk and understand more in detail what happens if you want to install arbitrary distributions with a static shim/grub.efi pair and secure boot key management, etc. This is complex. But if we need to provide different shim/grub.efi pairs (extracted from the install repo?!?) no additional or duplicate grub.cfg config code should be added. We need to find a mechanism to let these use the general grub.cfg entry point.

[philseeley]

This (shim/grub.efi) pair should automatically come from the distribution your cobbler server is running on, right?

The shim/grub.efi pair should be for the distributions you're trying to deploy, not the cobbler server's distro.

[thiller2018]

Could https://github.com/ValdikSS/Super-UEFIinSecureBoot-Disk help? It says that it can secureboot all distributions.