The BlueField boot flow comprises 4 main phases:
When booting from eMMC, these stages make use of two different types of storage within the eMMC part:
ATF and UEFI come from a special area known as an eMMC boot partition. Data from a boot partition is automatically streamed from the eMMC device to the eMMC controller under hardware control during the initial bootup. Each eMMC device has two boot partitions, and the partition that is used to stream the boot data is chosen by a non-volatile configuration register in the eMMC.
The operating system, applications, and user data come from the remainder of the chip, known as the user area. This area is accessed via block-size reads and writes, done by a device driver or similar software routine.
In most deployments, BlueField's ARM cores are expected to obtain their software stack from an on-board Embedded Multi-Media Card (eMMC) device. Even in environments where the final OS kernel is not kept on eMMC -- for instance, systems which boot over a network -- the initial booter code will still come from eMMC.
Most software stacks need to be modified or upgraded at some point in their lifetime. Doing this on a fielded BlueField system involves some risk; if there's a severe problem with the new software, system operation may be impaired to the extent that it is impossible to do further modifications to fix the problem. Ideally, one would like to be able to install a new software version on a BlueField system, try it out, and then fall back to a previous version if the new one does not work. In some environments, it's important that this fallback operation happen automatically, since there may be no physical access to the system. In others, there may be an external agent, like a service processor, which could manage the process.
In order to satisfy the requests listed above, we do the following:
Provide two software partitions on the eMMC, 0 and 1. At any given time, one area is designated the primary partition, and the other the backup partition. The primary partition is the one we'll boot from on the next reboot or reset.
Allow software running on the ARM cores to declare that the primary partition is now the backup partition, and vice versa. (For brevity, in the remainder of this document, we'll term this "swapping the partitions", although we're just modifying a pointer; the data on the partitions does not move.)
Allow an external agent, like a service processor, to swap the primary and backup partitions.
Allow software running on the ARM cores to reboot the system, while starting an upgrade watchdog timer. If the upgrade watchdog expires, the system will reboot, but the primary and backup partitions will first be swapped.
Access to all the boot partition management is via a program shipped
with the BlueField software called "mlxbf-bootctl". The binary is
shipped as part of our Yocto image (in /sbin) and the sources are
shipped in the "src" directory in the BlueField Runtime Distribution.
A simple make
command will build the utility.
The mlxbf-bootctl syntax is as follows:
syntax: mlxbf-bootctl [--help|-h] [--swap|-s] [--device|-d MMCFILE]
[--output|-o OUTPUT] [--read|-r INPUT]
[--bootstream|-b BFBFILE] [--overwrite-current]
[--watchdog-swap interval | --nowatchdog-swap]
--device: Use a device other than the default /dev/mmcblk0
--bootstream: Write the specified bootstream to the alternate
partition of the device. This queries the base device
(e.g. /dev/mmcblk0) for the alternate partition, and uses that
information to open the appropriate boot partition device,
e.g. /dev/mmcblk0boot0.
--overwrite-current: Used with "--bootstream" to overwrite the current
boot partition instead of the alternate one (not recommended)
--output: Used with "--bootstream" to specify a file to write the boot
partition data to (creating it if necessary), rather than using an
existing master device and deriving the boot partition device.
--read: Read a bootstream and convert it back to a BFBFILE specified
by --bootstream. For example use "mlxbf-bootctl --read /dev/mmcblk0boot0
--bootstream current.bfb" to read the current bfb installed on boot
partition zero.
--watchdog-swap: Arrange to start the ARM watchdog with a countdown
of the specified number of seconds until it fires; also, set the
boot software so that it will swap the primary and alternate
partitions at the next reset.
--nowatchdog-swap: Ensure that after the next reset, no watchdog
will be started, and no swapping of boot partitions will happen.
To update the boot partition on the ARM cores, let us assume we have a new bootstream file, e.g. "bootstream.new". We would like to install and validate this new bootstream. To just update and hope for the best, you can do:
mlxbf-bootctl --bootstream bootstream.new --swap
reboot
This will write the new bootstream to the alternate boot partition, swap alternate and primary so that the new bootstream will be used on the next reboot, then reboot to use it.
(You can also use --overwrite-current
instead of --swap
, which will
just overwrite your current boot partition, but this is not
recommended as there is no easy way to recover if the new booter code
does not bring the system up.)
With a BMC (board management processor) you can do better. The ARM cores simply notify the BMC prior to the reboot that an upgrade is about to happen. Software running on the BMC can then be implemented that will watch the ARM cores after reboot. If after some time the BMC has not seen the ARM cores come up properly (for whatever definition is appropriate for the application in question), it can use its USB debug connection to the ARM cores to properly reset the ARM cores, first setting a suitable mode bit that the ARM booter will respond to by switching the primary and alternate boot partitions as part of resetting into its original state.
Without a BMC, you can use the ARM watchdog to achieve similar results. If something goes wrong on the next reboot and the system does not come up properly, it will reboot and return to the original configuration. In this case, you might run:
mlxbf-bootctl --bootstream bootstream.new --swap --watchdog-swap 60
reboot
With these commands, you will reboot the system, and if it hangs for 60 seconds or more, the watchdog will fire and reset the chip, the booter will swap the partitions back again to the way they were before, and the system will reboot back with the original boot partition data. Similarly, if the system comes up but panics and resets, the booter will again swap the boot partition back to the way it was before.
You must ensure that Linux after the reboot is configured to boot up
with the sbsa_gwdt
driver enabled. This is the SBSA (Server Base
System Architecture) Generic WatchDog Timer. As soon as the driver is
loaded, it will start refreshing the watchdog and preventing it from
firing, which will allow your system to finish booting up safely. In
the example above, we allow 60 seconds from system reset until the
Linux watchdog kernel driver is loaded. At that point, your
application may open /dev/watchdog explicitly, and the application
would then become responsible for refreshing the watchdog frequently
enough to keep the system from rebooting.
For documentation on the Linux watchdog subsystem, see the Linux watchdog documentation:
https://www.kernel.org/doc/Documentation/watchdog/watchdog-api.txt
For example, to disable the watchdog completely, you can run:
echo V > /dev/watchdog
You can incorporate other features of the ARM generic watchdog into your application code using the programming API as well, if you wish.
Once the system has booted up, in addition to disabling or reconfiguring the watchdog itself if you wish, you should also clear the "swap on next reset" functionality from the booter by running:
mlxbf-bootctl --nowatchdog-swap
Otherwise, next time you reset the system (via reboot, external reset, etc) it will assume a failure or watchdog reset occurred and swap the eMMC boot partition automatically.
The above steps can be done manually, or can be done automatically by software running in the newly-booted system.
The above solutions simply update the boot partition to hold new booter software (ATF and UEFI). If you want to also provide a new kernel image and/or new userspace, you should partition your eMMC into multiple partitions appropriately. For example, you might have a single FAT partition that UEFI can read the kernel image file from, but the new bootstream contains a UEFI bootpath pointing to an updated kernel image. Similarly, you might have two Linux partitions, and your upgrade procedure would write a new filesystem into the "idle" Linux partition, then reboot with the bootstream holding kernel boot arguments that direct it to boot from the previously idle partition. The details on how exactly to do this depend on the specifics of how and what need to be upgraded for the specific application, but in principle any component of the system can be safely upgraded using this type of approach.