Installing Tow-Boot without a protective partition is heavily discouraged

Looking at the installation instructions, I see this recommends installing Tow-Boot "just like U-Boot", by haphazardly slapping bytes into offsets on the shared storage media.

This installation scheme is what, within Tow-Boot, I call "Legacy install".

The main issue with that is that there is now an invisible presence that needs to be accounted for any time the user needs to do operations on the storage media (e.g. partitioning, installing a distro).

This is also a concern for upgrades, as it requires yet again haphazardly slapping bytes into the offsets.

Finally, an upcoming concern is that this will prevent using the "with environment" variants for shared storage. What this means is that the user will be unable to save configuration option from the planned user interface, like the boot order. This is because the plan is to write the environment at the end of the protective partition for the firmware. Without a protective partition we can't guarantee using any location on the storage media is safe.

Steps to remediate

I hope it's easy enough. Since you're always targeting fresh installs, recommend starting from the shared.disk-image.img, which the end-user should dd at start of the storage media. No need for obscure offsets.

Then, when adding partitions for the operating system, make sure you start with the existing partitions, and add partitions.

I'm open for further questions if anything I described here isn't clear :).

@samueldr

I'm open for further questions if anything I described here isn't clear :).

What you describe is pretty clear, except this part:

Finally, an upcoming concern is that this will prevent using the "with environment" variants for shared storage. What this means is that the user will be unable to save configuration option from the planned user interface, like the boot order. This is because the plan is to write the environment at the end of the protective partition for the firmware.

So will you write the configuration before sector 32768, or on/after it? If it is before it, no harm done. Users are instructed to start the first partition on sector 32768, so it will not conflict.

The main issue with that is that there is now an invisible presence that needs to be accounted for any time the user needs to do operations on the storage media (e.g. partitioning, installing a distro).

This is not an issue for Arch Linux (ARM) users. As shown in the installation instructions for the Rock64, writing directly to the disk is a valid way to install a bootloader. This also follows the Arch Linux philosophy mostly.

Simplicity:

Arch Linux defines simplicity as without unnecessary additions or modifications.

User centrality:

Whereas many GNU/Linux distributions attempt to be more user-friendly, Arch Linux has always been, and shall always remain user-centric. The distribution is intended to fill the needs of those contributing to it, rather than trying to appeal to as many users as possible. It is targeted at the proficient GNU/Linux user, or anyone with a do-it-yourself attitude who is willing to read the documentation, and solve their own problems.

Now of course, there is also:

Pragmatism:

Arch is a pragmatic distribution rather than an ideological one. The principles here are only useful guidelines. Ultimately, design decisions are made on a case-by-case basis through developer consensus. Evidence-based technical analysis and debate are what matter, not politics or popular opinion.

And another part of simplicity:

It (Arch Linux) ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes: patches not accepted by upstream are avoided, and Arch's downstream patches consist almost entirely of backported bug fixes that are obsoleted by the project's next release.

So of course, this means that I am open to the issue and want to find the best way forward. 😃

I think that the dd method might be valid for Arch Linux ARM to write the bootloader to the disk directly. That way, it will make it easier and simpler to read and write a partition layout without any constraints (=the Arch Linux way), even if this would be a GPT partition layout or something else exotic (that we both haven't thought of yet).

Now, you mention that there is a desire to read and write configuration parameters in the space that Tow-Boot uses (as discussed earlier). A great idea, especially if the user can dynamically decide/change the stored boot order! However, if this requires storage space after sector 32768, I think that this should only be done if the bootloader is loaded from SPI, since it is unlikely that the user will interact with this space in any way.

Have you considered the option to save and load the boot configuration in/from a/any available partition that U-Boot can read (FAT, ext4, ...) for disk-based installs? That way, it would be easier for a user to manually review (especially if it is plain text or a standard U-Boot environment 'file'), backup and restore the configuration, which would be troublesome/more work with dd commands or additional tools. Tow-Boot already loads extlinux/extlinux.conf from supported partitions. Could it not be coded to read the configuration in such a way as well, earlier in the boot process?

For Arch Linux (ARM) users, that is not necessary as long as no data is written on or after sector 32768. I just think it would be a neat addition that would make Tow-Boot even more friendly to use.

Happy to think with you. Let me know your thoughts.

About simplicity

IMO, there is no simplicity in having an opaque invisible blob of a firmware living on a user-owned storage. So it must be made extremely obvious that there's this wart here that has meaning. In the end I think it is more simple that everything is laid bare in explicit terms.

[...] As shown in the installation instructions for the Rock64, writing directly to the disk is a valid way to install a bootloader. This also follows the Arch Linux philosophy mostly.

While it is the way U-Boot is installed almost universally, one of the aspects I bring forward with Tow-Boot is that doing it this way is wrong :).

I think that the dd method might be valid for Arch Linux ARM to write the bootloader to the disk directly.

Yes, it should work. The main difference is end-users don't create the partition table, and add to the existing table.

And additionally, the installation steps becomes universal for every target supported by Tow-Boot! Instead of requiring platform-specific knowledge about where to write the firmware, the firmware is written in a universal way. Now that's what I call simple.

About alternatives to saving in the partition

Have you considered the option to save and load the boot configuration in/from a/any available partition that U-Boot can read

Yes, and it is problematic since it can be any available partitions, including one from the "wrong" storage medium. At least, my current understanding of the available options. It also transgresses the line between what is "owned" by the firmware or not. Saving the environment on a partition that should be user owned is, IMO. rude :).

Though such a scheme will be required because of the Raspberry Pi family of hardware. The partition has to be a FAT32 filesystem. While we could have the filesystem not fill the partition, it's a risk that users might expand it even though they shouldn't touch it.

So maybe such a scheme would be usable by advanced end-users. But it wouldn't be the recommended setup since writing to a partition that isn't the firmware partition transgresses that line about ownership.

Way forward

The "shared" strategy firmware will not write to a fixed offset. Too dangerous. While ALARM and your instructions should be safe (gives ~16MiB of free space), other setups might not.

This is why there will always be a noenv variant that can be used, which will be provided for "retrofitting" Tow-Boot into existing setups. This one will never write to the user's storage. That's what your current setup will end up using.

Though, with the current instructions, existing users could add a protective partition... Except that it will probably rely on the partition table to be GPT. I'm not familar enough with parted to be sure, but I would hazard a guess that the instructions will produce an MBR partition scheme.

Luckily, I believe it should be fine to convert the partitions in-place to GPT for existing users. The only issue I can see is the the backup table at the end of the disk. Though I believe that tools that handle conversion handle that fine.

I'll be curious to hear your opinions and concerns still :).

Oh, and I'll concede, it is a change from the "normal". It can get uncomfortable at first :).

Let me first note that in Arch Linux land, there is no concept of storage space that is 'owned' by someone else than the user. Everything is owned by the user. The user builds up their own system (hence the lack of an installer). The user is given the opportunity to be familiar with and adjust (automated) update processes. The user is allowed to screw up. The user is allowed to repair their system. That is what makes Arch Linux (ARM) 'user centric'.

Now if you talk about storage space and resources being reserved for a bootloader and that it should not be touched by the operating system under normal circumstances, I think we are talking about the same thing using using different terms. 😉

While it is the way U-Boot is installed almost universally, one of the aspects I bring forward with Tow-Boot is that doing it this way is wrong :).

What you consider wrong is considered right in the Arch Linux (ARM) philosophy. As hinted on the PINE64 forum, it is unlikely that I will deviate from this, since I want a system that is as close to stock Arch Linux ARM as possible. Installing a bootloader should not force the user to use a specific (provided) partition table.

A good way to approach the needs of a specific bootloader is to document it, and let the user decide how to approach their partitioning needs when they install the system. If a user decides to follow the pure installation instructions and not make a protective partition (like for all other Arch Linux ARM supported devices), then that is fine. If the user decides to create a protective partition, then that is fine as well. Normal software will not touch the reserved space for a bootloader before sector 32768 if it does not contain a mounted partition. A warning could be added when installing the Tow-Boot package for the user to notify of the reserved space.k-ups

That takes care of user awareness and of any technical hiccups that can occur.

IMO, there is no simplicity in having an opaque invisible blob of a firmware living on a user-owned storage. So it must be made extremely obvious that there's this wart here that has meaning.

Agreed.

In the end I think it is more simple that everything is laid bare in explicit terms.

It already is. Check what is shown to the user, every time the package is upgraded. In the traditional U-Boot way, the bootloader is not updated until the Arch Linux ARM user deems it necessary. At every upgrade the user is reminded of this, and of how to do the upgrade themselves. Of course, upgrading U-Boot is never really vital, so manual user intervention to upgrade it is a valid solution in the Arch Linux (ARM) ecosystem.

On the Tow-Boot site I read:

Update should be handled out-of-band (through a firmware update option in the firmware, or from distro-independent systems like fwupd). The firmware is not "owned" by the currently running distro. The firmware is not updated or changed through package upgrades.

I do not think the way ALARM and Tow-Boot work have to conflict with each other. If Tow-Boot will not be automatically upgraded through user space, it will not automatically upgrade. Consider the ALARM package an installation package for it, similar to U-Boot. As noted earlier, a small warning can be added that if a user decides to manually (re-)install Tow-Boot, that any previous boot loader and its configuration will be overwritten.

About my earlier suggestion of making a configuration file on disk:

Yes, and it is problematic since it can be any available partitions, including one from the "wrong" storage medium. At least, my current understanding of the available options. It also transgresses the line between what is "owned" by the firmware or not. Saving the environment on a partition that should be user owned is, IMO. rude :).

The Tow-Boot site states:

Boot order is unsurprising: on shared storage, the storage from which the currently running Tow-Boot is running is prioritized by default.

What Tow-Boot can do is to prioritize the (first or active) partition on the storage medium from which it is running to work with data from a file. Considering the philosophy of Tow-Boot, I now do understand why you find this undesirable. So let us not explore this option further.

The "shared" strategy firmware will not write to a fixed offset. Too dangerous. While ALARM and your instructions should be safe (gives ~16MiB of free space), other setups might not.

The instructions given in this repository are not just the ALARM way, but the semi-official Manjaro ARM (and I believe the former Debian distribution) way on the Pinebook Pro as well. As far as I know, every installation instruction reserves everything after sector 63 and before sector 32768 for all stages of the bootloader (without a protective partition). If you say you need all of this space and only this space, I do not think it is required to deviate from the current way on how Tow-Boot is installed (with minor adjustments of how it is now, if there is only one binary blob to write to sector 64 and beyond).

If you ever desire to have more space, then I am afraid that you will find it very hard to get Tow-Boot become popular on Pinebook Pro and other Rockchip-based devices, since I see the unwritten rule of the first partition sector being 32768 returning everywhere. I assume that 16 MB is plenty, also because it aligns with the Pinebook Pro's SPI flash (which is 16 MB as well).

Therefore my suggestion (for ALARM):

If you decide not to load and save the Tow-Boot configuration from a file on a partition, consider the full space after sector 63 and before sector 32768 space reserved for Tow-Boot for reading and writing, and do not worry about what the user does. Leave it to others to take care that this space is not overwritten by accident by something else, either by the system or by the user.

If you want me to add an FAQ entry on why this space is reserved and why users should not mess with it unless they know what they are doing, that can be arranged. You can reference this FAQ entry if you every have any Arch Linux ARM users approach you directly if they screw up their own systems. I do not expect many users to do this, since anyone finding this repository and installing Arch Linux ARM on their Pinebook Pro are probably technical proficient enough to understand what they should and should not do, especially if they are instructed correctly.

The towboot-pinebookpro-bin ALARM package description is updated manually by me for now, unlike most upstream Manjaro ARM package descriptions. Expect an update soon with the new pre-release, once I finished testing.

Once Tow-Boot is formally released and stabilized (in terms of building environment and instructions, output filenames, etc.), I will consider making a towboot package description that also creates other relevant packages (including towboot-pinebookpro). This should help it to be included in other Arch Linux-based distributions, such as Manjaro ARM. Of course, I cannot vouch for any changes others will make if they decide to adopt it.

~~I'll be brief~~ I thought I was going to be brief, maybe a bit flippant, but the whole argument hinges around "we've always been doing this way". Hopefully I don't sound too harsh on a personal level.

With Tow-Boot I want the whole thought process about the firmware, not a bootloader, but around the firmware to be refreshed. And this requires rethinking some parts of how distributions handle it.

The intended way distributions should handle the firmware for a computer is to not handle it. And this is how Tow-Boot was designed. The user installs the firmware with a unique instruction set, and from that point on, the installation process for any distribution or operating system should be generic, and hopefully universal.

This is why we strive for EBBR compliance. This way, a standard UEFI install, just like on x86_64, simply works.

In fact, my hope is that I this can cause enough of a reaction across vendors such that they:

always provide an SPI Flash to store the firmware
provide an working standards-based firmware in the SPI Flash

This way, distros can stop having to produce board-specific instructions, and worse, board-specific images with the firmware pre-installed.

I also hope this can cause enough of a stir in distros such that they:

refer to installing any standards-compliant firmware (e.g. Tianocore for Raspberry Pi, or stock U-Boot)
produce universal images and instructions

This is already what some distributions I'm not involved with claim to do, or actually work towards.

So this is why installing to eMMC also is heavily discouraged. In fact thinking back, I should have led with that. It's not that installing without a protective partition is discouraged (though it is). Really installing to any shared storage is heavily discouraged, as it goes against this new look we should be using to look at the initial boot firmware.

So most things "usually done" with U-Boot is legacy. Some of it comes from platforms where you couldn't do better (without dedicated firmware storage). Some comes from, and not your fault alone, simply doing what has always been done.

From this discussion I think I see I need to better explain and trace a line in the sand about what should be seen as "the old ways", and what is "the future (actually the present)".

Sorry I'm not answering to your actual comment. Though as I hope I was able to get across, it builds on "the old ways", so really there's not much I can really answer for any point except "that's now how we all collectively should be doing things anymore".

Oh, and I don't want to force this discussion to an end. Really I wanted to try to clearly restate the issue, and where I'm coming from about it.

This is why we strive for EBBR compliance. This way, a standard UEFI install, just like on x86_64, simply works.

A noble cause.

The intended way distributions should handle the firmware for a computer is to not handle it. And this is how Tow-Boot was designed. The user installs the firmware with a unique instruction set, and from that point on, the installation process for any distribution or operating system should be generic, and hopefully universal.

I do not believe EBBR states how a firmware should initially be installed. The best method for me seems to be through operating system packages, especially on platforms that do not initially contain a self-updating firmware (of even a firmware at all). Note that I write installation, not upgrade. I understand your desire to not upgrade the firmware through regular packages.

Users can already do this with Arch Linux ARM on Pinebook Pro. In the new version of the towboot-pinebookpro-bin package, I added a short note upon installation or upgrade on how it can be installed to SPI flash, and that installing the firmware to either disk or flash will overwrite the previously installed firmware and its configuration. Furthermore, the language is explicit ('A new Tow-Boot version can be installed'), which is enough for Arch Linux ARM users to realize that it is optional.

So from a technical perspective, so far nothing is done 'wrong' here.

So this is why installing to eMMC also is heavily discouraged. In fact thinking back, I should have led with that. It's not that installing without a protective partition is discouraged (though it is). Really installing to any shared storage is heavily discouraged, as it goes against this new look we should be using to look at the initial boot firmware.

I get that you are focusing on new ARM-based devices with dedicated SPI flash that has something useful on it from the factory, but this does not apply to the Pinebook Pro or many other current devices. These need some firmware and bootloader loaded from something else (often eMMC or SD) in order to initially boot.

Just because something is heavily discouraged does not mean it should not be provided as an option to Arch Linux (ARM) users. Again, the user makes the choice.

From this discussion I think I see I need to better explain and trace a line in the sand about what should be seen as "the old ways", and what is "the future (actually the present)".

I would be careful with stating it like this. This is one possible future you envision. This is what I see, based on what you described. Stating that something is the future does not make it the future. It makes it seem like you are forcing something on people ('you have to do it this way!'), instead of showing people an alternative ('it could be done this way, with clearly defined advantage X, Y, Z') and letting them make up their own minds..

Oh, and I don't want to force this discussion to an end. Really I wanted to try to clearly restate the issue, and where I'm coming from about it.

Neither do I, and I appreciate the clarification.

I read the EBBR standard. Probably what started this whole discussion is this:

https://arm-software.github.io/ebbr/#section-mbr-parts

This is something I missed in your initial issue. Now I understand where this request is coming from, and I can answer you more directly. Taking that document, let's go back to its goal:

This Embedded Base Boot Requirements (EBBR) specification defines an interface between platform firmware and an operating system that is suitable for embedded platforms. EBBR compliant platforms present a consistent interface that will boot an EBBR compliant operating system without any custom tailoring required.

As far as I am aware, Arch Linux ARM does not have a stated goal to be EBBR compliant. As mentioned earlier, I want this repository to be as close to upstream as possible. Upstream currently does not 'dictate' (provide or instruct) the configuration of a protective partition for the boot sector, which is inherently why this repository does not do it either. This does not mean that a user is discouraged or even prevented from creating such a partition. It simply means that the default installation instructions do not mention it. This is similar to how LUKS encryption is not mentioned, for example. It is up to the user to customize their own boot if such a thing is desired. The reference installation instructions 'just work'.

Furthermore, I read in EBBR that MBR use is deprecated, and might not be allowed in the future. Technically, there is no need for the use of MBR on Pinebook Pro, as long as the bootloader (loaded from either SPI or from a hardcoded location on the storage device) supports whatever is used. The reason for why MBR is used and not GPT is because upstream uses it, because 'it just works', and because it is unlikely that a storage medium will be used which in terms of capacity is constrained by what an MBR can offer.

Now a user can use an NVMe drive that is more than 2 TB, but in Arch Linux (ARM) style the user can also figure out how to use it on their own. Similarly, if a user wants to make their instance of Arch Linux ARM on Pinebook Pro EBBR compliant, they are free to do so on their own accord.

If you want operating systems to be EBBR compliant, this repository is not the place to fight for it. Move upstream. If an EBBR compliant device has some constraints (such as that the included firmware demands the use of GPT, to be EBBR compliant with future versions), operating systems will move accordingly to support this, including Arch Linux ARM.

This repository follows Arch Linux ARM primarily not because of a philosophy, but because of practicality. Adding Pinebook Pro support to Arch Linux ARM is something I do in my free time, which is precious and limited. Less deviation from upstream is less upkeep I have to do on my own.

The reason why I initially chose Tow-Boot as the default boot loader was because of practicality as well. Other U-Boot versions were lagging behind a bit since they did not include a TF-A version that is needed to make Suspend to RAM on Pinebook Pro work. Furthermore, Tow-Boot is also nice because it has a menu that can be used to override the initial boot device. I incorporate Tow-Boot mostly in a way that is compliant to how upstream installs and uses bootloaders, with an additional instruction to flash it to SPI flash if the user desires.

To come to a point: if Arch Linux ARM decides that protective partitions are the way to go, then the instructions and root filesystem in this repository will be adjusted accordingly. If Tow-Boot would (most regrettably) become incompatible with the current way Arch Linux ARM is installed (on Pinebook Pro specifically or in general), it will probably be replaced with something else.

Since the discouragement of the absence of a protective partition comes from a document that is not recognized or adhered to in Arch Linux ARM, this issue can be closed without further action. This does not say that I do not support your efforts, specifically in creating this issue and your goal to make Tow-Boot EBBR compliant. Your efforts in realizing the latter are better spent somewhere else, though.

SvenKiljan / archlinuxarm-pbp