Allow to specify the root and boot partitions

[veigaMak]

Firstly I want to congratulate this amazing project.

I'm trying to personalize an image. My image is not the default RasPiOS. I'm implementing A/B boot and because of that I have a third partition (before the default two). So my partition table is something like this, where partition 2 and 3 are the 1 and 2 partitions in the default image: Partition 1: fat32 partition for AB boot; Partition 2: fat32 boot partition; Partition 3: ext4 root partition.

My question is if its possible to specify the boot and root partitions when customizing the image? Something like

sudo sdm --customize --boot-partition 1 --root-partition 2 --restart 2023-12-05-raspios-bookworm-arm64.img

If not, do you have any recomendation on how to create a partition before those two partitions? (it's mandatory that this new partition is the first one)

[gitbls]

Just an acknowledgement that I've read this. Thinking about it and what could be done. I'll undoubtedly have some questions later.

Is this Fat32 AB boot partition generally available somewhere?

[veigaMak]

The AB boot documentation can be a bit hard to find. I have two main sources Raspberry Documentation and Raspberry forum.

But a brief explanation on what my partitions are:

• 1 => FAT32

  This only has the file autoboot.txt. This file decides which partition should boot.

  [all]
  tryboot_a_b=1
  boot_partition=2
  [tryboot]
  boot_partition=5

• 2 => FAT32

  RasPiOS default boot partition.

• 3 => EXT4

  RasPiOS default root partition.

I'm trying to customize the image with those 3 partitions (at least 2 aditional partitions are added after, but they are not included in the .img file).

[gitbls]

So, you're asking if sdm can support autoboot/tryboot? That would seem to me to be much more useful for sdm than to add even more specific partition slinging.

I took a look at the pointers you provided (thanks, btw, it clearly points to a problem ripe for some sdm-ization), and it looks generally straightforward. The sdm --burn command could get a new switch that enables a special autoboot/tryboot burn, similar to the way --convert-root is done. Need some thought as to how you'd specify exactly what is to be done (for example and no commitments! autoboot contents, what img goes in each bootfs/rootfs), customizations of each of those (?), etc.

Does what I described above address this sufficiently for you? Appreciate your feedback.

[veigaMak]

So, you're asking if sdm can support autoboot/tryboot?

My idea is more broad than support for autoboot/tryboot. My idea was to, when customizing an image, the default behavior is that the first partition is the bootfs and the second the rootfs. But I can pass a switch like root-partition 3 that change that default behavior. In my opinion it makes more sense to add that to the customization step rather than the burn step.

I'm not super familiar with the code, but in my mind this is "simple", rather than using loop0p1 and loop0p2 is to use what is specified in the arguments.

Need some thought as to how you'd specify exactly what is to be done (for example and no commitments! autoboot contents, what img goes in each bootfs/rootfs), customizations of each of those (?), etc.

I don't know if this is what you are asking but in general is something like this (I don't want to share any specific code as this is company code):

# Use truncate to create a blank image

# Create 3 partitions in that image, 1 and 2 with fat32 and 3 with ext4

# Add the autoboot.txt file to the first partition

# Clone partitions from the default RasPiOS to this personalized image
# Partition 1 from the default image to the partition 2 from my image
# Partition 2 from the default image to the partition 3 from my image

# Use sdm to install things on this image etc..

For now I found a workaround, I download a default image, use sdm to install things on that image and then I do the process above.

[gitbls]

Interesting idea. My thinking: If sdm were to get autoboot/tryboot support added to it, it's easy to explain and there is relatively direct benefit to users of it.

If I understand what you're suggesting, it creates something that solves YOUR problem, but isn't of general usefulness and ease-of-use, and explainable as autoboot/tryboot.

Today sdm does nothing with the IMG partition structure before, during, or after a customization, other than extending rootfs. From the sdm perspective, what you are describing is much more like a specialized burn process. This is exactly how --convert-root is done.

You do realize that a burn is in many ways like customize, in that you can run plugins after the burn is done? And, there are burn plugins, which enable you to manipulate the unmounted burned disk after the burn is complete.

The most straightforward way to implement what you want is to use a burn plugin that runs after you do a --burnfile. Your burn plugin would then create the final output disk and do the partition whanging that makes your final disk. The steps would look pretty much as you've written them. Burn plugins to look at: explore, extractfs, and parted

[veigaMak]

I was not familiarized with --burnfile. I was hesitant to add this to the --burn command as I think it is much more useful to add this functionality to something that can output an .img file (not burn directly into a device). From what I can see using --burnfile generates a .img file, so adding this functionality to that seems appropriate.

If I understand what you're suggesting, it creates something that solves YOUR problem, but isn't of general usefulness and ease-of-use, and explainable as autoboot/tryboot.

I understand that it seems like it only solves my problem but I think adding something like this can open the doors that allow to create an img that can be used with dual boot for example.

[gitbls]

I understand that it seems like it only solves my problem but I think adding something like this can open the doors that allow to create an img that can be used with dual boot for example.

I'm all ears to hear what doors you think are opened. WRT dual boot, I would likely opt to add an sdm feature to install dual boot than add some partition slinging. Again, looking for direct benefit to users, which your suggestion doesn't do, in an of itself.

And, apologies, but doing any partition slinging during customize is a complete non-starter from my perspective.

[stb68]

WRT dual boot

Just to stick my paddle in...

I've always been interested in the idea of dual boot, especially in the sense of having two sets of boot / root partitions, with one set as a sort of always there backup. Then, if there is an issue, corruption, upgrade problem I could switch the boot to the working setup when required without finding the right card, pulling things out of the way, etc.

I have used a multi-boot environment to try out different OSes and can see that there would be other cases where I would want to do this, for example -

I am regularly booting into a Manjaro image to try that with a view to long term use but occasionally I want to boot a stable Raspberry OS which I have setup with SPI programming (for certain microcontrollers when they get in a state and won't program through USB on my laptop). Currently I'm using two Pis rather than swap cards (so clutter, wiring, USB/HDMI switch).

So I can see it being useful. Whether it's useful for typical users of SDM burning large numbers of cards (which isn't me) might be another question though. Though maybe in Education or if any Pi sellers use it for producing cards!

[gitbls]

To be clear, I have two issues with the original proposal:

• The biggest issue is that it be done on --customize. The customize command has knowledge of the IMG format (exactly two partitions with bootfs and rootfs being the biggest issue). The customize command does NO mucking with the IMG partitions, as previously mentioned. The request adds a lot of complexity to customize and is inconsistent with sdm's operation

• I'm perfectly willing to consider adding a specialized burn that will burn to a device (--burn) or IMG (--burnfile) with the additional partition(s) added.

If it were done as a specialized burn switch the burned disk or IMG could have 3 partitions (FAT32, bootfs, rootfs). The operation would be as follows:

• Customize a vanilla IMG as desired with packages, config files, etc., as done today
• Use a new TBD switch that requests the output burn include the first FAT32 partition (created, but no files in it), then the bootfs and rootfs from the source IMG
• The result would be an IMG or disk with 3 partitions on it. The new FAT32 would have nothing in it
• As part of the burn command, use a burn plugin to populate the new FAT32 in the IMG or disk after the burn has completed, if desired. This burn plugin would fill in the contents of the new FAT32.
• Similarly, sdm could implement a new burn plugin to implement/populate tryboot functionality so that's easily available for users.

@veigaMak hinted that --burnfile might be workable for him...I'd like him to ask some questions about what I've described so that he completely understands and is clear on what would be implemented and thus can confirm that this would work for him.

@stb68 always appreciate you sticking your paddle in 🤣

[veigaMak]

@veigaMak hinted that --burnfile might be workable for him...I'd like him to ask some questions about what I've described so that he completely understands and is clear on what would be implemented and thus can confirm that this would work for him.

If I understood correctly, there would be 3 new plugins that could be use with --burn and --burnfile:

• Plugin 1: Create the new empty FAT32 partition before the bootfs and rootfs
• Plugin 2: Populate the new FAT32 partition
• Plugin 3: Easy tryboot (something that would use the functionality from the previous two plugins. It would create the new partition and populate it with the correct file so that it can boot)

The whole process will then be:

• Use sdm --customize to install packages, user, config files etc
• Use sdm --burnfile to create a new .img file with the correct partitions for tryboot use.

Another question that I have: This will generate an img (or burn directly to a device) with 3 partitions so that the person using sdm can use dual boot for example. Maybe, the rootfs partition should not be expanded to use the full remaining disk space because this will mean that there would be no remaining space for a second OS. I'm not sure I have a suggestion on how should sdm approach this. In my case I create 5 partitions with fixed size (1 FAT32 and 2 OS, each one with a bootfs and a rootfs) and a 6 partition that will fill the remaining disk size. But I'm not sure that this solutions is the best for the general user as I know that I have a minimal disk size of 250GB and so I can afford to use a rootfs partition with more that enough sufficient size for my use case.

I don´t know if I explained it well enough 😅. Any question please let me know. Other than that It seems it would work for me and for new people playing with tryboot.

Then, if there is an issue, corruption, upgrade problem I could switch the boot to the working setup when required without finding the right card, pulling things out of the way, etc.

This is exactly my use case of tryboot. In the company I work for we use sdm to burn a large number of SSDs with multiple partitions so we can do full update (OS, kernel, etc) without worries, because we have a working backup.

[gitbls]

Thanks for your feedback.

A clarification on plugins: The burn process itself is NOT implemented as plugins; it's part of the core sdm code. At burn time, plugins (both "standard plugins" and "burn plugins") are run AFTER the burn process has completed.

I've considered restructuring the burn process into plugins, but it's a fair amount of work with unclear payback. Not to mention that it could break existing workflows and scripts.

So, in the burn process, if --tryboot (for example) were specified, the burn process would:

• Create/populate the new empty FAT32 partition
• Copy the desired bootfs/rootfs onto the target disk/IMG (e.g., burn them) as partition 2 and partition 3
• Run specified plugins and burn plugins on that just-burned bootfs/rootfs

There may be a need for a tryboot plugin to complete and enable desired tryboot capabilities. (your plugin 3). This would run after the burn has completed. I would look for your inputs on this (once designed) to ensure that it meets your needs as well.

At the moment, sdm has widespread knowledge that an IMG (and burned disks/IMGs) have exactly two partitions. This will need to be fixed, obviously as part of this enhancement.

The steps you've outlined (customize then burnfile) are exactly correct. The devil is in the details, as they say 🤔

Regarding your "another question": sdm has a --no-expand-root switch that disables the automatic rootfs expansion. Is that what you want?

And, FYI, the parted burn plugin can be used to create additional partitions after the burn has completed. But, this may not be sufficient for your needs, as it simply creates partitions and file systems on those partitions.

The previously-mentioned tryboot burn plugin could append additional IMGs to the end of the burned disk/IMG . These may need to be fully-customized, as it may not be feasible to, for instance, run plugins against these additional IMGs as part of this process, at least not in the first go.

The goal is for sdm to automate the whole process as much as possible, so your thoughts are welcome.

[veigaMak]

Sorry for the delayed response. But yeah I see no problem with everything you said.

I would look for your inputs on this (once designed) to ensure that it meets your needs as well.

Of course.

[gitbls]

OK, thx. I've been looking at the code, starting to figure out the work. More soon.

[Lastin]

Seems like this issue is somewhat related to mine. I'm trying to figure out the way to preserve partitions when burning. I got 3 partitions: 1: FAT32 (for boot) /dev/nvme0n1p0 2: EXT4 (for root) /dev/nvme0n1p1 3: EXT4 - persistent storage /dev/nvme0n1p2

Is there a way to burn new image into /dev/nvme0n1p0 and /dev/nvme0n1p1 while preserving the partition table and /dev/nvme0n1p2 partition?

[gitbls]

Seems like this issue is somewhat related to mine. I'm trying to figure out the way to preserve partitions when burning. I got 3 partitions: 1: FAT32 (for boot) /dev/nvme0n1p0 2: EXT4 (for root) /dev/nvme0n1p1 3: EXT4 - persistent storage /dev/nvme0n1p2

Is there a way to burn new image into /dev/nvme0n1p0 and /dev/nvme0n1p1 while preserving the partition table and /dev/nvme0n1p2 partition?

This question is completely different than the others in this thread, and more appropriately belongs in a new issue, but NVM that now.

sdm assumes that it "owns" the entire burn disk, and has no capability to automatically preserve and restore a partition. There are significant issues, such as

• What if the disk you want to burn has a larger 1st or 2nd partition than the one on your NVMe?
• What if you want to convert the NVMe from MBR partition format to GPT?

That said, you can certainly use sdm to do that manually by (rough outline):

• Save the partition you want to preserve (/dev/nvme0n1p2) to a scratch disk via dd
• Note the size of this partition
• Burn your new IMG to the NVME, and use the burn plugin parted to create a third partition with the size you noted in the above step
• Restore the preserved partition from the scratch disk back to the newly-created 3rd partition after the burn completes

If you have any further questions on this or would like further guidance, please open a new issue rather than piling onto an unrelated topic. Thx

[Lastin]

Thanks for getting back. Apologies for mixing up the subjects.