support for disk layout (partitions)

[dmuhamedagic]

I would like to create bootable images with more than just rootfs, i.e. to include several partitions. Something like what FAI does, e.g. in disk_config/LVM:

# <type> <mountpoint> <size>   <fs type> <mount options> <misc options>

# entire disk with LVM, separate /home

disk_config disk1 fstabkey:uuid align-at:1M

primary /boot   200     ext2    rw,noatime
primary -       4G-     -       -

disk_config lvm

vg vg1  disk1.2
vg1-root     /       3G-50G   ext4    noatime,rw
vg1-swap     swap    200-4G   swap    sw
vg1-home     /home   600-     ext4    noatime,nosuid,nodev,rw

[eduble]

Interesting. The added difficulty for implementing this in debootstick is the following: when we run the tool, we do not work directly with the final disk. Instead we output an image, with minimal size, without knowing the size of the disk it will be copied on. So that would mean we should implement the partition sizing logic when the generated system is first booted. And since this generated system may be a minimalistic one, we will be limited about the tools we have there.

[dmuhamedagic]

So that would mean we should implement the partition sizing logic when the generated system is first booted. And since this generated system may be a minimalistic one, we will be limited about the tools we have there.

True, but it would be up to the user to make sure that some tools such as lvm or parted would be required for such a feature. I'm still considering what would be the best way to go about it. My use case is a smallish rootfs and the rest of the disk occupied by one PV, with one VG, and a few LVs. At any rate, I'd need functionality from both debootstick and FAI. Unfortunately, the two cannot be so easily put to work together.

[dmuhamedagic]

So that would mean we should implement the partition sizing logic when the generated system is first booted.

But are you sure about that? FAI creates a bootable image complete with the partition table and all file systems on different partitions.

[eduble]

Well at least this is how debootstick works as of now. The people I know who are using it usually follow these steps:

1. generate an image with debootstick
2. copy it to a USB stick (using dd)
3. boot a physical machine with it

This explains why debootstick tries to generate an image as small as possible (we don't know what will be the size of the USB stick). And finally, when the system is first booted, it auto-expands on the USB stick. Generating an image has some advantages over manipulating the disk directly. Mainly, you can test the image by using kvm (with option -hda <image-file>), before dumping it on a USB stick. Do you use it differently?

[dmuhamedagic]

No, I've been using it exactly like that. Those are great features, both the auto-expand and the "install" type which copies the image to the local disk. debootstick does a really wonderful job. All I would need for this use case is that it creates more partitions. The goal is to have at least /tmp, /var, and /var/log as separate filesystems. Since debootstick already creates a VG for the rootfs, multiple LVs would also do the job for now. Would it perhaps be possible to limit the rootfs LV, i.e. that it doesn't take all the space, but only say 8G?

[eduble]

All I would need for this use case is that it creates more partitions. The goal is to have at least /tmp, /var, and /var/log as separate filesystems.

OK.

Since debootstick already creates a VG for the rootfs, multiple LVs would also do the job for now. Would it perhaps be possible to limit the rootfs LV, i.e. that it doesn't take all the space, but only say 8G?

I got the idea of just limiting the rootfs LV too. That would be the easiest, of course. But if you want to automate the whole imaging process, that would not be very easy to automate the creation of the partitions, after debootstick ends.

So, even if we do not offer the same level of detail you have with FAI, we can probably do more.

What about an option such as --partition-volume /var,35%?

[dmuhamedagic]

I got the idea of just limiting the rootfs LV too. That would be the easiest, of course.

In that case one could create more partitions also after the boot.

But if you want to automate the whole imaging process, that would not be very easy to automate the creation of the partitions, after debootstick ends.

Right. I took a peek at scripts/create-image/target/pc/formatting:partition_image() and, at least at that point, it looks like relatively easy to add new partitions. But I still don't know enough about the general structure of debootstick.

So, even if we do not offer the same level of detail you have with FAI, we can probably do more.

What about an option such as --partition-volume /var,35%?

Yes, that sounds great. BTW, do you think there could be a possibility to add some general hook mechanism to deal with this? How much effort would that be?

[eduble]

Yes, that sounds great.

Actually it would be --volume /var,35%. Because for a PC image, we use LVM, whereas for a raspberry pi, we use plain partitions, and the name --volume can fit both cases.

What about the filesystem? Do you think this option should handle several kinds of filesystems? For now, on PC, debootstick always creates an ext4 partition. And the choice of filesystem options appeared to be quite sensitive (a filesystem initialized on a small partition and later extended on a much larger disk can cause various boot problems). (Note that we might just keep for later the support of other filesystems in this option.)

BTW, do you think there could be a possibility to add some general hook mechanism to deal with this? How much effort would that be?

Maybe not so much. debootstick already uses a kind of plugin system internally, since it now handles 2 different target images (PC image and Raspberry Pi SD card image). Feel free to open another issue for this.

[dmuhamedagic]

I'm not sure about the filesystems either, but I think that for now ext4 should suffice. OK, I will open a new issue for the hooks. Thanks!

[eduble]

I started to work on this. Instead of the --volume /var,35% option, I will actually let the user provide an alternate disk layout configuration file, as you initially suggested.

First step is to modify code to use a default disk layout file. I will provide one for PCs and one for Raspberry Pis. I am working on this part. When you run debootstick, it will indicate that it is using this layout file (giving its file path). Next, I will provide one more option to specify an alternate one.

The default layout file for PC will contain:

# gpt or dos partition table
gpt

# oredered list of partitions (with mountpoint, type, size)
partition   /boot/efi  efi     auto
partition   none       bios    auto
partition   none       lvm     max

# lvm volumes (with mountpoint, type, size)
lvm_volume  /          ext4    max

About the size of a partition or lvm volume:

• auto means debootstick will select an appropriate size
• <xx>[G|M] (e.g. 1G or 50M) means debootstick should allocate exactly the specified size to this partition/volume
• <xx>% (e.g. 10%) means debootstick should allocate 10% of the remaining free space to this partition/volume
• max means debootstick should allocate any remaining free space to this partition/volume

Restrictions:

• only one lvm partition (partition with type=lvm) is allowed
• max size is only allowed:
  • on the last partition
  • on one lvm volume only
• <xx>% size notation is only allowed:
  • on the last partition
  • on lvm volumes
• your custom layout should not remove partitions and lvm volumes present in the default disk layout, nor alter their type and mountpoint (you may modify their size), nor change the partition table type (dos or gpt).

Feel free to comment on this.

[dmuhamedagic]

OK, I'll have to review this comment. Would you consider using setup-storage(8) which is part of FAI? It seems like it can be used independently from the rest of FAI. https://fai-project.org/doc/man/setup-storage.html

[dmuhamedagic]

One of the examples shipped with fai-setup-storage:

[0]capote:config > cat disk_config/LVM 
# <type> <mountpoint> <size>   <fs type> <mount options> <misc options>

# entire disk with LVM, separate /home

disk_config disk1 fstabkey:uuid align-at:1M

primary /boot   200 ext2    rw,noatime
primary -       4G- -       -

disk_config lvm

vg vg1  disk1.2
vg1-root     /       3G-50G   ext4    noatime,rw
vg1-swap     swap    200-4G   swap    sw
vg1-home     /home   600-     ext4    noatime,nosuid,nodev,rw
[0]capote:config > 

[eduble]

About fai-setup-storage, I looked at it, but I have one or two concerns.

From what I understood, FAI works quite differently from debootstick. Tell me if I am wrong, because my knowledge about it is limited. FAI builds installation media, not live systems. That means: the installation media itself has a completely custom structure, that does not match at all the target system to be installed (in terms of partitions, etc.). Whereas, with debootstick, the image already contains the target system (no further installation is needed). If you decide to migrate this OS to a larger disk, the OS remains unchanged, except that partitions can be resized.

Thus, in the case of debootstick, the disk layout configuration should be meaningful both:

• for live usage (e.g. on a USB disk with limited space)
• after the user decided to migrate to a larger disk

And there is one more important requirement: even if the disk image contains the whole live OS, with target partitioning already in place, it should have a minimal size.

Thus debootstick partitioning will be quite specific:

• for the image generation step, most of the partition or lvm volume sizing information will be discarded in favour of minimal sizing. [1]
• when booted in live mode or migrated to a larger disk, the partitions or lvm volumes are already there, but they can be resized, taking into account provided size information, this time.

I believe this 2-steps process is quite different from what fai-setup-storage can provide.

Moreover, resizing filesystems can cause major issues. When you create an ext4 filesystem on a minimal disk space it will auto-select some settings for more performance given this small size. If, later, you resize it to fit a much larger disk, previous settings do not match at all. resize2fs will try to adapt it as much as possible, by activating unusual filesystem features, some of which are not understood by grub, causing boot failure. So, in terms of filesystem options, I do not want to give much freedom to the user. All in all, using fai-setup-storage would bring much configurability, but I doubt I can test all those possible settings work properly in the specific case of debootstick. So I prefer to provide a limited set of features I have properly tested, and add more of them over time.

[1] I said "most of", because only the last partition can be extended. If you specify that 2nd partition must be 2G-large, and it is followed by a third partition, then the generated image will be quite large because it has to allocate those 2G. Of course, using LVM volumes is much more appropriate in this case.

[dmuhamedagic]

The general idea was to use the setup-storage only to create the partition table and filesystems and volumes, i.e. not to have to implement the whole parsing etc shebang. I'm all for using one tool for one task and to have that one tool do a good job. I do understand all of your concerns, but maybe the FAI developer would be willing to help with this and we could together find some way to make fai-setup-storage fit the needs/constraints/requirements of debootstick.

[dmuhamedagic]

As for minimum space, that is indeed very important. I was fiddling a bit with the idea of using a variant of sparse files, but it's not quite clear to me how one could manage that. tar(1) for instance has support for that and it would be conceivable to have a bit different procedure: not to manage the disk layout during the USB stick creation, but have that happen once the USB stick boots. That way one could even have compressed images. I would find such a feature very attractive for my use case. I'm also not sure if this could be considered a debootstick problem domain, but would like to hear your opinion.

[eduble]

As for minimum space, that is indeed very important. I was fiddling a bit with the idea of using a variant of sparse files, but it's not quite clear to me how one could manage that.

I can confirm that debootstick-generated images are sparse files. For instance, if you compare the image size ls prints versus the disk space really needed (du output):

root@vm-debian:~# ls -lh buster.dd 
-rw-r--r-- 1 root root 632M mars  27 10:40 buster.dd
root@vm-debian:~# du -sh buster.dd 
546M    buster.dd
root@vm-debian:~#

So of course, even if the disk layout spans over several GB, and makes the image quite large (at least for what it seems with ls), you can tar the image efficiently with option --sparse, and then dump it directly to the device by piping tar and dd.

not to manage the disk layout during the USB stick creation, but have that happen once the USB stick boots

That would be pretty far from the philosophy of debootstick, with the OS already installed.

[eduble]

The general idea was to use the setup-storage only to create the partition table and filesystems and volumes, i.e. not to have to implement the whole parsing etc shebang. I'm all for using one tool for one task and to have that one tool do a good job. I do understand all of your concerns, but maybe the FAI developer would be willing to help with this and we could together find some way to make fai-setup-storage fit the needs/constraints/requirements of debootstick.

OK. You know I always think twice before adding a dependency over other software. The benefit has to be obvious, because it can also cause major issues (support ended, breaking change, etc). Take the case of LVM, it caused most important debootstick issues and a big part of the maintenance work I had to do. But, still, I consider the benefit is high in this case (for handling the OS migration process).

For the parsing, I wrote this part already in disk-layouts development branch (https://github.com/drakkar-lig/debootstick/blob/disk-layouts/test.sh), and it is not the hardest part.

What I believe could be a good improvement for you (as a user of FAI and debootstick) would be to support the same file format, if ever this is possible.

Probably an important difference is the following: debootstick needs, in many cases (for image generation and when the user sets size to auto), to allocate the minimum space for a partition or volume, given the content of the file tree that will be mounted there. I suppose FAI does not know at this time the content of the file tree, so I am not sure it is possible.

Anyway, feel free to contact the author of FAI, we may be able to make things converge. I suppose that being a regular user of his software, your request could be regarded favorably. :)

In the meantime, I will continue working on the file format I designed, and adding support for another format will be quite trivial if needed.

[dmuhamedagic]

FAI will probably be around for the foreseeable time. I'm also very new to FAI, but it seems to be the thing for automatic installations. Anyway, I'll take a closer look and see if/how we could put the two together. But I'm currently swamped with other stuff, so it may take a while...

BTW, I recently read some study on software engineering and according to some statistics the implementation is about 20% and maintenance about 80% of the total cost.

[eduble]

Hi Dejan, FYI I made some progress on this topic. It still needs some more tests, so this still in branch disk-layouts; but it is starting to work well.

Now, debootstick prints this at startup:

[...]
I: using default disk layout: /usr/share/debootstick/disk-layouts/target/pc/disk-layout
[...]

The content of this file is:

# gpt or dos partition table
gpt

# ordered list of partitions (with mountpoint, type, size)
partition   /boot/efi  efi     auto
partition   none       bios    auto
partition   none       lvm     max

# lvm volumes (with mountpoint, type, size)
lvm_volume  /          ext4    max

By looking at the mountpoint, debootstick will automatically dispatch files from the initial chroot directory to the approriate LVM volume or partition. For instance, the content of [source_chroot]/boot/efi will be put in partition 1, whereas the rest will be in the root lvm volume.

You can make a copy of this file, modify it, and add option --disk-layout <new-layout-file> when you call debootstick.

For instance, you can append the following line to the default layout:

lvm_volume  /var       ext4    25%

Then content of /var will be directed to a specific volume. Moreover, when the system is booted, the size of this lvm volume will be set to 25% of the disk size (unless the disk is too small; in this case debootstick will try to adapt you request the best it can).

I tested the following:

• PC image with default layout, with live (expansion on disk) procedure
• PC image with default layout, with migration procedure
• PC image with slightly modified layout, with live (expansion on disk) procedure
• PC image with slightly modified layout, with migration procedure
• Raspberry pi image with default layout (live only, no migration is possible in this case)

I still have to test more cases, handle fixed size specification, handle volume group renaming, ... But if you want to try it, I would be happy to get your feedback.

[dmuhamedagic]

Hi Etienne,

Cool! I will get back to you in about two weeks, because I'll be away from computers in the meantime.

Thanks!

[eduble]

Hi Dejan, I completed the disk layout feature (volumes with fixed size, volume group naming and some less-related improvements), updated the man page and merged the branch into master. If you have a little time to look at it and have comments, they will be very welcome. For now I close this issue.