Calamares auto-partitioning leaves unallocated space at the beginning and end of the disk

kq01526 commented 3 years ago

Hi,

Calamares auto-partitioning leaves unallocated space at the beginning and end of the disk.

Steps to reproduce: Boot in EFI mode, then choose "Erase Disk -> No Swap -> Encrypt System" and install. After installation, check with GParted or GNOME Disks.

Expected behavior: The two partitions (boot and root) which are created, should use the entire disk space.

Actual behavior: There is ~2MB of unallocated space at the beginning of the disk and ~2MB of unallocated space at the end of the disk.

Things tried:

GitHub issue tracker has been checked for related existing issues, but non have been found.
A quick search on the Internet came up with https://www.reddit.com/r/linuxquestions/comments/89tkce/finding_the_exact_reason_for_unallocated_space_at/ which seems to be discussing the very same issue with Calamares, but without any conclusion.

Is this behavior intentional?

Regards

abucodonosor commented 3 years ago

Expected behavior:
The two partitions (boot and root) which are created, should use the entire disk space.

So ? :) Who told you that?

rago666 commented 3 years ago

Can Confirm on manjaro

kq01526 commented 3 years ago

@abucodonosor asked:

So ? :) Who told you that?

Common sense did. Because:

No other installer is showing this behavior.
It might actually be a security issue with encryption, since wear leveling of SSDs might cause data to be written into that unallocated (and unencrypted) space?

Regards

abucodonosor commented 3 years ago

Actually, it is in the GPT & UEFI ( BIOS too if you wonder about the 1MiB ) spec and explained in the code itself.


   // Partition sizes are expressed in MiB, should be multiples of
    // the logical sector size (usually 512B). EFI starts with 2MiB
    // empty and a EFI boot partition, while BIOS starts at
    // the 1MiB boundary (usually sector 2048).
    int empty_space_sizeB = isEfi ? 2_MiB : 1_MiB;

    // Since sectors count from 0, if the space is 2048 sectors in size,
    // the first free sector has number 2048 (and there are 2048 sectors
    // before that one, numbered 0..2047).
    qint64 firstFreeSector = CalamaresUtils::bytesToSectors( empty_space_sizeB, dev->logicalSize() );

GPT on BIOS needs actually 8MiB unallocated space ( and the bios_grub flag to boot ) but calamares does not support this in auto mode yet.

kq01526 commented 3 years ago

@abucodonosor wrote:

and explained in the code itself.

The code snippet above:

Would not explain why this behavior is not observed in any other installer.
Would only explain the ~2MB at the start of the disk, not the ~2MB at the end of the disk.

As reported:

Actual behavior: There is ~2MB of unallocated space at the beginning of the disk and ~2MB of unallocated space at the end of the disk.

abucodonosor commented 3 years ago

and ~2MB of unallocated space at the end of the disk.

Let's make this short, the GPT backup is stored at end of the disk so needs free sectors at the end.

Would not explain why this behavior is not observed in any other installer.

Irrelevant, ask the authors of these installers maybe.

kkofler commented 3 years ago

It might actually be a security issue with encryption, since wear leveling of SSDs might cause data to be written into that unallocated (and unencrypted) space?

I don't see how that would be the case. The wear-leveling happens at the firmware level. The drive firmware only sees what the kernel sends it, and if your partitions on the SSD are encrypted, that will be only encrypted data. So there is no way it can write unencrypted data anywhere, neither during wear-leveling nor at any other time.

stikonas commented 3 years ago

I think 1 MiB at the beginning and the end should be enough even for GPT with BIOS.

That said, having 2 MiB boundaries is definitely harmless and in some cases might be slightly more convenient (e.g. you install into ESP but you also want to manually create BIOS boot partition)....

So int empty_space_sizeB = isEfi ? 2_MiB : 1_MiB; looks a bit strange to me but I wouldn't call this a bug.

I think other installers just leave 1 MiB at the beginning and end but again, that doesn't matter... We are not partitioning floppies.

abucodonosor commented 3 years ago

@stikonas

The reason why it is 2MiB on EFI is bc you can end up with >1MiB, to be exact 1,03MiB (+), so 1MiB is then too little, and then depending on the used tool and/or OS you start having problems.

kq01526 commented 3 years ago

@stikonas commented:

I think other installers just leave 1 MiB at the beginning and end but again

The Fedora installer (Anaconda) and the Ubuntu installer (Ubiquity) do not seem to leave any unallocated space at the beginning nor at the end of the disk.

@abucodonosor commented:

so 1MiB is then too little, and then depending on the used tool and/or OS you start having problems.

Fedora and Ubuntu do not seem to have any problems with 0 MB.

stikonas commented 3 years ago

@stikonas commented: I think other installers just leave 1 MiB at the beginning and end but again

The Fedora installer (Anaconda) and the Ubuntu installer (Ubiquity) do not seem to leave any unallocated space at the beginning nor at the end of the disk.

That's not true. Please check your partition table before claiming this. E.g. please paste output of

sudo sfdisk --json /dev/sda | jq .partitiontable.partitions[0].start

on any Fedora or Ubuntu system where you claim no unallocated space. I'm sure it will show 2048 (which means partition starts at 512 bytes (sectore size) * 2048 = 1 MiB.

@abucodonosor commented: so 1MiB is then too little, and then depending on the used tool and/or OS you start having problems.

Fedora and Ubuntu do not seem to have any problems with 0 MB.

Again, not true. With 0 MB you wouldn't be able to even grow GPT table to enable more partitions since GPT partition table has no fixed size.

kq01526 commented 3 years ago

@stikonas commented: I'm sure it will show 2048

You're right, it does. And for a distro installed through Calamares it shows 4096.

That being said, please see the steps to reproduce in the first post:

After installation, check with GParted or GNOME Disks.

To elaborate: With a distro installed with Calamares, the GNOME Disks (and GParted) apps will actually show the unallocated space at the beginning and at the end of the disk. In GNOME Disks you can actually see it visually in the bar chart showing the volumes, where the unallocated (free) space actually shows up as boxes.

On Fedora or Ubuntu, this is not visible. Checking GNOME Disks (or GParted) on Fedora or Ubuntu will not show any unallocated (free) space.

Maybe it doesn't show up there because it considers 1 MiB (512 bytes sector size x 2048) as negligible whereas the 2MiB (512 bytes sector size x 4096) that are left unallocated by Calamares is large enough for it to recognize it as worthy enough to show up?

Regards

stikonas commented 3 years ago

GUI partition utilities don't show unallocated spaces that are unusable if partition alignment is enabled. (If you disable it, it should show up. But you can significantly decrease performance/lifetime of you drive if you don't align partitions).

In any case it's just 1 MiB. Now drives have terabyte sizes, so one megabyte doesn't really matter. In any case, you can either do manual partitioning or just move partition after installation.

shmerl commented 3 years ago

I think the installer should allow more granular alignment configuration, i.e. amount of sectors like KDE Partition Manager does.

Case in point, some Samsung NVMe SSDs (like Evo) have erase block sizes that work better with 3 MB alignment instead of 1 MB. So partitions should be aligned to 6144 sectors instead of 2048 for good performance. Being able to set that would be a good addition.

I can open a separate bug report if that would be useful.

stikonas commented 3 years ago

I think the installer should allow more granular alignment configuration, i.e. amount of sectors like KDE Partition Manager does.

Case in point, some Samsung NVMe SSDs (like Evo) have erase block sizes that work better with 3 MB alignment instead of 1 MB. So partitions should be aligned to 6144 sectors instead of 2048 for good performance. Being able to set that would be a good addition.

I can open a separate bug report if that would be useful.

I'm not sure if erase block size actually has anything to do with alignment. As far as I understand erases (unlike reads or writes) should happen when drive is otherwise idle.

In any case that's a separate issue from the one that was originally reported.

shmerl commented 3 years ago

Aligning to erase block size makes writing operation faster. I tested Smasung SSD with fio to confirm that.

adriaandegroot commented 2 years ago

Closing as :"wontfix". Calamares chooses specifically to allow a little more space.

es20490446e commented 2 years ago

@adriaandegroot On my SanDisk SSD, if I do:

Move the 300MiB EFI partition to the right.
Move the Root partition to the right.
Extend the Root partition to fill the empty space.

I can perceive smoother performance. If I don't do that, on my low end laptop, I can perceive delay on the Plasma UI when I click on buttons.

Probably you can't tell if you have better computers. That's why I like developing on crappy equipment, to more clearly notice these kinds of things.

Likely the current partition scheme is assuming too much about how the storage shall be aligned, and it's just safer to assume that the device has been designed to perform better with the simplest partition scheme.

That is with its first partition starting as soon as possible, and partitions ending on a 1/10/100MiB boundary.

Also I have never experienced any drawback on a device with no empty space, neither on BIOS nor UEFI.

Hence I suspect this report shall have a deeper look.

es20490446e commented 2 years ago

If those comments prove true, the code would benefit to have links to the references.

es20490446e commented 2 years ago

Optimal IO size is reported at: /sys/block/$DEV/queue/optimal_io_size

As said on: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/iolimpartitionfstools

The heuristics parted uses are as follows:

Always use the reported alignment_offset as the offset for the start of the first primary partition.
If optimal_io_size is defined (i.e. not 0), align all partitions on an optimal_io_size boundary.
If optimal_io_size is undefined (i.e. 0), alignment_offset is 0, and minimum_io_size is a power of 2, use a 1MB default alignment.

This is the catch-all for "legacy" devices which don't appear to provide I/O hints. As such, by default all partitions will be aligned on a 1MB boundary.

Note Red Hat Enterprise Linux 7 cannot distinguish between devices that don't provide I/O hints and those that do so with alignment_offset=0 and optimal_io_size=0. Such a device might be a single SAS 4K device; as such, at worst 1MB of space is lost at the start of the disk.

es20490446e commented 2 years ago

For example my SSD has alignment_offset=0 (the optimal offset for the start of the first primary partition). But calamares don't honor it.

stikonas commented 2 years ago

This confuses me. You can't have the first primary partition on 0-th sector. That's where partition table is.

On 6 October 2022 16:15:56 BST, Alberto Salvia Novella @.***> wrote:

For example my SSD has alignment_offset=0 (the optimal offset for the start of the first primary partition). But calamares don't honor it.

-- Reply to this email directly or view it on GitHub: https://github.com/calamares/calamares/issues/1605#issuecomment-1270246420 You are receiving this because you were mentioned.

Message ID: @.***> -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

es20490446e commented 2 years ago

@stikonas Because offset isn't a sector, it is an offset.

Offset 0 shall be the increment from the first usable sector for partitioning, aka sector 1.

1+0=1

Hope it helps.

calamares / calamares

Calamares auto-partitioning leaves unallocated space at the beginning and end of the disk #1605