search

kdave / btrfs-progs

Development of userspace BTRFS tools
GNU General Public License v2.0
527 stars 239 forks source link

btrfs balance on mixed capacity drives #802

Open CHerSun opened 4 weeks ago

CHerSun commented 4 weeks ago

Hi. Sorry if this is a duplicate, but I wasn't able to find issue on this.

I'm using RAID1 on 5 drives, all of the drives were 3 TB. Recently I've had to switch a failed drive to a 4 TB one. And only then I noted, that btrfs balances based on unallocated space on the drive and not on percentage used (used/total) of the drive. Between 3 and 4 TB drives that won't cause too large skew, but googling I've noted cases where ppl use configurations like 1+1+3+3 drives, and there this causes a huge skew, forcing to basically 2 disks until ~2 TB is written. I'm considering to upgrade to ~18 TB drives one by one - then I will get this skew too.

So the question is if it's possible to switch to percentage-based balancing. If not - could that be added? Is there any doc where I could read more on this decision?

Forza-tng commented 4 weeks ago

Btrfs currently uses the disk with most unallocated space first, given the constraints of the active profile. You can determine how much is usable across all devices using the btrfs disk space calculator https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&d=4&d=3&d=3&d=3. With 3+3+3+4 TiB, 1TiB will not be usable with a RAID1 profile. With 4x3TiB + 1x4TiB all space will be usable.

You can reduce the usable size of the new disk to 3TiB by using:

btrfs filesystem resize <id>:<size> <mount-point>

You can list device id using btrfs fi us -T <mnt>.

Just remember that btrfs won't reclaim the 1TiB slack until you resize the filesystem again. You can use <id>:max for that.

CHerSun commented 4 weeks ago

You can reduce the usable size of the 4th disk to 3TiB by using:

Thank you for idea, but well, that's not why one would increase drives capacity and use btrfs, quite the opposite.

Forza-tng commented 4 weeks ago

You can reduce the usable size of the 4th disk to 3TiB by using:

Thank you for idea, but well, that's not why one would increase drives capacity and use btrfs, quite the opposite.

Well, perhaps i misunderstood. Btrfs only uses 2 mirrors for any allocation with RAID1. So more disks will not make things faster or more resilient. With RAID10, Btrfs would use stripes across devices, potentially making sequential reads/writes faster.

Some related information: https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html#profiles https://wiki.tnonline.net/w/Btrfs/Profiles

CHerSun commented 4 weeks ago

Well, perhaps i misunderstood. Btrfs only uses 2 mirrors for any allocation with RAID1. So more disks will not make things faster or more resilient. With RAID10, Btrfs would use stripes across devices, potentially making sequential reads/writes faster.

Yep, 2 mirrors for single block, but next block can use 2 mirrors on different disks, etc. So more disks = more speed, especially for larger transfers (considering 1GiB allocations) or concurrent accesses. Placing all of allocations on larger disks gives a huge skew to both possible performance and drives wear. That's why most hardware storages, that do allow mixing capacities normally do distrubution of allocations based on % - to utilize all of them more evenly.

Just to illustrate, let's say I'd still have 3x3 TiB and replace 2 drives with say 10 TiB drives. With 7 TiB of actual data in RAID1 btrfs would distribute data like this currently:

disk capacity data usage
1 3 TiB 0 TiB
2 3 TiB 0 TiB
3 3 TiB 0 TiB
4 10 TiB 7 TiB
5 10 TiB 7 TiB

putting all the workload on the 2 drives.

What % based-distribution would do:

disk capacity data usage
1 3 TiB 1.45 TiB
2 3 TiB 1.45 TiB
3 3 TiB 1.45 TiB
4 10 TiB 4.8 TiB
5 10 TiB 4.8 TiB
Forza-tng commented 4 weeks ago

It would be a good option, had it existed. For now, the easiest way to do this is to use slack, and then reduce as needed.