trapexit / mergerfs-tools

Optional tools to help manage data in a mergerfs pool
ISC License
385 stars 44 forks source link

Feature: Allow mergerfs.balance to balance against a specified amount of free space per disk #104

Open undaunt opened 4 years ago

undaunt commented 4 years ago

2 - Allow mergerfs.balance to balance disks against a target of total free space or total used space, similar to mfs/lfs policies.

Use case:

disk1 8TB, 50% full (4TB) disk2 12TB, 50% full (6TB) disk3 12TB, 75% full (9TB) disk4 12TB, 0% full (0TB)

Currently, running balance would get each disk to be 2% apart in usage, winding up in this situation:

disk1 8TB, 43% full (3.45TB) disk2 12TB, 43% full (5.18TB) disk3 12TB, 43% full (5.18TB) disk4 12TB, 43% full (5.18TB)

If possible, I'd like to be able to balance against a value of free space similar to empfs's behavior. IE: mergerfs.balance knows there are approximately 25TB free across all disks, so it will balance until there is approximately free space / number of disks, or in this case 25/4 = 6.25TB free on each disk, leaving us here:

disk1 8TB, 21.8% full (1.75TB) disk2 12TB, 48% full (5.75TB) disk3 12TB, 48% full (5.75TB) disk4 12TB, 48% full (5.75TB)

This would then allow net new data (in a mfs policy scenario) to write alternating across all four drives. Without this, data would go to disk2, disk3, and disk4 until they each had 4.55 TB of free space, similar to disk1 in the current state.

celogeek commented 4 years ago

Hi,

I replace the method "freespace_percentage" by:

def freespace_percentage(srcmounts):
    lfsp = []
    total_fs = 0
    for srcmount in srcmounts:
        vfs = os.statvfs(srcmount)
        avail = vfs.f_bavail * vfs.f_frsize
        total_fs += avail
        lfsp.append((srcmount,avail))

    lfsp = [(x[0], x[1] / total_fs) for x in lfsp]
    return sorted(lfsp, key=lambda x: x[1])

This sum the available freespace, then create a percentage by disk against it.

The effect is that it will balance using a "mfs" rules.