Prevent concurrent balance+defrag+scrub+trim (boo#1088010)

[okurz]

When the services are triggered at the same time the access to the physical devices is the limiting factor which can cause a severe performance impact on the system.

The statements of "After=" do not prevent this when the services are triggered individually, for example by the corresponding timer units, hence they would start in parallel. Specifying a more fine-grained timer schedule with selected operation windows will still trigger the services at the same time when they could not be triggered earlier, for example when the system was suspended so we need to handle the exclusive access within the script calls.

Related bug: https://bugzilla.suse.com/show_bug.cgi?id=1088010

[jeffmahoney]

We may be approaching this in the wrong way. No matter what magic we do in systemd service files, the user can still create what amounts to a hung file system manually by running balance and trim simultaneously.

We should handle this in the kernel. Btrfs already has the BTRFS_FS_EXCL_OP bit that creates a class of mutually exclusive operations. The list is currently: balance, device add/remove, and resize. I'd argue that scrub and trim should be added to this list.

[okurz]

@jeffmahoney I am very glad to receive a reply. If you already have the means within the kernel that sounds like an even better solution if this can be managed within a reasonable time :)

[Zygo]

I'd argue that scrub and trim should be added to this list.

There are multiple mechanisms like this in the kernel:

• EXCL_OP style: balance vs delete/resize/add/replace, also send vs dedupe. Second concurrent caller gets an error and returns immediately while the first caller is running.
• mutex style: balance vs snapshot delete. Second caller blocks until first caller releases the mutex. For long-running operations, callers release mutex from time to time (e.g. between balance block groups or deleted subvols) to allow all callers to make forward progress.
• explicit priority style: scrub vs transaction commit. Functions are ordered by priority. Inferior caller is paused whenever the superior function runs, then resumed when the superior function is done. Priority is fixed regardless of calling order, i.e. commit pauses scrub, but scrub never pauses commit.
• free-for-all style: scrub vs balance. Nobody blocks, both callers fight for the disks iop by iop.

Moving something from free-for-all style to EXCL_OP style will cause user-visible breakage: it used to be possible to start a scrub and a balance and then walk away, knowing that both will be completed eventually, but if scrub and balance are exclusive with EXCL_OP style, one or the other of these operations will fail. Userspace will have to add a retry loop to get the old behavior back.

There doesn't seem to be an obvious priority ordering between balance and scrub, and until that changes, explicit priority style is out. There are separate use cases that prefer each over the other. Both are run as scheduled and triggered actions launched from scripts, but both are also used as ad-hoc recovery tools by sysadmins. An ad-hoc balance request should override a scheduled scrub, but an ad-hoc scrub should override an automatically triggered balance.

That leaves mutex style, which seems OK. It doesn't break the "start both and forget" expectation, and alternating between the two callers after every block group does make both callers run considerably faster than letting them fight for disks iop by iop.

There might be a priority ordering between balance/scrub and trim (fstrim-style, not discard-style). Trim is normally a relatively fast operation compared to scrub or balance, so making trim pause the other operations automatically is reasonable.

[jeffmahoney]

As further feedback on the initial proposal, we should use the FSID in the file name or we create mutual exclusion across multiple file systems.

[kdave]

Jeff implemented the file lock exclusion inside the maint scripts, which works for both systemd timers and cron, so I'm going to apply the patch. The kernel-level exclusion is a different questino and as noted there are implications.

[okurz]

hi, do you still need this PR open? I understood that https://github.com/kdave/btrfsmaintenance/commit/93b00548b212604a491d47531f37534d7d10f0a2 should make this PR obsolete, isn't it?

[jeffmahoney]

You're correct. With that commit, this PR is obsolete.