Scrub configuration when subvolid=5 is not mounted

[norbusan]

I would like to scrub the whole btrfs multi-device filesystem, but I have only mounted subvolumes (also root is a subvolume). Is there a way the configure btrfsmaintainence to scrub the whole filesystem and not only the mounted subvolumes without manually mounting subvolid=5 to a dir and starting btrfs scrub dir?

[sten0]

Hi Norbert!

Cool to hear you're using btrfs :-)

Norbert Preining @.***> writes:

I would like to scrub the whole btrfs multi-device filesystem, but I have only mounted subvolumes (also root is a subvolume). Is there a way the configure btrfsmaintainence to scrub the whole filesystem and not only the mounted subvolumes without manually mounting subvolid=5 to a dir and starting btrfs scrub dir?

Scrub operates at the volume rather than subvolume level. To test this, create a subvolume, fill it with a small amount of data, then

btrfs scrub start @test_subvol btrfs scrub status @test_subvol

You'll see that "Total to scrub" is surprisingly large ;-)

The default BTRFS_SCRUB_MOUNTPOINTS="auto" in /etc/default/btrfsmaintenance will scrub all mounted volume. Eg: if four disks are in two raid1 profile volumes, and both volumes are mounted, it will scrub them both.

[sten0]

Correction: "operates at the volume level" is not truly accurate. The man page is not clear if btrfs scrub /dev/sdb, for a two disk raid1 profile volume will also scrub /dev/sda, but seems to indicate that it won't. I've never tested that functionality. IIRC scrubs do work on lower level data structures, but I can't remember which one.

[Zygo]

The man page says:

Start a scrub on all devices of the mounted filesystem identified by path or on a single device.

Do you have a suggested rewording to make that clearer?

In the kernel, scrub is a device-level operation. For convenience, the userspace btrfs scrub tool will accept a path to a mounted filesystem(*), enumerate its devices, and run a scrub thread in parallel on each device it finds, but this convenience does not make scrub become not a device-level operation. So in the raid1 example:

btrfs scrub /dev/sdb

only blocks on /dev/sdb will be checked or repaired. If you run scrub on the mountpoint:

btrfs scrub start /mnt/raid1

the result is equivalent to:

btrfs scrub start /dev/sda & btrfs scrub start /dev/sdb &

except the userspace tool will try to prevent that because the tool's progress and status reporting doesn't support what the kernel can do. The kernel doesn't care, so if you invoke the bare scrub ioctl on the individual devices, the kernel will happily scrub each device sequentially or in parallel, as long as there are not two scrubs running on the same device at the same time.

Scrub checks metadata blocks that are not part of any file (and therefore not part of any subvolume), such as filesystem superblocks and the free space tree. Scrub couldn't work by subvolume without fundamentally redefining what scrub is.

The expand_auto_mountpoint function enumerates device nodes of mounted btrfs filesystems, picks a random mount point path for each filesystem, and hands that path to btrfs scrub. Then btrfs scrub turns the mount point back into a list of devices so it can do its work. There's no filtering by subvol ID (or anything else) on the mount points.

(*) If you give scrub a device, scrub will find a mount point for the filesystem so it can run the ioctl, then resolve the device back to a btrfs devid so it can run scrub on the device. btrfs is complicated.

[norbusan]

Hi @sten0 nice, didn't know you are involved here. Yes, heavy btrfs users since many years, and started multiple device with raid like 1.5 years ago.

Let us summarize what I think I have understood or learned

• When calling scrub on a device live /dev/sdg(N), only the blocks on that device are scrubbed (that I tested)
• When calling scrub on a mount point it "Start a scrub on all devices of the mounted filesystem identified by path"

Now indeed, if I start a scrub on /home (which is a subvolume mounted), I see the full size:

UUID:             911600cb-bd76-4299-9445-666382e8ad20
Scrub started:    Tue Feb  1 09:35:01 2022
Status:           running
Duration:         0:22:02
Time left:        0:31:56
ETA:              Tue Feb  1 10:29:02 2022
Total to scrub:   7.84TiB
Bytes scrubbed:   3.20TiB  (40.82%)
Rate:             2.48GiB/s
Error summary:    no errors found

But honestly, the speed is a bit too fast for my understanding: 2.48GiB/s ... is this reasonable? Also, looking at the time it should finish in about 1h.

Yesterday I mounted subvolid=5 and run the scrub on that mount point and it took about 5 hours.

So looking at the time and speed, it seems that despite the Total to scrub indicating the total size, it works only on a part of the file system.

But all these are wild guesses ...