Problem on btrfs root on nvme / ssd drives

[GuillaumeSeren]

Hello, Thank you for this nice project.

I have been testing bees as replacement of duperemove, for the non root btrfs it works fine, for my root btrfs on ssd / nvme / ssd raid1 there is a consistent issue after a few minutes:

- Systemd: State 'stop-watchdog' timeout, killing.
- Systemd: Killing process 643 (systemd-journal) with signal SIGKILL.
- Systemd: Process still around after SIGKILL ignoring.

At this point it try to start a new journal process, which work but still can't write to the disk, so it loops again, and add journal process each time.

Here what I can see in the /proc/643/stack:

- btrfs_put_transaction
- btrfs_commit_transaction
- btrfs_cont_expand
- do_truncate
- do_sys_ftruncate
- do_syscall
- entry_SYSCALL_64_after_hwframe

Also I am using the provided service config with (nice,ionice,etc).

I try changing the scheduler to bfq but that doesn't seem to change anything and the issue come back, but my scheduler are:

- ssd mq-deadline
- nvme noop
- ssd-raid1 mq-deadline

Kernel used are 5.4.83 and 5.4.88

Configuration of bees is:

OPTIONS="--strip-paths --no-timestamps --thread-count 1 --verbose=5"

There should be a way to limit a bit more the amount of change asked to the drive at once (guessing) ?

[GuillaumeSeren]

Proably similar of #165

[kakra]

This looks a lot like the IO stalls I was seeing with btrfs on bcache due to work queue congestion. There's a proposed patch series but that won't help you as you're not using bcache: https://lore.kernel.org/linux-bcache/CAC2ZOYtrrOXD35ZWLer8R8n1dcyqGi_dipHp+y1JNt_eaue_9A@mail.gmail.com/T/#t

There should be a way to limit a bit more the amount of change asked to the drive at once (guessing) ?

You could try the loadavg limiter --loadavg-target...

Kernel 5.10 made a lot of things better for me.

If you enabled discard, maybe disable it?

[GuillaumeSeren]

Hey @kakra ,

This looks a lot like the IO stalls

Yes seems so, talking about this on irc I was suggested to try a post 5.7 kernel, some patch set for performance improvement seems to fix it.

You could try the loadavg limiter --loadavg-target...

What are the value you suggest for this param ?

So as suggested on irc I try again with a newer kernel 5.10.12, seem to run better memory stay low, and the transaction barely show in the journal stack.

I will try again on 5.4 with those patches to try to see if that fix it (b25b0b871f20, cfc0eed0ec89, ed58f2e66e84, 7ac8b88ee668).

[kakra]

You could try the loadavg limiter --loadavg-target...

What are the value you suggest for this param ?

I'm using a value of 5 on desktops, and 10 on servers. Thread factor is 0.5 but on a high-core-count CPU, you might rather stick to a static value like 4 or 6.

[GuillaumeSeren]

@kakra, I tested longer with a 5.10.12 without reproducing the issue, then I cherry-picked the patch (b25b0b871f20, cfc0eed0ec89, ed58f2e66e84, 7ac8b88ee668), on top of a 5.4.88 and the issue is gone now.

Also the patches are in the next 5.4 queue so hopefully they will be in the next release (https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/queue-5.4).

So I think backport the patch or upgrade to 5.7 fix the issue.

About the '--loadavg-target' I am testing your suggestion with value 4, it seems to lower the cpu usage a bit, so it is not mandatory but it can help for sure.

[kakra]

About the '--loadavg-target' I am testing your suggestion with value 4, it seems to lower the cpu usage a bit, so it is not mandatory but it can help for sure.

@GuillaumeSeren It helps preventing desktop stalls or stealing too much throughput from other applications that temporarily demand it. In the latter case especially, it helps preventing dominating the writeback cache of the kernel - otherwise foreground processes become easily stuck for some seconds. So I usually set the limit a little higher than I would expect from my normal usage so bees wouldn't try to overshoot the load the system does usually handle. It works very well for me.

[GuillaumeSeren]

@kakra I will test again on a unpatched 5.4 to see if that fix my issue.

[kakra]

@GuillaumeSeren I will probably only mitigate but not fix it without the patches, after all it stalls in the kernel no matter what. With the loadavg target you may just prevent it from happening too often.

[Zygo]

loadavg-target simply makes bees slow down its requests when it detects impact on the system beyond that which comes from bees itself. It's intended to detect when the system is mostly idle and wake up bees threads then, and put bees to sleep at other times. If you have a mostly idle desktop that's really useful, but if you're running a file server and relying on bees to reduce space usage, bees might not run fast enough to keep up.

If the kernel has a performance bug, loadavg-target can spread out the impact over time, but will not be able to eliminate it. bees will pause for a few seconds after a lockup but will start up again soon after. If the lockups last for minutes and bees sleeps for a few seconds between them, then this doesn't help very much. The only real fix is in the kernel.

[GuillaumeSeren]

Hey, @kakra I have tested again (sorry for the delay), on a 5.4.83 with the following paramaters:

OPTIONS="--strip-paths --no-timestamps --loadavg-target 5 --verbose=5"

Same issue.

OPTIONS="--strip-paths --no-timestamps --thread-count 1 --loadavg-target 2 --verbose=5"

Also same freeze after a few minutes, not really slower. With the 5.4.88 + the patches no issue.

@Zygo Thank you for the clarification, with the patches you suggest me it works really faster.

[GuillaumeSeren]

Hey, I've noticed the patches are now in stable 5.4.96 as you can see here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-5.4.y

So if you have the issue make sure to have at least 5.4.96 installed.

I think we can close this thank you for the help.