search

naota / linux

Linux kernel source tree
Other
3 stars 1 forks source link

100% cpu on `btrfs-cleaner` and `kswapd0` with linux 6.10.x #67

Closed brauliobo closed 2 months ago

brauliobo commented 3 months ago

Hi,

After upgrading to 6.10 (currently using 6.10.6), I'm having an eternal cpu loop of btrfs-cleaner and kswapd0

Reverting back to 6.9.x fixes the issue.

I'm using snapper in all BTRFS volumes. I've tried disabling quotas in all volumes but still the CPU loop hasn't resolved.

image

brauliobo commented 3 months ago

tried rebalancing without any effect

braulio @ bhavapower ➜  scrapers git:(main) ✗  sudo systemctl status btrfs-balance 
○ btrfs-balance.service - Balance block groups on a btrfs filesystem
     Loaded: loaded (/usr/lib/systemd/system/btrfs-balance.service; static)
     Active: inactive (dead) since Wed 2024-08-21 08:57:24 -03; 5min ago
   Duration: 10min 23.471s
 Invocation: fb4dfba05dbd437dbf380357ed2db4dc
TriggeredBy: ● btrfs-balance.timer
       Docs: man:btrfs-balance
    Process: 348924 ExecStart=/usr/share/btrfsmaintenance/btrfs-balance.sh (code=exited, status=0/SUCCESS)
   Main PID: 348924 (code=exited, status=0/SUCCESS)
   Mem peak: 1.5G
        CPU: 39.321s

ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: flock: executando btrfs
ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: After balance of /mnt/extensor2
ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: Data, single: total=2.73TiB, used=2.73TiB
ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: System, DUP: total=32.00MiB, used=320.00KiB
ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: Metadata, DUP: total=10.00GiB, used=9.51GiB
ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: GlobalReserve, single: total=512.00MiB, used=0.00B
ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: Sist. Arq.      Tam. Usado Disp. Uso% Montado em
ago 21 08:57:24 bhavapower btrfs-balance.sh[348955]: /dev/sde1       5,1T  3,1T  2,0T  61% /mnt/extensor2
ago 21 08:57:24 bhavapower systemd[1]: btrfs-balance.service: Deactivated successfully.
ago 21 08:57:24 bhavapower systemd[1]: btrfs-balance.service: Consumed 39.321s CPU time, 1.5G memory peak.
brauliobo commented 3 months ago

after many hours on this loop, it finished by itself

brauliobo commented 3 months ago

after a backup rsync during the dawn, the issue is back :cry:

brauliobo commented 3 months ago

might be related to the following hangs in dmesg:

[131484.126660] INFO: task btrfs-transacti:378128 blocked for more than 122 seconds.
[131484.126667]       Tainted: G           OE      6.10.6-arch1-1 #1
[131484.126669] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[131484.126670] task:btrfs-transacti state:D stack:0     pid:378128 tgid:378128 ppid:2      flags:0x00004000
[131484.126673] Call Trace:
[131484.126674]  <TASK>
[131484.126677]  __schedule+0x3d5/0x1520
[131484.126684]  schedule+0x27/0xf0
[131484.126687]  wait_for_commit+0x11f/0x1d0 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[131484.126730]  ? __pfx_autoremove_wake_function+0x10/0x10
[131484.126734]  btrfs_commit_transaction+0x805/0xc80 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[131484.126764]  ? start_transaction+0x22c/0x830 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[131484.126789]  transaction_kthread+0x159/0x1c0 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[131484.126815]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[131484.126839]  kthread+0xcf/0x100
[131484.126842]  ? __pfx_kthread+0x10/0x10
[131484.126844]  ret_from_fork+0x31/0x50
[131484.126847]  ? __pfx_kthread+0x10/0x10
[131484.126848]  ret_from_fork_asm+0x1a/0x30
[131484.126851]  </TASK>
[132025.518365] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[132025.855468] ata5.00: configured for UDMA/133
[132624.747691] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[132624.752441] ata5.00: configured for UDMA/133
[133224.463639] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[133224.471246] ata5.00: configured for UDMA/133
[133831.902896] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[133831.910486] ata5.00: configured for UDMA/133
[134187.525181] INFO: task btrfs-transacti:378128 blocked for more than 122 seconds.
[134187.525187]       Tainted: G           OE      6.10.6-arch1-1 #1
[134187.525189] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[134187.525190] task:btrfs-transacti state:D stack:0     pid:378128 tgid:378128 ppid:2      flags:0x00004000
[134187.525194] Call Trace:
[134187.525195]  <TASK>
[134187.525198]  __schedule+0x3d5/0x1520
[134187.525205]  schedule+0x27/0xf0
[134187.525208]  wait_for_commit+0x11f/0x1d0 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[134187.525251]  ? __pfx_autoremove_wake_function+0x10/0x10
[134187.525255]  btrfs_commit_transaction+0x805/0xc80 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[134187.525285]  ? start_transaction+0x22c/0x830 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[134187.525310]  transaction_kthread+0x159/0x1c0 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[134187.525336]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs fa52ccff77e5274b4cf47bada1d91262a3cd2775]
[134187.525360]  kthread+0xcf/0x100
[134187.525363]  ? __pfx_kthread+0x10/0x10
[134187.525364]  ret_from_fork+0x31/0x50
[134187.525367]  ? __pfx_kthread+0x10/0x10
[134187.525368]  ret_from_fork_asm+0x1a/0x30
[134187.525371]  </TASK>
[134187.525372] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
brauliobo commented 2 months ago

it seems fixed on linux 6.10.7