lxc / lxcfs

FUSE filesystem for LXC
https://linuxcontainers.org/lxcfs
Other
1.05k stars 251 forks source link

cpuview: fix ABBA deadlock in find_proc_stat_node #579

Closed mihalicyn closed 1 year ago

mihalicyn commented 1 year ago

Thanks to detailed report from Nikhil it was discovered that on some workloads reads from lxcfs getting stuck.

After analysis of kernel crashdump it was found, that many "mtail" processes waiting on read() from /proc/stat file.

First suspect was my last commit that fixes use-after-free, but unfortunately it also adds ABBA deadlock.

Thread 1                                                   Thread 2

find_proc_stat_node():
rwlock_read                                               rwlock_read
    mutex_lock(some_node) [taken]                       mutex_lock(some_node) [wait T1]
rwlock_unlock
rwlock_wrlock (prune_proc_stat_history call) [wait T2]

BOOM. That's deadlock.

Fix is simple, let's just move prune_proc_stat_history call before taking mutex on cg_proc_stat node.

Fixes: 54db3e71b ("cpuview: fix possible use-after-free in find_proc_stat_node") Issue #471

Reported-by: Nikhil Kshirsagar nikhil.kshirsagar@canonical.com Signed-off-by: Alexander Mikhalitsyn aleksandr.mikhalitsyn@canonical.com

brauner commented 1 year ago

Thanks!