kdave / btrfs-progs

Development of userspace BTRFS tools
GNU General Public License v2.0
527 stars 239 forks source link

btrfs prop set causing system to reboot #699

Closed axet closed 7 months ago

axet commented 8 months ago

Hello!

sudo find "/dev/snd" -exec btrfs prop set {} compression zstd \;

causing system to stuck with /dev/snd/pcmC1D0p file and then insta reboot after 5 minutes.

Also prop set stuck with following (~/.local/share/parcellite/fifo_c):

prw-r----- 1 axet axet   0 окт  7 13:42 fifo_c

6.1.0-13-amd64 btrfs-progs 6.2-1 debian 12

kdave commented 8 months ago

So this is trying to set compression on a device node. I haven't looked but yeah this could cause problems.

Do you have any stack traces or last messages before reboot? Do you have enabled options like panic_on_warn, it could be something (relatively) harmless but when the system is configured to reboot on such problems it's going to make it worse.

kdave commented 8 months ago

Running prop on fifo files gets stuck, but is interruptible. I haven't investigated where exactly it's stuck, the properties run on top of xattrs, so that could in general work (like for ACL or security labels). We could skip anything that's not a file or directory for properties.

Regarding the crashes I can't reproduce that, running prop get in /dev/ skips basically all files with message ERROR: object is not a btrfs object.

axet commented 8 months ago

How could be here a stack trace? I guess this call causing memory leaks or watchdog rebooting the kernel.

ls -al /dev/snd/pcmC1D0p
crw-rw----+ 1 root audio 116, 5 окт 31 17:50 /dev/snd/pcmC1D0p
kdave commented 8 months ago

Ok, so it's watchdog, the process is stuck trying to read from snd device. For the fifo file the open() is blocked until the other end is opened (per manual page), this can be fixed by O_NONBLOCK. The same applies for the sound device, it fails right away with ERROR: failed to open pcmC0D0p: Device or resource busy.

kdave commented 8 months ago

The CI fails for weird reasons with the fix, loop devices seem to be half initialized and mount fails saying multi-device filesystem devices cannot be found. https://github.com/kdave/btrfs-progs/actions/runs/6712572409 and other failed build tests with fix attempts (sleep, sync, udevadm settle). Reverting the commit fixes the problem. I'll need to find alternate solution, so this fix won't make it to 6.6.

axet commented 8 months ago

Alternative? Let me guess:

btrfs prop set -r / compress=lzo
kdave commented 8 months ago

I mean a different way to fix it but it seems that the CI failures were not related to this patch so I'll add it back to 6.6 queue.

kdave commented 8 months ago

Fix in devel.

axet commented 8 months ago

With patch:

# find /dev/snd/ -exec btrfs prop set {} compress zstd \;
ERROR: object is not a btrfs object: /dev/snd/
ERROR: object is not a btrfs object: /dev/snd/controlC1
ERROR: object is not a btrfs object: /dev/snd/hwC1D0
ERROR: object is not a btrfs object: /dev/snd/pcmC1D0c
ERROR: failed to open /dev/snd/pcmC1D0p: Device or resource busy
ERROR: object is not a btrfs object: /dev/snd/pcmC1D0p
ERROR: object is not a btrfs object: /dev/snd/by-path
ERROR: object is not a btrfs object: /dev/snd/by-path/pci-0000:03:00.6
ERROR: object is not a btrfs object: /dev/snd/by-path/pci-0000:03:00.1
ERROR: object is not a btrfs object: /dev/snd/controlC0
ERROR: object is not a btrfs object: /dev/snd/hwC0D0
ERROR: object is not a btrfs object: /dev/snd/pcmC0D3p
ERROR: object is not a btrfs object: /dev/snd/seq
ERROR: object is not a btrfs object: /dev/snd/timer
root@axet-laptop:/home/axet# 

following command reboot the machine:

# find /dev/watchdog* -exec btrfs prop set {} compress zstd \;

machine still got rebooted after ~ 1 mins

kdave commented 8 months ago

Ok, so that was only a partial fix for the hang on file open. The properties only set extended attributes, it is possible that this is interpreted by watchdog as reboot trigger or blocks the keep-alive pings, I don't know and more investigation is needed.

As a workaround I'll add exception for character files for properties.

kdave commented 7 months ago

The workaround is now in devel. I can't reproduce the watchdog so I'd appreciate if you could test it, thanks.

axet commented 7 months ago

Working fine! Thanks!

kdave commented 7 months ago

Thank you, 6.6.2 on the way.