search

tuxera / ntfs-3g

NTFS-3G Safe Read/Write NTFS Driver
https://www.tuxera.com/company/open-source
GNU General Public License v2.0
1k stars 149 forks source link

livelock scenario #56

Open youngjun-89 opened 2 years ago

youngjun-89 commented 2 years ago

[Environment] 22.04 5.15.0-43-generic ubuntu kernel. ntfs-3g version ntfs-3g 2021.8.22 integrated FUSE 28 - Third Generation NTFS Driver

[Problem] I bumped on livelock and analyze it. and concluded that it is needed to be fixed. it happends when 3 operation concurrently progressing.

1) usb detach by user. and kernel detect it. 2) mount.ntfs umount request & device release operation 3) pool-udisksd umount operation.

[Conclusion]

  1. mounted target device file must be released after /dev/fuse release. it makes deadlocky scenario.
  2. fuse file system "fuse_simple_request" should not be waiting forever. it is hard to solve this situation by interrupting application. huntask panic configuration make user kernel panic. user don't know why.

[Analysis] I got a kernel crash dump file. and analyze it.

here is the scenario description. 1

  1. kworker detect usb is detached from computer. it is blocked by umount operation (pool-udiskd) PID: 8 TASK: ffff88810029e400 CPU: 0 COMMAND: "kworker/0:1" 0 [ffffc9000006f6f0] schedule at ffffffff81d57c0d 1 [ffffc9000006f778] schedule at ffffffff81d57fae 2 [ffffc9000006f798] rwsem_down_read_slowpath at ffffffff81d5a2de 3 [ffffc9000006f830] down_read at ffffffff81d5a373 // wait s_umount 4 [ffffc9000006f848] get_super at ffffffff813799ca 5 [ffffc9000006f878] fsync_bdev at ffffffff815c6f46 6 [ffffc9000006f8a0] delete_partition at ffffffff815e9328 7 [ffffc9000006f8c0] blk_drop_partitions at ffffffff815e9b3e 8 [ffffc9000006f8e8] del_gendisk at ffffffff815e75f1 // lock mutex 9 [ffffc9000006f930] sd_remove at ffffffff818cd325 10 [ffffc9000006f958] device_release_driver at ffffffff8184006f 11 [ffffc9000006f990] device_release_driver at ffffffff818400a9 12 [ffffc9000006f9b0] bus_remove_device at ffffffff8183f28e 13 [ffffc9000006f9d8] device_del at ffffffff818399ac 14 [ffffc9000006fa28] __scsi_remove_device at ffffffff818c2628 15 [ffffc9000006fa50] scsi_forget_host at ffffffff818c029f 16 [ffffc9000006fa70] scsi_remove_host at ffffffff818b3727 17 [ffffc9000006fa98] usb_stor_disconnect at ffffffffc0659b20 [usb_storage] 18 [ffffc9000006fac0] usb_unbind_interface at ffffffff8194ef30 19 [ffffc9000006fb18] __device_release_driver at ffffffff8184006f 20 [ffffc9000006fb50] device_release_driver at ffffffff818400a9 21 [ffffc9000006fb70] bus_remove_device at ffffffff8183f28e 22 [ffffc9000006fb98] device_del at ffffffff818399ac 23 [ffffc9000006fbe8] usb_disable_device at ffffffff8194cdce 24 [ffffc9000006fc30] usb_disconnect.cold at ffffffff81d19e09 25 [ffffc9000006fc80] hub_port_connect at ffffffff81944bb8 26 [ffffc9000006fd00] hub_port_connect_change at ffffffff819454b1 27 [ffffc9000006fd70] port_event at ffffffff81945d77 28 [ffffc9000006fe08] hub_event at ffffffff819460a7 29 [ffffc9000006fe78] process_one_work at ffffffff810d9e7b 30 [ffffc9000006fec8] worker_thread at ffffffff810da073 31 [ffffc9000006ff10] kthread at ffffffff810e1cba 32 [ffffc9000006ff50] ret_from_fork at ffffffff81004bc2

    3 [ffffc9000006f830] down_read at ffffffff81d5a373 ffffc9000006f838: [ffff888154d36800:kmalloc-2k] ffffc9000006f870 ffffc9000006f848: get_super+154

  2. it request umount. and release /dev/sdc file before release /dev/fuse. but. it is blocked by usb detach. PID: 18681 TASK: ffff88810e5b8000 CPU: 1 COMMAND: "mount.ntfs" 0 [ffffc90000ea7c50] schedule at ffffffff81d57c0d 1 [ffffc90000ea7cd8] schedule at ffffffff81d57fae 2 [ffffc90000ea7cf8] schedule_preempt_disabled at ffffffff81d5839e 3 [ffffc90000ea7d08] mutex_lock.constprop.0 at ffffffff81d59053 4 [ffffc90000ea7d80] __mutex_lock_slowpath at ffffffff81d59353 5 [ffffc90000ea7d90] mutex_lock at ffffffff81d59394 // wait open mutex 6 [ffffc90000ea7da8] blkdev_put at ffffffff815c765a 7 [ffffc90000ea7de0] blkdev_close at ffffffff815c86e7 8 [ffffc90000ea7df8] fput at ffffffff8137704f 9 [ffffc90000ea7e30] __fput at ffffffff8137724e 10 [ffffc90000ea7e40] task_work_run at ffffffff810deb6d 11 [ffffc90000ea7e68] exit_to_user_mode_loop at ffffffff81160fc0 12 [ffffc90000ea7e90] exit_to_user_mode_prepare at ffffffff8116106c 13 [ffffc90000ea7ea8] syscall_exit_to_user_mode at ffffffff81d4f327 14 [ffffc90000ea7ec0] do_syscall_64 at ffffffff81d4b1b9 15 [ffffc90000ea7ef8] exit_to_user_mode_prepare at ffffffff81161007 16 [ffffc90000ea7f10] syscall_exit_to_user_mode at ffffffff81d4f327 17 [ffffc90000ea7f28] do_syscall_64 at ffffffff81d4b1b9 RIP: 00007f3e315fb117 RSP: 00007fff16bcb628 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 000055cc1b45c710 RCX: 00007f3e315fb117 RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003 RBP: 000055cc1b45c970 R8: 000055cc1b476940 R9: 0000000000000000 R10: 0000000000000005 R11: 0000000000000246 R12: 00007f3e314e36c0 R13: 000055cc1b45c710 R14: 000055cc1b467030 R15: 00000000ffffffff ORIG_RAX: 0000000000000003 CS: 0033 SS: 002b

    3 [ffffc90000ea7d08] __mutex_lock.constprop.0 at ffffffff81d59053 ffffc90000ea7d10: [ffff8881caad1880:kmalloc-512] 00000002547b2ff8 ffffc90000ea7d20: [ffff8881caad1880:kmalloc-512] [ffff8881caad1880:kmalloc-512] ffffc90000ea7d30: [ffff88810e5b8000:task_struct] 0000000008000008 ffffc90000ea7d40: ffffc90000ea7de0 fcee7cb500545100 ffffc90000ea7d50: [ffff888110f99f40:bdev_cache] [ffff8881caad1870:kmalloc-512] ffffc90000ea7d60: [ffff8881caad1800:kmalloc-512] [ffff8881caad1870:kmalloc-512] ffffc90000ea7d70: [ffff8881bcd4ccc0:dentry] ffffc90000ea7d88 ffffc90000ea7d80: mutex_lock_slowpath+19 4 [ffffc90000ea7d80] mutex_lock_slowpath at ffffff

  3. we can know that it is releasing /dev/sdc file and not release /dev/fuse. crash> files 18681 PID: 18681 TASK: ffff88810e5b8000 CPU: 1 COMMAND: "mount.ntfs" ROOT: / CWD: / FD FILE DENTRY INODE TYPE PATH 0 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR /dev/null 1 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR /dev/null 2 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR /dev/null 4 ffff888154001f00 ffff8881020cce40 ffff888100f90f50 CHR /dev/fuse 5 ffff888102f74200 ffff8881c2c5dd80 ffff88815db73e40 SOCK UNIX 8 ffff888106412800 ffff888104785540 ffff888104a9e470 REG /proc/804/mountinfo 9 ffff888106412100 ffff888100443840 ffff8881022a9688 REG /proc/swaps 14 ffff88810a720700 ffff888104785540 ffff888104a9e470 REG /proc/804/mountinfo

  4. pool udisksd is progressing umount process (cuz uhelper=udisksd2) but it is waiting forever cuz mount.ntfs does not progress. (if it is to be released, /dev/fuse must be released. but mount.ntfs blocked by kworker)

PI3: 18741 TASK: ffff8881a702b200 CPU: 1 COMMAND: "pool-udisksd" 0 [ffffc900022a3be8] schedule at ffffffff81d57c0d 1 [ffffc900022a3c70] schedule at ffffffff81d57fae 2 [ffffc900022a3c90] request_wait_answer at ffffffff814ea190 3 [ffffc900022a3cf0] fuse_simple_request at ffffffff814ec9c1 4 [ffffc900022a3d30] fuse_send_destroy at ffffffff814f743f 5 [ffffc900022a3db8] fuse_conn_destroy at ffffffff814f7622 6 [ffffc900022a3dd0] fuse_kill_sb_blk at ffffffff814f7d82 7 [ffffc900022a3e00] deactivate_locked_super at ffffffff8137863b 8 [ffffc900022a3e20] deactivate_super at ffffffff813786f0 // lock s_umount 9 [ffffc900022a3e38] cleanup_mnt at ffffffff8139f350 10 [ffffc900022a3e78] cleanup_mnt at ffffffff8139f422 11 [ffffc900022a3e88] task_work_run at ffffffff810deb6d 12 [ffffc900022a3eb0] exit_to_user_mode_loop at ffffffff81160fc0 13 [ffffc900022a3ed8] exit_to_user_mode_prepare at ffffffff8116106c 14 [ffffc900022a3ef0] syscall_exit_to_user_mode at ffffffff81d4f327 15 [ffffc900022a3f08] do_syscall_64 at ffffffff81d4b1b9 16 [ffffc900022a3f50] entry_SYSCALL_64_after_hwframe at ffffffff81e0007c RIP: 00007f110d17bbeb RSP: 00007f110beb63b8 RFLAGS: 00000246 RAX: 0000000000000000 RBX: 00007f10fc0257e0 RCX: 00007f110d17bbeb RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f10fc012bb0 RBP: 00007f10fc09f480 R8: 0000000000000000 R9: 00007f10fc05edf0 R10: 00007f10fc05ee30 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f10fc012bb0 R14: 00007f10fc09f590 R15: 0000000000000000 ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b