search

QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
535 stars 47 forks source link

Linux 6.5.5 deadlock on lvrename? #8575

Closed marmarek closed 10 months ago

marmarek commented 1 year ago

How to file a helpful issue

Qubes OS release

R4.2

Brief summary

A test hanged on VM shutdown, specifically lvrename step: https://openqa.qubes-os.org/tests/83530

I'm creating the issue to write down collected logs, but I'll investigate only when it will happen for the second time.

Steps to reproduce

Run integration tests and hope to trigger the issue. I'm not sure if it's hardware dependent, or just hard to reproduce

Expected behavior

No hang

Actual behavior

Shutting down a VM hangs.

Process list:

        root        7835  0.0  0.7 251848 31104 pts/8    D<L+ 21:30   0:00 lvm lvrename -- qubes_dom0/vm-test-inst-vm-private qubes_dom0/vm-test-inst-vm-private-1696714259-back
        root        7839  0.0  0.2 240888  9600 pts/8    S+   21:30   0:00 lvm lvremove --force -- qubes_dom0/vm-test-inst-vm-volatile
        root        7841  0.0  0.2 240888  9472 pts/8    S+   21:30   0:00 lvm lvremove --force -- qubes_dom0/vm-test-inst-vm-root-snap
        root        7846  0.0  0.2 240916  9592 pts/8    S+   21:31   0:00 lvm lvs --noheadings -o vg_name,pool_lv,name,lv_size,data_percent,lv_attr,origin,lv_metadata_size,metadata_percent --units b --reportformat=json
Task backtraces (via sysrq) ``` [ 3485.834038] task:lvm state:D stack:0 pid:7835 ppid:5665 flags:0x00004006 [ 3485.834043] Call Trace: [ 3485.834046] [ 3485.834048] __schedule+0x23d/0x670 [ 3485.834052] schedule+0x5e/0xd0 [ 3485.834057] io_schedule+0x46/0x70 [ 3485.834061] dm_wait_for_bios_completion+0xfc/0x110 [ 3485.834067] ? __pfx_autoremove_wake_function+0x10/0x10 [ 3485.834072] __dm_suspend+0x7e/0x1b0 [ 3485.834077] dm_internal_suspend_noflush+0x5c/0x80 [ 3485.834083] pool_presuspend+0xcc/0x130 [dm_thin_pool] [ 3485.834092] dm_table_presuspend_targets+0x3f/0x60 [ 3485.834098] __dm_suspend+0x41/0x1b0 [ 3485.834103] dm_suspend+0xc0/0xe0 [ 3485.834107] dev_suspend+0xa5/0xd0 [ 3485.834111] ctl_ioctl+0x26e/0x350 [ 3485.834115] ? __pfx_dev_suspend+0x10/0x10 [ 3485.834120] dm_ctl_ioctl+0xe/0x20 [ 3485.834124] __x64_sys_ioctl+0x94/0xd0 [ 3485.834129] do_syscall_64+0x5c/0x90 [ 3485.834133] ? do_syscall_64+0x6b/0x90 [ 3485.834137] ? do_syscall_64+0x6b/0x90 [ 3485.834141] ? xen_pv_evtchn_do_upcall+0x54/0xb0 [ 3485.834146] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 3485.834151] RIP: 0033:0x7f1cb77cfe0f [ 3485.834154] RSP: 002b:00007fff870f2560 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 3485.834159] RAX: ffffffffffffffda RBX: 00005b8d13c16580 RCX: 00007f1cb77cfe0f [ 3485.834164] RDX: 00005b8d144a2180 RSI: 00000000c138fd06 RDI: 0000000000000003 [ 3485.834169] RBP: 00005b8d144a2180 R08: 00005b8d132b1190 R09: 00007fff870f2420 [ 3485.834174] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000c [ 3485.834179] R13: 00005b8d132aacf0 R14: 00005b8d1324414d R15: 00005b8d144a21b0 [ 3485.834185] [ 3485.834187] task:lvm state:S stack:0 pid:7839 ppid:5665 flags:0x00000002 [ 3485.834193] Call Trace: [ 3485.834196] [ 3485.834198] __schedule+0x23d/0x670 [ 3485.834202] schedule+0x5e/0xd0 [ 3485.834207] locks_lock_inode_wait+0xb7/0x1d0 [ 3485.834214] ? __pfx_autoremove_wake_function+0x10/0x10 [ 3485.834218] __do_sys_flock+0x1a4/0x1f0 [ 3485.834224] do_syscall_64+0x5c/0x90 [ 3485.834231] ? syscall_exit_to_user_mode+0x2b/0x40 [ 3485.834235] ? do_syscall_64+0x6b/0x90 [ 3485.834239] ? do_syscall_64+0x6b/0x90 [ 3485.834243] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 3485.834248] RIP: 0033:0x7b558416087b [ 3485.834251] RSP: 002b:00007ffdb5220108 EFLAGS: 00000202 ORIG_RAX: 0000000000000049 [ 3485.834256] RAX: ffffffffffffffda RBX: 00007ffdb5220380 RCX: 00007b558416087b [ 3485.834261] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000006 [ 3485.834266] RBP: 00007ffdb5220320 R08: 00005ee1a9d908c0 R09: 0000000000000042 [ 3485.834271] R10: 0000000000000008 R11: 0000000000000202 R12: 00005ee1ab110020 [ 3485.834276] R13: 0000000000000002 R14: 00007ffdb5220380 R15: 0000000000000000 [ 3485.834282] [ 3485.834285] task:lvm state:S stack:0 pid:7841 ppid:5665 flags:0x00000002 [ 3485.834291] Call Trace: [ 3485.834293] [ 3485.834296] __schedule+0x23d/0x670 [ 3485.834300] schedule+0x5e/0xd0 [ 3485.834304] locks_lock_inode_wait+0xb7/0x1d0 [ 3485.834309] ? __pfx_autoremove_wake_function+0x10/0x10 [ 3485.834314] __do_sys_flock+0x1a4/0x1f0 [ 3485.834319] do_syscall_64+0x5c/0x90 [ 3485.834324] ? exc_page_fault+0x7c/0x170 [ 3485.834327] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 3485.834331] RIP: 0033:0x782067da787b [ 3485.834334] RSP: 002b:00007fff11cdab08 EFLAGS: 00000202 ORIG_RAX: 0000000000000049 [ 3485.834340] RAX: ffffffffffffffda RBX: 00007fff11cdad80 RCX: 0000782067da787b [ 3485.834345] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000005 [ 3485.834350] RBP: 00007fff11cdad20 R08: 0000557e4813c8c0 R09: 0000000000000042 [ 3485.834355] R10: 0000000000000008 R11: 0000000000000202 R12: 00007fff11cdace4 [ 3485.834360] R13: 0000000000000002 R14: 00007fff11cdaca0 R15: 0000000000000000 [ 3485.834365] [ 3485.834368] task:lvm state:S stack:0 pid:7846 ppid:5665 flags:0x00000002 [ 3485.834374] Call Trace: [ 3485.834376] [ 3485.834378] __schedule+0x23d/0x670 [ 3485.834383] schedule+0x5e/0xd0 [ 3485.834387] locks_lock_inode_wait+0xb7/0x1d0 [ 3485.834392] ? __pfx_autoremove_wake_function+0x10/0x10 [ 3485.834397] __do_sys_flock+0x1a4/0x1f0 [ 3485.834402] do_syscall_64+0x5c/0x90 [ 3485.834406] ? syscall_exit_work+0x103/0x130 [ 3485.834410] ? syscall_exit_to_user_mode+0x2b/0x40 [ 3485.834414] ? do_syscall_64+0x6b/0x90 [ 3485.834418] ? __x64_sys_rt_sigaction+0xbd/0x110 [ 3485.834423] ? syscall_exit_work+0x103/0x130 [ 3485.834427] ? syscall_exit_to_user_mode+0x2b/0x40 [ 3485.834431] ? do_syscall_64+0x6b/0x90 [ 3485.834435] ? do_syscall_64+0x6b/0x90 [ 3485.834439] ? do_syscall_64+0x6b/0x90 [ 3485.834444] ? exc_page_fault+0x7c/0x170 [ 3485.834447] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 3485.834451] RIP: 0033:0x7e46fda1987b [ 3485.834454] RSP: 002b:00007ffdc0e266f8 EFLAGS: 00000206 ORIG_RAX: 0000000000000049 [ 3485.834460] RAX: ffffffffffffffda RBX: 00007ffdc0e26970 RCX: 00007e46fda1987b [ 3485.834465] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000006 [ 3485.834470] RBP: 00007ffdc0e26910 R08: 00005ee4e9a708c0 R09: 0000000000000042 [ 3485.834475] R10: 0000000000000008 R11: 0000000000000206 R12: 00007ffdc0e268d4 [ 3485.834480] R13: 0000000000000002 R14: 00007ffdc0e26890 R15: 0000000000000000 [ 3485.834485] ... [ 3485.846084] Showing busy workqueues and worker pools: [ 3485.846090] workqueue writeback: flags=0x4a [ 3485.846094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3485.846096] in-flight: 8033:wb_workfn [ 3485.846121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=0s workers=5 idle: 8178 7976 12 1959 ```
Blocked tasks ``` [ 4246.558313] sysrq: Show Blocked State [ 4246.558388] task:journal-offline state:D stack:0 pid:8098 ppid:1 flags:0x00000002 [ 4246.558407] Call Trace: [ 4246.558414] [ 4246.558422] __schedule+0x23d/0x670 [ 4246.558440] schedule+0x5e/0xd0 [ 4246.558450] io_schedule+0x46/0x70 [ 4246.558461] folio_wait_bit_common+0x13d/0x350 [ 4246.558475] ? __pfx_wake_page_function+0x10/0x10 [ 4246.558488] folio_wait_writeback+0x2c/0x90 [ 4246.558498] mpage_prepare_extent_to_map+0x15c/0x4d0 [ 4246.558512] ext4_do_writepages+0x25f/0x770 [ 4246.558523] ext4_writepages+0xad/0x180 [ 4246.558533] do_writepages+0xcf/0x1e0 [ 4246.558543] ? __seccomp_filter+0x32a/0x4f0 [ 4246.558554] filemap_fdatawrite_wbc+0x63/0x90 [ 4246.558567] __filemap_fdatawrite_range+0x5c/0x80 [ 4246.558578] file_write_and_wait_range+0x4a/0xb0 [ 4246.558588] ext4_sync_file+0x88/0x380 [ 4246.558598] __x64_sys_fsync+0x3b/0x70 [ 4246.558609] do_syscall_64+0x5c/0x90 [ 4246.558621] ? exit_to_user_mode_prepare+0xb2/0xd0 [ 4246.558632] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 4246.558644] RIP: 0033:0x7710cf124d0a [ 4246.558654] RSP: 002b:00007710ccdfda40 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [ 4246.558668] RAX: ffffffffffffffda RBX: 000064bb92f67e60 RCX: 00007710cf124d0a [ 4246.558679] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000028 [ 4246.558691] RBP: 000064bb92f72670 R08: 0000000000000000 R09: 00007710ccdfe6c0 [ 4246.558702] R10: 00007710cf0adfee R11: 0000000000000293 R12: 000064bb92505940 [ 4246.558713] R13: 0000000000000002 R14: 00007ffc05649500 R15: 00007710cc5fe000 [ 4246.558728] [ 4246.558836] task:lvm state:D stack:0 pid:7835 ppid:5665 flags:0x00004006 [ 4246.558852] Call Trace: [ 4246.558857] [ 4246.558863] __schedule+0x23d/0x670 [ 4246.558874] schedule+0x5e/0xd0 [ 4246.558884] io_schedule+0x46/0x70 [ 4246.558894] dm_wait_for_bios_completion+0xfc/0x110 [ 4246.558909] ? __pfx_autoremove_wake_function+0x10/0x10 [ 4246.558922] __dm_suspend+0x7e/0x1b0 [ 4246.558932] dm_internal_suspend_noflush+0x5c/0x80 [ 4246.558946] pool_presuspend+0xcc/0x130 [dm_thin_pool] [ 4246.558968] dm_table_presuspend_targets+0x3f/0x60 [ 4246.558980] __dm_suspend+0x41/0x1b0 [ 4246.558991] dm_suspend+0xc0/0xe0 [ 4246.559001] dev_suspend+0xa5/0xd0 [ 4246.559011] ctl_ioctl+0x26e/0x350 [ 4246.559020] ? __pfx_dev_suspend+0x10/0x10 [ 4246.559032] dm_ctl_ioctl+0xe/0x20 [ 4246.559041] __x64_sys_ioctl+0x94/0xd0 [ 4246.559052] do_syscall_64+0x5c/0x90 [ 4246.559062] ? do_syscall_64+0x6b/0x90 [ 4246.559072] ? do_syscall_64+0x6b/0x90 [ 4246.559081] ? xen_pv_evtchn_do_upcall+0x54/0xb0 [ 4246.559093] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 4246.559104] RIP: 0033:0x7f1cb77cfe0f [ 4246.559112] RSP: 002b:00007fff870f2560 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 4246.559141] RAX: ffffffffffffffda RBX: 00005b8d13c16580 RCX: 00007f1cb77cfe0f [ 4246.559152] RDX: 00005b8d144a2180 RSI: 00000000c138fd06 RDI: 0000000000000003 [ 4246.559164] RBP: 00005b8d144a2180 R08: 00005b8d132b1190 R09: 00007fff870f2420 [ 4246.559175] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000c [ 4246.559186] R13: 00005b8d132aacf0 R14: 00005b8d1324414d R15: 00005b8d144a21b0 [ 4246.559199] [ 4246.559207] task:kworker/u8:3 state:D stack:0 pid:8033 ppid:2 flags:0x00004000 [ 4246.559222] Workqueue: writeback wb_workfn (flush-253:4) [ 4246.559238] Call Trace: [ 4246.559244] [ 4246.559249] __schedule+0x23d/0x670 [ 4246.559260] schedule+0x5e/0xd0 [ 4246.559270] io_schedule+0x46/0x70 [ 4246.559280] folio_wait_bit_common+0x13d/0x350 [ 4246.559290] ? __pfx_wake_page_function+0x10/0x10 [ 4246.559302] mpage_prepare_extent_to_map+0x309/0x4d0 [ 4246.559314] ext4_do_writepages+0x25f/0x770 [ 4246.559324] ext4_writepages+0xad/0x180 [ 4246.559334] do_writepages+0xcf/0x1e0 [ 4246.559344] ? find_busiest_group+0x42/0x1a0 [ 4246.559354] __writeback_single_inode+0x3d/0x280 [ 4246.559368] writeback_sb_inodes+0x1ed/0x4a0 [ 4246.559381] __writeback_inodes_wb+0x4c/0xf0 [ 4246.559393] wb_writeback+0x298/0x310 [ 4246.559403] wb_do_writeback+0x230/0x2b0 [ 4246.559414] wb_workfn+0x5f/0x260 [ 4246.559424] ? _raw_spin_unlock+0xe/0x30 [ 4246.559434] ? finish_task_switch.isra.0+0x95/0x2b0 [ 4246.559447] ? __schedule+0x245/0x670 [ 4246.559457] process_one_work+0x1df/0x3e0 [ 4246.559466] worker_thread+0x51/0x390 [ 4246.559475] ? __pfx_worker_thread+0x10/0x10 [ 4246.559484] kthread+0xe5/0x120 [ 4246.559495] ? __pfx_kthread+0x10/0x10 [ 4246.559504] ret_from_fork+0x31/0x50 [ 4246.559514] ? __pfx_kthread+0x10/0x10 [ 4246.559523] ret_from_fork_asm+0x1b/0x30 [ 4246.559536] ```
marmarek commented 12 months ago

Similar thing happens on 6.5.6 too. Whole disk I/O got frozen, no lvrename involved this time.

sysrq: Show Blocked State ``` [ 522.190523] sysrq: Show Blocked State [ 522.190621] task:dmcrypt_write/2 state:D stack:0 pid:1245 ppid:2 flags:0x00004000 [ 522.190649] Call Trace: [ 522.190660] [ 522.190681] __schedule+0x23d/0x670 [ 522.190710] ? __pfx_wbt_inflight_cb+0x10/0x10 [ 522.190735] ? __pfx_wbt_cleanup_cb+0x10/0x10 [ 522.190757] ? __pfx_dmcrypt_write+0x10/0x10 [dm_crypt] [ 522.190786] schedule+0x5e/0xd0 [ 522.190802] io_schedule+0x46/0x70 [ 522.190818] rq_qos_wait+0xc0/0x140 [ 522.190835] ? __pfx_rq_qos_wake_function+0x10/0x10 [ 522.190850] ? __pfx_wbt_inflight_cb+0x10/0x10 [ 522.190867] wbt_wait+0xa6/0x110 [ 522.190882] __rq_qos_throttle+0x24/0x40 [ 522.190895] blk_mq_get_new_requests+0xc9/0x180 [ 522.190912] blk_mq_submit_bio+0x332/0x550 [ 522.190927] __submit_bio+0x8b/0x170 [ 522.190941] submit_bio_noacct_nocheck+0x120/0x1f0 [ 522.190955] ? submit_bio_noacct+0x49/0x560 [ 522.190967] dmcrypt_write+0x11a/0x140 [dm_crypt] [ 522.191059] kthread+0xe5/0x120 [ 522.191075] ? __pfx_kthread+0x10/0x10 [ 522.191095] ret_from_fork+0x31/0x50 [ 522.191108] ? __pfx_kthread+0x10/0x10 [ 522.191121] ret_from_fork_asm+0x1b/0x30 [ 522.191141] [ 522.191151] task:kworker/u12:5 state:D stack:0 pid:1324 ppid:2 flags:0x00004000 [ 522.191172] Workqueue: writeback wb_workfn (flush-253:4) [ 522.191194] Call Trace: [ 522.191201] [ 522.191208] __schedule+0x23d/0x670 [ 522.191224] schedule+0x5e/0xd0 [ 522.191238] schedule_preempt_disabled+0x15/0x30 [ 522.191255] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.191272] down_read+0x48/0xa0 [ 522.191286] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.191319] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.191348] __map_bio+0x44/0x1b0 [ 522.191363] dm_split_and_process_bio+0x1fa/0x510 [ 522.191378] dm_submit_bio+0x4e/0x100 [ 522.191389] __submit_bio+0xb0/0x170 [ 522.191402] __submit_bio_noacct+0x90/0x220 [ 522.191415] __block_write_full_folio+0x1e1/0x400 [ 522.191431] ? __pfx_end_buffer_async_write+0x10/0x10 [ 522.191445] ? __pfx_blkdev_get_block+0x10/0x10 [ 522.191462] writepage_cb+0x1a/0x70 [ 522.191476] write_cache_pages+0x143/0x3a0 [ 522.191489] ? __pfx_writepage_cb+0x10/0x10 [ 522.191508] do_writepages+0x164/0x1e0 [ 522.191524] __writeback_single_inode+0x3d/0x280 [ 522.191542] writeback_sb_inodes+0x1ed/0x4a0 [ 522.191560] __writeback_inodes_wb+0x4c/0xf0 [ 522.191577] wb_writeback+0x298/0x310 [ 522.191592] wb_do_writeback+0x1f8/0x2b0 [ 522.191608] wb_workfn+0x5f/0x260 [ 522.191623] ? _raw_spin_unlock+0xe/0x30 [ 522.191634] ? finish_task_switch.isra.0+0x95/0x2b0 [ 522.191651] ? __schedule+0x245/0x670 [ 522.191666] process_one_work+0x1df/0x3e0 [ 522.191679] worker_thread+0x51/0x390 [ 522.191691] ? __pfx_worker_thread+0x10/0x10 [ 522.191704] kthread+0xe5/0x120 [ 522.191716] ? __pfx_kthread+0x10/0x10 [ 522.191729] ret_from_fork+0x31/0x50 [ 522.191740] ? __pfx_kthread+0x10/0x10 [ 522.191752] ret_from_fork_asm+0x1b/0x30 [ 522.191769] [ 522.191778] task:jbd2/dm-4-8 state:D stack:0 pid:1359 ppid:2 flags:0x00004000 [ 522.191795] Call Trace: [ 522.191802] [ 522.191809] __schedule+0x23d/0x670 [ 522.191825] schedule+0x5e/0xd0 [ 522.191839] schedule_preempt_disabled+0x15/0x30 [ 522.191854] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.191869] down_read+0x48/0xa0 [ 522.191880] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.191905] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.191931] __map_bio+0x44/0x1b0 [ 522.191941] dm_split_and_process_bio+0x1fa/0x510 [ 522.191954] dm_submit_bio+0x4e/0x100 [ 522.191965] __submit_bio+0xb0/0x170 [ 522.191976] __submit_bio_noacct+0x90/0x220 [ 522.191987] jbd2_journal_commit_transaction+0x63d/0x1a10 [ 522.192030] ? finish_task_switch.isra.0+0x95/0x2b0 [ 522.192046] kjournald2+0xad/0x280 [ 522.192060] ? __pfx_autoremove_wake_function+0x10/0x10 [ 522.192076] ? __pfx_kjournald2+0x10/0x10 [ 522.192088] kthread+0xe5/0x120 [ 522.192099] ? __pfx_kthread+0x10/0x10 [ 522.192111] ret_from_fork+0x31/0x50 [ 522.192120] ? __pfx_kthread+0x10/0x10 [ 522.192131] ret_from_fork_asm+0x1b/0x30 [ 522.192146] [ 522.192153] task:systemd-journal state:D stack:0 pid:1445 ppid:1 flags:0x00000002 [ 522.192169] Call Trace: [ 522.192176] [ 522.192183] __schedule+0x23d/0x670 [ 522.192197] schedule+0x5e/0xd0 [ 522.192210] schedule_preempt_disabled+0x15/0x30 [ 522.192225] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.192239] down_read+0x48/0xa0 [ 522.192249] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.192274] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.192299] __map_bio+0x44/0x1b0 [ 522.192310] dm_split_and_process_bio+0x1fa/0x510 [ 522.192322] dm_submit_bio+0x4e/0x100 [ 522.192332] __submit_bio+0xb0/0x170 [ 522.192343] __submit_bio_noacct+0x90/0x220 [ 522.192354] ext4_read_bh+0x46/0x90 [ 522.192367] ext4_bread+0x52/0x70 [ 522.192378] __ext4_read_dirblock+0x52/0x340 [ 522.192393] htree_dirblock_to_tree+0xac/0x370 [ 522.192409] ext4_htree_fill_tree+0x113/0x3d0 [ 522.192425] ext4_dx_readdir+0x108/0x370 [ 522.192437] ext4_readdir+0x5b2/0x6f0 [ 522.192448] iterate_dir+0x90/0x170 [ 522.192463] __x64_sys_getdents64+0x88/0x130 [ 522.192477] ? __pfx_filldir64+0x10/0x10 [ 522.192490] ? syscall_trace_enter.constprop.0+0x124/0x1a0 [ 522.192504] do_syscall_64+0x5c/0x90 [ 522.192519] ? syscall_exit_work+0x103/0x130 [ 522.192531] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.192544] ? do_syscall_64+0x6b/0x90 [ 522.192557] ? syscall_exit_work+0x103/0x130 [ 522.192569] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.192580] ? do_syscall_64+0x6b/0x90 [ 522.192593] ? syscall_exit_work+0x103/0x130 [ 522.192604] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.192615] ? do_syscall_64+0x6b/0x90 [ 522.192628] ? exit_to_user_mode_prepare+0xbc/0xd0 [ 522.192640] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.192656] RIP: 0033:0x7fb522cf5bc7 [ 522.192668] RSP: 002b:00007ffe8c54c5b8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9 [ 522.192686] RAX: ffffffffffffffda RBX: 0000564cdf0a27a0 RCX: 00007fb522cf5bc7 [ 522.192700] RDX: 0000000000008000 RSI: 0000564cdf0a27a0 RDI: 0000000000000025 [ 522.192714] RBP: 0000564cdf0a2774 R08: 0000000000008030 R09: 0000000000000000 [ 522.192727] R10: 0000000000000000 R11: 0000000000000293 R12: fffffffffffffdb8 [ 522.192741] R13: 0000000000000000 R14: 0000564cdf0a2770 R15: 00007ffe8c54d068 [ 522.192757] [ 522.192764] task:journal-offline state:D stack:0 pid:4053 ppid:1 flags:0x00000002 [ 522.192778] Call Trace: [ 522.192787] [ 522.192794] __schedule+0x23d/0x670 [ 522.192807] schedule+0x5e/0xd0 [ 522.192818] schedule_timeout+0x151/0x160 [ 522.192833] io_schedule_timeout+0x50/0x80 [ 522.192845] wait_for_completion_io+0x8a/0x160 [ 522.192856] submit_bio_wait+0x61/0x90 [ 522.192869] blkdev_issue_flush+0x4d/0x80 [ 522.192882] ? __pfx_submit_bio_wait_endio+0x10/0x10 [ 522.192896] ext4_sync_file+0x2b8/0x380 [ 522.192908] __x64_sys_fsync+0x3b/0x70 [ 522.192919] do_syscall_64+0x5c/0x90 [ 522.192931] ? syscall_exit_work+0x103/0x130 [ 522.192942] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.192953] ? do_syscall_64+0x6b/0x90 [ 522.192965] ? syscall_exit_work+0x103/0x130 [ 522.192975] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.192986] ? do_syscall_64+0x6b/0x90 [ 522.193011] ? switch_fpu_return+0x50/0xe0 [ 522.193023] ? exit_to_user_mode_prepare+0xb2/0xd0 [ 522.193034] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.193046] RIP: 0033:0x7fb522d24d0a [ 522.193054] RSP: 002b:00007fb5211fea40 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [ 522.193068] RAX: ffffffffffffffda RBX: 0000564cdf05ee70 RCX: 00007fb522d24d0a [ 522.193079] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000017 [ 522.193091] RBP: 0000564cdf053d80 R08: 0000000000000000 R09: 00007fb5211ff6c0 [ 522.193104] R10: 00007fb522cadfee R11: 0000000000000293 R12: 0000564cde251940 [ 522.193115] R13: 0000000000000002 R14: 00007ffe8c54cac0 R15: 00007fb5209ff000 [ 522.193129] [ 522.193134] task:journal-offline state:D stack:0 pid:4054 ppid:1 flags:0x00000002 [ 522.193147] Call Trace: [ 522.193153] [ 522.193159] __schedule+0x23d/0x670 [ 522.193171] schedule+0x5e/0xd0 [ 522.193183] schedule_timeout+0x151/0x160 [ 522.193195] io_schedule_timeout+0x50/0x80 [ 522.193206] wait_for_completion_io+0x8a/0x160 [ 522.193217] submit_bio_wait+0x61/0x90 [ 522.193229] blkdev_issue_flush+0x4d/0x80 [ 522.193241] ? __pfx_submit_bio_wait_endio+0x10/0x10 [ 522.193255] ext4_sync_file+0x2b8/0x380 [ 522.193265] __x64_sys_fsync+0x3b/0x70 [ 522.193275] do_syscall_64+0x5c/0x90 [ 522.193289] ? syscall_exit_work+0x103/0x130 [ 522.193299] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.193312] ? do_syscall_64+0x6b/0x90 [ 522.193324] ? exit_to_user_mode_prepare+0xb2/0xd0 [ 522.193335] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.193403] RIP: 0033:0x7fb522d24d0a [ 522.193411] RSP: 002b:00007fb522310a40 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [ 522.193424] RAX: ffffffffffffffda RBX: 0000564cdf06f450 RCX: 00007fb522d24d0a [ 522.193436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000002a [ 522.193447] RBP: 0000564cdf052c60 R08: 0000000000000000 R09: 00007fb5223116c0 [ 522.193459] R10: 00007fb522cadfee R11: 0000000000000293 R12: 0000564cde251940 [ 522.193471] R13: 0000000000000002 R14: 00007ffe8c54cac0 R15: 00007fb521b11000 [ 522.193484] [ 522.193491] task:dmeventd state:D stack:0 pid:2049 ppid:1 flags:0x00000002 [ 522.193505] Call Trace: [ 522.193510] [ 522.193516] __schedule+0x23d/0x670 [ 522.193529] schedule+0x5e/0xd0 [ 522.193541] schedule_preempt_disabled+0x15/0x30 [ 522.193554] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.193567] down_read+0x48/0xa0 [ 522.193577] dm_pool_get_metadata_transaction_id+0x22/0x60 [dm_thin_pool] [ 522.193602] pool_status+0x122/0x600 [dm_thin_pool] [ 522.193626] retrieve_status+0x15d/0x210 [ 522.193638] table_status+0x9b/0x140 [ 522.193650] ctl_ioctl+0x26e/0x350 [ 522.193660] ? __pfx_table_status+0x10/0x10 [ 522.193675] dm_ctl_ioctl+0xe/0x20 [ 522.193685] __x64_sys_ioctl+0x94/0xd0 [ 522.193697] do_syscall_64+0x5c/0x90 [ 522.193709] ? do_syscall_64+0x6b/0x90 [ 522.193720] ? __rseq_handle_notify_resume+0x36/0x60 [ 522.193733] ? exit_to_user_mode_loop+0x42/0x150 [ 522.193746] ? exit_to_user_mode_prepare+0xbc/0xd0 [ 522.193756] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.193766] ? do_syscall_64+0x6b/0x90 [ 522.193776] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.193786] ? do_syscall_64+0x6b/0x90 [ 522.193796] ? do_syscall_64+0x6b/0x90 [ 522.193806] ? do_syscall_64+0x6b/0x90 [ 522.193817] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.193828] RIP: 0033:0x72fbc9b09e0f [ 522.193835] RSP: 002b:000072fbc91229a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 522.193846] RAX: ffffffffffffffda RBX: 000072fbc4a4e4a0 RCX: 000072fbc9b09e0f [ 522.193859] RDX: 000072fbc4067c30 RSI: 00000000c138fd0c RDI: 0000000000000007 [ 522.193870] RBP: 000000000000001e R08: 0000000000000004 R09: 000072fbc9c51858 [ 522.193881] R10: 000072fbc9c4f576 R11: 0000000000000246 R12: 000072fbc9c4f576 [ 522.193892] R13: 000072fbc9c4f576 R14: 000072fbc4067c60 R15: 000072fbc9c4f576 [ 522.193904] [ 522.193916] task:alsactl state:D stack:0 pid:2658 ppid:1 flags:0x00000002 [ 522.193928] Call Trace: [ 522.193933] [ 522.193939] __schedule+0x23d/0x670 [ 522.193951] schedule+0x5e/0xd0 [ 522.193961] schedule_preempt_disabled+0x15/0x30 [ 522.193974] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.193986] down_read+0x48/0xa0 [ 522.194006] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.194028] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.194049] __map_bio+0x44/0x1b0 [ 522.194058] dm_split_and_process_bio+0x1fa/0x510 [ 522.194069] dm_submit_bio+0x4e/0x100 [ 522.194078] __submit_bio+0xb0/0x170 [ 522.194087] __submit_bio_noacct+0x90/0x220 [ 522.194097] ext4_mpage_readpages+0x2ba/0x7b0 [ 522.194110] read_pages+0x85/0x260 [ 522.194121] page_cache_ra_unbounded+0x12d/0x180 [ 522.194135] filemap_get_pages+0xfc/0x350 [ 522.194146] filemap_read+0xd9/0x350 [ 522.194155] ? generic_fillattr+0x49/0x100 [ 522.194168] vfs_read+0x1fe/0x350 [ 522.194181] ksys_read+0x6f/0xf0 [ 522.194191] do_syscall_64+0x5c/0x90 [ 522.194202] ? do_syscall_64+0x6b/0x90 [ 522.194212] ? do_syscall_64+0x6b/0x90 [ 522.194223] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.194234] RIP: 0033:0x7612dae96281 [ 522.194242] RSP: 002b:00007ffd04324f88 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 522.194253] RAX: ffffffffffffffda RBX: 000061446b3502c0 RCX: 00007612dae96281 [ 522.194264] RDX: 0000000000001000 RSI: 000061446b35d190 RDI: 0000000000000006 [ 522.194275] RBP: 00007612daf6b600 R08: 0000000000001000 R09: 0000000000000000 [ 522.194286] R10: 0000000000000000 R11: 0000000000000246 R12: 000061446b351430 [ 522.194297] R13: 0000000000000d68 R14: 00007612daf6aa00 R15: 0000000000000d68 [ 522.194309] [ 522.194318] task:chronyd state:D stack:0 pid:4065 ppid:1 flags:0x00000002 [ 522.194331] Call Trace: [ 522.194335] [ 522.194341] __schedule+0x23d/0x670 [ 522.194352] schedule+0x5e/0xd0 [ 522.194363] schedule_preempt_disabled+0x15/0x30 [ 522.194375] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.194388] down_read+0x48/0xa0 [ 522.194397] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.194417] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.194438] __map_bio+0x44/0x1b0 [ 522.194447] dm_split_and_process_bio+0x1fa/0x510 [ 522.194458] dm_submit_bio+0x4e/0x100 [ 522.194466] __submit_bio+0xb0/0x170 [ 522.194475] __submit_bio_noacct+0x90/0x220 [ 522.194485] ext4_mpage_readpages+0x2ba/0x7b0 [ 522.194497] read_pages+0x85/0x260 [ 522.194508] page_cache_ra_unbounded+0x12d/0x180 [ 522.194522] filemap_get_pages+0xfc/0x350 [ 522.194532] filemap_read+0xd9/0x350 [ 522.194541] ? dput+0x3a/0x310 [ 522.194552] vfs_read+0x1fe/0x350 [ 522.194561] ksys_read+0x6f/0xf0 [ 522.194570] do_syscall_64+0x5c/0x90 [ 522.194580] ? switch_fpu_return+0x50/0xe0 [ 522.194591] ? exit_to_user_mode_prepare+0xb2/0xd0 [ 522.194601] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.194612] RIP: 0033:0x7545397231b8 [ 522.194619] RSP: 002b:000075453922b238 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 522.194631] RAX: ffffffffffffffda RBX: 0000754534000ba0 RCX: 00007545397231b8 [ 522.194642] RDX: 0000000000001000 RSI: 0000754534003660 RDI: 0000000000000009 [ 522.194652] RBP: 00007545397f3600 R08: 0000000000000001 R09: 0000000000000000 [ 522.194663] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a [ 522.194673] R13: 0000000000000d68 R14: 00007545397f2a00 R15: 0000000000000d68 [ 522.194686] [ 522.194691] task:libvirtd state:D stack:0 pid:2708 ppid:1 flags:0x00000002 [ 522.194704] Call Trace: [ 522.194709] [ 522.194714] __schedule+0x23d/0x670 [ 522.194727] schedule+0x5e/0xd0 [ 522.194737] schedule_preempt_disabled+0x15/0x30 [ 522.194749] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.194760] down_read+0x48/0xa0 [ 522.194768] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.194787] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.194806] __map_bio+0x44/0x1b0 [ 522.194814] dm_split_and_process_bio+0x1fa/0x510 [ 522.194824] dm_submit_bio+0x4e/0x100 [ 522.194832] __submit_bio+0xb0/0x170 [ 522.194840] __submit_bio_noacct+0x90/0x220 [ 522.194849] ext4_mpage_readpages+0x2ba/0x7b0 [ 522.194861] read_pages+0x85/0x260 [ 522.194871] page_cache_ra_unbounded+0x12d/0x180 [ 522.194884] filemap_get_pages+0xfc/0x350 [ 522.194894] filemap_read+0xd9/0x350 [ 522.194902] ? dput+0x3a/0x310 [ 522.194912] vfs_read+0x1fe/0x350 [ 522.194921] ksys_read+0x6f/0xf0 [ 522.194929] do_syscall_64+0x5c/0x90 [ 522.194939] ? count_memcg_events.constprop.0+0x24/0x30 [ 522.194951] ? handle_mm_fault+0x9e/0x350 [ 522.194959] ? do_user_addr_fault+0x179/0x640 [ 522.194970] ? exc_page_fault+0x7c/0x170 [ 522.194978] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.194989] RIP: 0033:0x70c45f5331b8 [ 522.195005] RSP: 002b:00007ffcb9677588 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 522.195017] RAX: ffffffffffffffda RBX: 00006526d8896000 RCX: 000070c45f5331b8 [ 522.195027] RDX: 0000000000001000 RSI: 00006526d88a3710 RDI: 0000000000000016 [ 522.195037] RBP: 000070c45f603600 R08: 0000000000000001 R09: 0000000000000000 [ 522.195047] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a [ 522.195057] R13: 0000000000000d68 R14: 000070c45f602a00 R15: 0000000000000d68 [ 522.195069] [ 522.195080] task:kworker/u12:7 state:D stack:0 pid:2731 ppid:2 flags:0x00004000 [ 522.195093] Workqueue: dm-thin do_worker [dm_thin_pool] [ 522.195113] Call Trace: [ 522.195118] [ 522.195123] __schedule+0x23d/0x670 [ 522.195134] schedule+0x5e/0xd0 [ 522.195144] io_schedule+0x46/0x70 [ 522.195154] bit_wait_io+0x11/0x70 [ 522.195162] __wait_on_bit+0x4c/0x150 [ 522.195169] ? __pfx_bit_wait_io+0x10/0x10 [ 522.195177] out_of_line_wait_on_bit+0x95/0xc0 [ 522.195187] ? __pfx_wake_bit_function+0x10/0x10 [ 522.195199] dm_bufio_write_dirty_buffers+0x21f/0x2a0 [ 522.195212] ? metadata_ll_commit+0xbb/0xe0 [dm_persistent_data] [ 522.195244] __commit_transaction+0x18f/0x280 [dm_thin_pool] [ 522.195265] ? __slab_free+0xf1/0x320 [ 522.195276] dm_pool_commit_metadata+0x30/0x60 [dm_thin_pool] [ 522.195298] commit+0x31/0x120 [dm_thin_pool] [ 522.195317] process_deferred_bios+0x2b2/0x2e0 [dm_thin_pool] [ 522.195337] do_worker+0xfb/0x170 [dm_thin_pool] [ 522.195356] process_one_work+0x1df/0x3e0 [ 522.195365] worker_thread+0x51/0x390 [ 522.195373] ? __pfx_worker_thread+0x10/0x10 [ 522.195382] kthread+0xe5/0x120 [ 522.195391] ? __pfx_kthread+0x10/0x10 [ 522.195400] ret_from_fork+0x31/0x50 [ 522.195407] ? __pfx_kthread+0x10/0x10 [ 522.195416] ret_from_fork_asm+0x1b/0x30 [ 522.195428] [ 522.195478] task:rpm2archive state:D stack:0 pid:4040 ppid:4039 flags:0x00000002 [ 522.195490] Call Trace: [ 522.195495] [ 522.195500] __schedule+0x23d/0x670 [ 522.195511] schedule+0x5e/0xd0 [ 522.195523] schedule_preempt_disabled+0x15/0x30 [ 522.195534] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.195546] down_read+0x48/0xa0 [ 522.195554] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.195573] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.195593] __map_bio+0x44/0x1b0 [ 522.195601] dm_split_and_process_bio+0x1fa/0x510 [ 522.195611] dm_submit_bio+0x4e/0x100 [ 522.195619] __submit_bio+0xb0/0x170 [ 522.195627] __submit_bio_noacct+0x90/0x220 [ 522.195636] ext4_mpage_readpages+0x2ba/0x7b0 [ 522.195648] read_pages+0x85/0x260 [ 522.195658] page_cache_ra_unbounded+0x12d/0x180 [ 522.195670] filemap_get_pages+0x2cf/0x350 [ 522.195680] filemap_read+0xd9/0x350 [ 522.195688] ? perf_event_task_tick+0x56/0xa0 [ 522.195700] vfs_read+0x1fe/0x350 [ 522.195709] ksys_read+0x6f/0xf0 [ 522.195716] do_syscall_64+0x5c/0x90 [ 522.195727] ? handle_percpu_irq+0x3b/0x60 [ 522.195737] ? handle_irq_desc+0x3b/0x50 [ 522.195746] ? __evtchn_fifo_handle_events+0x1b4/0x1e0 [ 522.195760] ? xen_sched_clock+0x15/0x30 [ 522.195770] ? sched_clock+0x10/0x30 [ 522.195780] ? sched_clock_cpu+0xf/0x190 [ 522.195790] ? irqtime_account_irq+0x40/0xc0 [ 522.195800] ? __irq_exit_rcu+0x4b/0xc0 [ 522.195810] ? xen_pv_evtchn_do_upcall+0x54/0xb0 [ 522.195823] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.195833] RIP: 0033:0x750fb0b38281 [ 522.195839] RSP: 002b:00007ffc465a6e38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 522.195849] RAX: ffffffffffffffda RBX: 000060cea227b5d0 RCX: 0000750fb0b38281 [ 522.195858] RDX: 000000000001f000 RSI: 000060cea22be0f7 RDI: 0000000000000003 [ 522.195868] RBP: 0000750fb0c0d600 R08: 0000000000000000 R09: 0000000000000172 [ 522.195877] R10: 00007ffc465a6ee0 R11: 0000000000000246 R12: 000060cea22be0f7 [ 522.195887] R13: 000000000001f8cc R14: 0000000000000d68 R15: 0000750fb0c0ca00 [ 522.195898] [ 522.195904] task:kworker/u12:9 state:D stack:0 pid:4045 ppid:2 flags:0x00004000 [ 522.195915] Workqueue: writeback wb_workfn (flush-253:4) [ 522.195928] Call Trace: [ 522.195932] [ 522.195937] __schedule+0x23d/0x670 [ 522.195947] schedule+0x5e/0xd0 [ 522.195956] schedule_preempt_disabled+0x15/0x30 [ 522.195967] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.195977] down_read+0x48/0xa0 [ 522.195985] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.196013] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.196032] __map_bio+0x44/0x1b0 [ 522.196039] dm_split_and_process_bio+0x1fa/0x510 [ 522.196048] dm_submit_bio+0x4e/0x100 [ 522.196057] __submit_bio+0xb0/0x170 [ 522.196066] __submit_bio_noacct+0x90/0x220 [ 522.196074] ext4_io_submit+0x24/0x40 [ 522.196081] ext4_do_writepages+0x3af/0x770 [ 522.196091] ext4_writepages+0xad/0x180 [ 522.196101] do_writepages+0xcf/0x1e0 [ 522.196110] __writeback_single_inode+0x3d/0x280 [ 522.196121] writeback_sb_inodes+0x1ed/0x4a0 [ 522.196133] __writeback_inodes_wb+0x4c/0xf0 [ 522.196144] wb_writeback+0x298/0x310 [ 522.196154] wb_do_writeback+0x230/0x2b0 [ 522.196164] wb_workfn+0x5f/0x260 [ 522.196173] ? _raw_spin_unlock+0xe/0x30 [ 522.196180] ? finish_task_switch.isra.0+0x95/0x2b0 [ 522.196191] ? __schedule+0x245/0x670 [ 522.196200] process_one_work+0x1df/0x3e0 [ 522.196208] worker_thread+0x51/0x390 [ 522.196216] ? __pfx_worker_thread+0x10/0x10 [ 522.196224] kthread+0xe5/0x120 [ 522.196232] ? __pfx_kthread+0x10/0x10 [ 522.196241] ret_from_fork+0x31/0x50 [ 522.196247] ? __pfx_kthread+0x10/0x10 [ 522.196255] ret_from_fork_asm+0x1b/0x30 [ 522.196266] [ 522.196271] task:(tmpfiles) state:D stack:0 pid:4059 ppid:3425 flags:0x00000002 [ 522.196283] Call Trace: [ 522.196287] [ 522.196292] __schedule+0x23d/0x670 [ 522.196302] schedule+0x5e/0xd0 [ 522.196311] schedule_preempt_disabled+0x15/0x30 [ 522.196322] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.196332] down_read+0x48/0xa0 [ 522.196340] dm_thin_find_block+0x32/0x80 [dm_thin_pool] [ 522.196357] thin_bio_map.isra.0+0x15d/0x380 [dm_thin_pool] [ 522.196375] __map_bio+0x44/0x1b0 [ 522.196383] dm_split_and_process_bio+0x1fa/0x510 [ 522.196392] dm_submit_bio+0x4e/0x100 [ 522.196400] __submit_bio+0xb0/0x170 [ 522.196407] __submit_bio_noacct+0x90/0x220 [ 522.196415] ext4_mpage_readpages+0x2ba/0x7b0 [ 522.196426] read_pages+0x85/0x260 [ 522.196436] page_cache_ra_unbounded+0x12d/0x180 [ 522.196447] filemap_get_pages+0xfc/0x350 [ 522.196456] filemap_read+0xd9/0x350 [ 522.196465] __kernel_read+0x134/0x2e0 [ 522.196474] search_binary_handler+0x71/0x260 [ 522.196485] exec_binprm+0x51/0x1a0 [ 522.196495] bprm_execve.part.0+0x180/0x220 [ 522.196505] do_execveat_common.isra.0+0x1ad/0x250 [ 522.196516] __x64_sys_execve+0x36/0x40 [ 522.196524] do_syscall_64+0x5c/0x90 [ 522.196534] ? __count_memcg_events+0x4d/0x90 [ 522.196543] ? count_memcg_events.constprop.0+0x24/0x30 [ 522.196553] ? handle_mm_fault+0x9e/0x350 [ 522.196561] ? do_user_addr_fault+0x225/0x640 [ 522.196569] ? syscall_exit_work+0x103/0x130 [ 522.196578] ? exc_page_fault+0x7c/0x170 [ 522.196586] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.196595] RIP: 0033:0x7f2bd24fa4db [ 522.196602] RSP: 002b:00007fff1c6a1328 EFLAGS: 00000206 ORIG_RAX: 000000000000003b [ 522.196613] RAX: ffffffffffffffda RBX: 00005d221a754700 RCX: 00007f2bd24fa4db [ 522.196623] RDX: 00005d221a7a3610 RSI: 00005d221a754700 RDI: 00005d221a8baad0 [ 522.196632] RBP: 00007fff1c6a15f0 R08: 0000000000000007 R09: 0000000000000030 [ 522.196641] R10: 0000000000000011 R11: 0000000000000206 R12: 00005d221a791ad0 [ 522.196651] R13: 00000000ffffffff R14: 0000000000000000 R15: 00005d221a792038 [ 522.196662] [ 522.196667] task:lvm state:D stack:0 pid:4062 ppid:2673 flags:0x00000002 [ 522.196678] Call Trace: [ 522.196682] [ 522.196687] __schedule+0x23d/0x670 [ 522.196697] schedule+0x5e/0xd0 [ 522.196706] schedule_preempt_disabled+0x15/0x30 [ 522.196717] rwsem_down_read_slowpath+0x29e/0x4f0 [ 522.196727] down_read+0x48/0xa0 [ 522.196735] dm_pool_get_metadata_transaction_id+0x22/0x60 [dm_thin_pool] [ 522.196755] pool_status+0x122/0x600 [dm_thin_pool] [ 522.196773] retrieve_status+0x15d/0x210 [ 522.196783] table_status+0x9b/0x140 [ 522.196792] ctl_ioctl+0x26e/0x350 [ 522.196800] ? __pfx_table_status+0x10/0x10 [ 522.196813] dm_ctl_ioctl+0xe/0x20 [ 522.196821] __x64_sys_ioctl+0x94/0xd0 [ 522.196829] do_syscall_64+0x5c/0x90 [ 522.196838] ? syscall_exit_work+0x103/0x130 [ 522.196846] ? syscall_exit_to_user_mode+0x2b/0x40 [ 522.196855] ? do_syscall_64+0x6b/0x90 [ 522.196863] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 522.196873] RIP: 0033:0x7eea6b7c4e0f [ 522.196878] RSP: 002b:00007ffe60c65e60 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 522.196888] RAX: ffffffffffffffda RBX: 00005644067b2c20 RCX: 00007eea6b7c4e0f [ 522.196896] RDX: 0000564406ffd0e0 RSI: 00000000c138fd0c RDI: 0000000000000004 [ 522.196905] RBP: 0000564406ffd0e0 R08: 0000564404e71190 R09: 00007ffe60c65d20 [ 522.196914] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000001e [ 522.196923] R13: 0000564404e6acf0 R14: 0000564404e6acf0 R15: 0000564406ffd110 [ 522.196936] ```
marmarek commented 11 months ago

At one time I've got the issue without lvm among blocked processes

Blocked processes ``` [ 299.388109] sysrq: Show Blocked State [ 299.391268] task:rpm2archive state:D stack:0 pid:3050 ppid:3049 flags:0x00000002 [ 299.392837] Call Trace: [ 299.393430] [ 299.393880] __schedule+0x526/0x18b0 [ 299.394338] ? folio_wait_bit_common+0xd1/0x360 [ 299.394793] schedule+0x5e/0xd0 [ 299.395247] io_schedule+0x46/0x70 [ 299.395693] folio_wait_bit_common+0x13d/0x360 [ 299.396174] ? __pfx_wake_page_function+0x10/0x10 [ 299.396717] filemap_get_pages+0x60b/0x640 [ 299.397247] ? lock_release+0xd1/0x280 [ 299.397686] filemap_read+0xd9/0x350 [ 299.398127] vfs_read+0x1f3/0x320 [ 299.398547] ksys_read+0x75/0x100 [ 299.399007] do_syscall_64+0x5c/0x90 [ 299.399504] ? irqtime_account_irq+0x40/0xc0 [ 299.399996] ? __irq_exit_rcu+0x59/0x100 [ 299.400507] ? irq_exit_rcu+0xe/0x20 [ 299.400998] ? lockdep_hardirqs_on+0x51/0x100 [ 299.401511] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 299.402011] RIP: 0033:0x7b8f18bf9281 [ 299.402521] RSP: 002b:00007ffebf0070d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 299.403031] RAX: ffffffffffffffda RBX: 000059320f2ca5d0 RCX: 00007b8f18bf9281 [ 299.403561] RDX: 000000000001f000 RSI: 000059320f30d35f RDI: 0000000000000003 [ 299.404102] RBP: 00007b8f18cce600 R08: 000059320f30c9c0 R09: 00000000000027a9 [ 299.404575] R10: 00007ffebf007180 R11: 0000000000000246 R12: 000059320f30d35f [ 299.405006] R13: 000000000001f664 R14: 0000000000000d68 R15: 00007b8f18ccda00 [ 299.405459] ```
marmarek commented 11 months ago

Upstream report: https://lore.kernel.org/all/ZTNH0qtmint%2FzLJZ@mail-itl/

qubesos-bot commented 10 months ago

Automated announcement from builder-github

The component linux-kernel-latest (including package kernel-latest-6.6.2-1.qubes.fc32) has been pushed to the r4.1 testing repository for dom0. To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

qubesos-bot commented 10 months ago

Automated announcement from builder-github

The component linux-kernel-latest (including package kernel-latest-6.6.2-1.qubes.fc32) has been pushed to the r4.1 stable repository for dom0. To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update