search

TritonDataCenter / illumos-kvm-cmd

qemu-kvm for illumos-kvm
Other
64 stars 40 forks source link

qemu-kvm-sys 100% SYS CPU, guest hangs #8

Closed estibi closed 12 years ago

estibi commented 12 years ago

Host machine: oi_151a, oi_151a2 Both kvm and kvm-cmd are built by hand from github - the latest version.

When multiple guests are booted at the same time, some of them hangs.

VMs is are Linux hosts.

mpstat 1 3

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 97 0 24 3821 1847 1499 1 38 464 0 813 0 11 0 89 1 56 0 61 2192 1054 1502 3 33 441 0 1224 0 20 0 79 2 43 0 24 181 19 1434 2 22 600 0 469 0 7 0 93 3 40 0 13 227 28 1215 2 24 674 0 913 0 22 0 78 4 97 0 47 199 17 1097 1 23 226 0 784 0 3 0 97 5 41 0 40 204 22 999 1 20 138 0 1180 0 4 0 96 6 50 0 41 1819 912 1535 2 17 222 0 764 0 4 0 96 7 94 0 96 3759 1842 1828 3 22 202 0 904 0 6 0 94 8 26 0 22 469 156 1646 1 24 450 0 442 0 9 0 91 9 24 0 11 190 19 741 1 17 222 0 497 0 9 0 90 10 30 0 33 1789 865 1667 2 19 993 0 397 0 10 0 90 11 19 0 15 163 17 977 1 13 291 0 270 0 8 0 92 12 56 0 59 1069 482 1678 2 21 186 0 1117 0 7 0 93 13 34 0 68 2327 1127 1770 2 17 468 0 310 0 6 0 94 14 39 0 60 2461 1186 1685 2 19 209 0 651 0 4 0 96 15 73 0 41 190 21 958 1 16 127 0 333 0 2 0 98 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 2399 1147 135 0 18 9 0 347 0 0 0 100 1 0 0 0 4 0 0 3 0 0 0 3 0 100 0 0 2 0 0 0 113 10 167 0 6 1 0 216 1 0 0 99 3 0 0 0 64 14 2051 4 4 301 0 121 0 1 0 99 4 21 0 1 128 9 211 2 9 2 0 336 0 0 0 100 5 0 0 4 74 14 1636 2 16 210 0 97 0 33 0 67 6 0 0 5 3979 1982 3300 2 15 174 0 44 0 3 0 97 7 0 0 4 2177 1075 2321 3 20 98 0 730 0 2 0 98 8 0 0 0 2006 1000 0 4 0 0 0 4 0 100 0 0 9 0 0 0 44 14 22 0 1 0 0 95 0 0 0 100 10 0 0 0 100 7 2273 0 21 286 0 315 0 1 0 99 11 0 0 0 223 72 146 0 3 19 0 665 0 0 0 100 12 0 0 5 2078 1025 3248 6 25 242 0 342 1 2 0 97 13 0 0 2 4076 2031 1740 1 15 31 0 51 0 14 0 86 14 0 0 2 83 19 72 6 12 1 0 73 0 55 0 45 15 0 0 0 64 14 50 0 11 3 0 18 0 0 0 100 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 2413 1157 183 0 15 20 0 662 1 1 0 98 1 0 0 0 5 1 0 3 0 0 0 3 0 100 0 0 2 0 0 0 100 2 158 0 2 0 0 142 0 0 0 100 3 0 0 0 33 2 1982 3 0 257 0 5 0 2 0 98 4 0 0 0 80 20 50 0 7 0 0 153 0 1 0 99 5 0 0 1 90 31 4798 2 13 637 0 620 0 4 0 96 6 0 0 1 4009 2001 208 0 10 10 0 10 0 2 0 98 7 8 0 1 2098 1034 2505 8 19 158 0 652 0 2 0 98 8 0 0 0 2004 1000 0 3 0 0 0 3 0 100 0 0 9 0 0 0 14 6 8 0 1 0 0 0 0 0 0 100 10 0 0 0 115 4 2310 0 8 353 0 298 0 2 0 98 11 0 0 0 217 72 144 0 2 11 0 665 0 0 0 100 12 0 0 2 2047 1010 4036 2 14 52 0 125 0 2 0 98 13 0 0 3 4079 2031 871 3 15 23 0 76 0 1 0 99 14 0 0 0 18 10 0 4 0 0 0 4 0 100 0 0 15 0 0 0 103 40 99 0 14 1 0 31 1 0 0 99

PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 2317 root 0.0 100 0.0 0.0 0.0 0.0 0.0 0.0 0 18 18 0 qemu-kvm-sys/6 2317 root 0.0 100 0.0 0.0 0.0 0.0 0.0 0.0 0 16 16 0 qemu-kvm-sys/5 2317 root 0.0 100 0.0 0.0 0.0 0.0 0.0 0.0 0 16 16 0 qemu-kvm-sys/4 2320 root 0.0 3.0 0.0 0.0 0.0 0.0 97 0.4 9K 15 917 6 qemu-kvm-sys/3 2323 root 0.0 2.5 0.0 0.0 0.0 0.0 97 0.4 9K 2 1K 0 qemu-kvm-sys/3 2317 root 0.0 1.5 0.0 0.0 0.0 0.0 98 0.4 9K 17 17 0 qemu-kvm-sys/3

pstack 2317

2317: qemu-kvm -enable-kvm -nographic -smp 4 -m 12288 -vnc 0.0.0.0:51 -no-hp ----------------- lwp# 1 / thread# 1 -------------------- fffffd7fff2cfeda pollsys (fffffd7fffdf9650, 4, fffffd7fffdf9760, 0) fffffd7fff25c154 pselect () + 18c fffffd7fff25c668 select () + 70 00000000004bcf1f main_loop_wait () + 19f 00000000004cf70b kvm_main_loop () + bb 00000000004be406 main () + 9a6 00000000004b031c _start () + 6c ----------------- lwp# 2 / thread# 2 -------------------- fffffd7fff2cf54a sigtimedwait (f75cc0, fffffd7ffee9ecb0, 0) fffffd7fff2b8704 sigwaitinfo () + c 00000000004db088 sigwait_compat () + 48 fffffd7fff2c6fd4 _thrp_setup () + bc fffffd7fff2c72a0 _lwp_start () ----------------- lwp# 3 / thread# 3 -------------------- fffffd7fff2cfc5a ioctl (b, 2000ae80, 0) 00000000004cecd9 kvm_cpu_exec () + 9 00000000004cf33a ap_main_loop () + 22a fffffd7fff2c6fd4 _thrp_setup () + bc fffffd7fff2c72a0 _lwp_start () ----------------- lwp# 4 / thread# 4 -------------------- fffffd7fff2cfc5a ioctl () + a 00000000004cecd9 kvm_cpu_exec () + 9 00000000004cf33a ap_main_loop () + 22a fffffd7fff2c6fd4 _thrp_setup () + bc fffffd7fff2c72a0 _lwp_start () ----------------- lwp# 5 / thread# 5 -------------------- fffffd7fff2cfc5a ioctl () + a 00000000004cecd9 kvm_cpu_exec () + 9 00000000004cf33a ap_main_loop () + 22a fffffd7fff2c6fd4 _thrp_setup () + bc fffffd7fff2c72a0 _lwp_start () ----------------- lwp# 6 / thread# 6 -------------------- fffffd7fff2cfc5a ioctl () + a 00000000004cecd9 kvm_cpu_exec () + 9 00000000004cf33a ap_main_loop () + 22a fffffd7fff2c6fd4 _thrp_setup () + bc fffffd7fff2c72a0 _lwp_start () ----------------- lwp# 7 / thread# 7 -------------------- fffffd7fff2cf54a sigtimedwait (3011f4530, fffffd7ffe162cb0, 0) fffffd7fff2b8704 sigwaitinfo () + c 00000000004db088 sigwait_compat () + 48 fffffd7fff2c6fd4 _thrp_setup () + bc fffffd7fff2c72a0 _lwp_start ()

rmustacc commented 12 years ago

Please provide the kernel threads. Also, you are going to want to use the latest illumos bits, there have been various bugs that QEMU/kvm has caused that have been fixed.

estibi commented 12 years ago

illumos-gate debug build r13629

pstack 100563

100563: qemu-kvm -enable-kvm -nographic -smp 2 -m 8192 -vnc 0.0.0.0:52 -no-hpe ----------------- lwp# 1 / thread# 1 -------------------- fffffd7fff2b3e2a pollsys (fffffd7fffdf9660, 4, fffffd7fffdf9760, 0) fffffd7fff23c504 pselect () + 18c fffffd7fff23ca18 select () + 70 00000000004bcf1f main_loop_wait () + 19f 00000000004cf70b kvm_main_loop () + bb 00000000004be406 main () + 9a6 00000000004b031c _start () + 6c ----------------- lwp# 2 / thread# 2 -------------------- fffffd7fff2b349a sigtimedwait (f75cc0, fffffd7ffee7ecb0, 0) fffffd7fff29a144 sigwaitinfo () + c 00000000004db088 sigwait_compat () + 48 fffffd7fff2aaf24 _thrp_setup () + bc fffffd7fff2ab1f0 _lwp_start () ----------------- lwp# 3 / thread# 3 -------------------- fffffd7fff2b3baa ioctl (b, 2000ae80, 0) 00000000004cecd9 kvm_cpu_exec () + 9 00000000004cf33a ap_main_loop () + 22a fffffd7fff2aaf24 _thrp_setup () + bc fffffd7fff2ab1f0 _lwp_start () ----------------- lwp# 4 / thread# 4 -------------------- fffffd7fff2b3baa ioctl () + a 00000000004cecd9 kvm_cpu_exec () + 9 00000000004cf33a ap_main_loop () + 22a fffffd7fff2aaf24 _thrp_setup () + bc fffffd7fff2ab1f0 _lwp_start () ----------------- lwp# 5 / thread# 5 -------------------- fffffd7fff2b349a sigtimedwait (2011b4ca0, fffffd7ffe540cb0, 0) fffffd7fff29a144 sigwaitinfo () + c 00000000004db088 sigwait_compat () + 48 fffffd7fff2aaf24 _thrp_setup () + bc fffffd7fff2ab1f0 _lwp_start ()

100563 root 0.0 100 0.0 0.0 0.0 0.0 0.0 0.0 2 17 19 0 qemu-kvm-sys/4 100567 root 0.0 4.0 0.0 0.0 0.0 0.0 95 0.6 9K 12 1K 0 qemu-kvm-sys/3 100564 root 0.0 3.6 0.0 0.0 0.0 0.0 96 0.7 9K 1 829 0 qemu-kvm-sys/3 100567 root 0.0 2.1 0.0 0.0 0.0 0.0 97 0.6 5K 17 17 0 qemu-kvm-sys/5 100563 root 0.0 1.8 0.0 0.0 0.0 0.0 98 0.6 9K 0 0 0 qemu-kvm-sys/3 100567 root 0.0 1.4 0.0 0.0 0.0 0.0 98 0.2 5K 11 11 0 qemu-kvm-sys/6 100567 root 0.0 1.3 0.0 0.0 0.0 0.0 98 0.2 5K 13 73 6 qemu-kvm-sys/4 100564 root 0.0 1.3 0.0 0.0 0.0 0.0 99 0.2 5K 0 0 0 qemu-kvm-sys/4 100567 root 0.1 0.1 0.0 0.0 0.0 0.0 100 0.0 181 0 1K 0 qemu-kvm-sys/1 100563 root 0.1 0.1 0.0 0.0 0.0 0.0 100 0.0 177 4 1K 0 qemu-kvm-sys/1 100816 root 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 1 1 60 0 intrd/1 102018 root 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 31 0 428 0 prstat/1

::stacks -v mdb: stacks: processing kernel threads mdb: stacks: 111 unique stacks / 1492 threads mdb: stacks: done THREAD STATE SOBJ COUNT ffffff0170011c40 SLEEP CV 550 swtch+0x1e6 cv_wait+0x7f taskq_thread_wait+0x94 taskq_thread+0x324 thread_start+8

ffffff345df4cc40 SLEEP CV 165 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c lwp_park+0x157 syslwp_park+0x31 _sys_sysenter_post_swapgs+0x237

ffffff017009bc40 FREE 119

ffffff01719b7c40 SLEEP CV 79 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c taskq_thread_wait+0x84 taskq_d_thread+0x1b7 thread_start+8 ffffff346fab1860 SLEEP CV 43 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_waituntil_sig+0xba nanosleep+0x120 _sys_sysenter_post_swapgs+0x237 ffffff0171aabc40 SLEEP CV 42 swtch+0x1e6 cv_wait+0x7f mac_soft_ring_worker+0xc5 thread_start+8 ffffff3468ba8440 SLEEP SHUTTLE 36 swtch_to+0xf0 shuttle_resume+0x325 door_return+0x236 doorfs32+0x134 sys_syscall32+0x1f7 ffffff34605a3040 SLEEP SHUTTLE 29 swtch+0x1e6 shuttle_swtch+0x256 door_return+0x25e doorfs32+0x134 _sys_sysenter_post_swapgs+0x237 ffffff0171a9fc40 SLEEP CV 28 swtch+0x1e6 cv_wait+0x7f mac_srs_worker+0x1f7 thread_start+8 ffffff34600e60a0 SLEEP SHUTTLE 28 swtch_to+0xf0 shuttle_resume+0x325 door_return+0x236 doorfs32+0x134 _sys_sysenter_post_swapgs+0x237 ffffff3464e2d7c0 SLEEP CV 26 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_relwaituntil_sig+0x2c poll_common+0x4c1 pollsys+0xea _sys_sysenter_post_swapgs+0x237 ffffff017059ec40 SLEEP CV 23 swtch+0x1e6 cv_wait+0x7f evch_delivery_hold+0x92 evch_delivery_thr+0x260 thread_start+8 ffffff0171415c40 SLEEP CV 19 swtch+0x1e6 cv_wait+0x7f i_mac_notify_thread+0xdc thread_start+8 ffffff0171847c40 SLEEP CV 18 swtch+0x1e6 cv_wait+0x7f squeue_polling_thread+0xa1 thread_start+8 ffffff0171841c40 SLEEP CV 18 swtch+0x1e6 cv_wait+0x7f squeue_worker+0xd1 thread_start+8 ffffff0171853c40 SLEEP CV 16 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c kcfpool_svc+0x83 thread_start+8 ffffff01705aac40 ONPROC 16 swtch+0x1e6 cpu_pause+0xb0 thread_start+8 ffffff3460285060 SLEEP SHUTTLE 14 swtch+0x1e6 shuttle_swtch+0x256 door_return+0x25e doorfs32+0x134 sys_syscall32+0x1f7 ffffff346756d7e0 SLEEP CV 11 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c lwp_park+0x157 syslwp_park+0x31 sys_syscall32+0x1f7 ffffff348d69c0c0 SLEEP CV 10 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_reltimedwait_sig+0x77 cv_relwaituntil_sig+0x25 poll_common+0x4c1 pollsys+0xea _sys_sysenter_post_swapgs+0x237 ffffff3464e26080 SLEEP CV 10 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 pause+0x48 _sys_sysenter_post_swapgs+0x237 ffffff01701c7c40 SLEEP CV 8 swtch+0x1e6 cv_wait+0x7f mpt_doneq_thread+0x62 thread_start+8 ffffff017020fc40 SLEEP CV 8 swtch+0x1e6 cv_wait+0x7f mptsas_doneq_thread+0x60 thread_start+8 ffffff01702f8c40 FREE 7 apic_setspl+0x5c do_splx+0x82 cyclic_coverage+0x3b cyclic_softint+0x4fc dosoftint_epilog+0x10b dispatch_softint+0x3f ffffff3464e26800 SLEEP CV 7 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_reltimedwait_sig+0x77 cv_relwaituntil_sig+0x25 poll_common+0x5e0 pollsys+0xea _sys_sysenter_post_swapgs+0x237 ffffff345fed9800 SLEEP CV 7 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_waituntil_sig+0xba lwp_park+0x157 syslwp_park+0x31 _sys_sysenter_post_swapgs+0x237 ffffff01700cbc40 FREE 6 cyclic_coverage+0x3b cyclic_softint+0x4fc dosoftint_epilog+0x10b dispatch_softint+0x3f ffffff348e9370a0 SLEEP CV 6 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c sigtimedwait+0x1eb ffffff348e685c00 SLEEP CV 6 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 kvm_vcpu_block+0x6b __vcpu_run+0x84 kvm_arch_vcpu_ioctl_run+0x110 kvm_ioctl+0x119 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e ffffff345fdaa4a0 SLEEP CV 6 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 sigsuspend+0x107 _sys_sysenter_post_swapgs+0x237 ffffff0170005c40 ONPROC 6 cpu_idle_dtrace_enter+0x24 0x815 acpi_cpu_cstate+0x223 cpu_acpi_idle+0x97 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff348e67a780 SLEEP CV 4 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_reltimedwait_sig+0x77 cv_relwaituntil_sig+0x25 poll_common+0x4c1 pollsys+0xea ffffff34605b73e0 SLEEP CV 4 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_waituntil_sig+0xba lwp_park+0x157 syslwp_park+0x31 sys_syscall32+0x1f7 ffffff34668627c0 SLEEP CV 4 swtch+0x1e6 cv_wait_sig+0x181 cte_get_event+0xdd ctfs_endpoint_ioctl+0xa7 ctfs_bu_ioctl+0x51 fop_ioctl+0x7b ioctl+0x18e sys_syscall32+0x1f7 ffffff345fdbc0a0 SLEEP CV 4 swtch+0x1e6 cv_wait_sig+0x181 door_unref+0x89 doorfs32+0x82 _sys_sysenter_post_swapgs+0x237 ffffff345d113be0 SLEEP CV 4 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c sigtimedwait+0x1eb _sys_sysenter_post_swapgs+0x237 ffffff34685d3840 SLEEP CV 4 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 waitid+0x2a0 waitsys32+0x30 _sys_sysenter_post_swapgs+0x237 ffffff01702f2c40 FREE 3 apic_send_ipi+0x92 send_dirint+0x19 cbe_xcall+0xde cyclic_reprogram_here+0x67 cyclic_reprogram+0x14f apic_setspl+0x5c dosoftint_epilog+0x10b dispatch_softint+0x3f ffffff346067bb40 SLEEP CV 3 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 fifo_read+0xf5 fop_read+0x6b read+0x2b8 read32+0x22 _sys_sysenter_post_swapgs+0x237 ffffff3468ba3820 SLEEP CV 3 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 pause+0x48 sys_syscall32+0x1f7 ffffff3468a820c0 SLEEP CV 3 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 pause+0x75 _sys_sysenter_post_swapgs+0x237 ffffff01700c5c40 FREE 2 igb_tx_recycle_head_wb+0x177 apic_intr_exit+0x32 intr_thread_epilog+0x110 dispatch_hardint+0x41 ffffff01708c5c40 SLEEP CV 2 swtch+0x1e6 cv_timedwait_hires+0xee cv_timedwait+0x5a txg_thread_wait+0x8d txg_sync_thread+0x149 thread_start+8 ffffff0172a4dc40 SLEEP CV 2 swtch+0x1e6 cv_timedwait_hires+0xee cv_timedwait+0x5a zone_status_timedwait+0x5f auto_do_unmount+0xcb thread_start+8 ffffff34a60eb880 SLEEP CV 2 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_reltimedwait_sig+0x77 cv_relwaituntil_sig+0x25 poll_common+0x4c1 pollsys+0xea sys_syscall32+0x1f7 ffffff348e945440 SLEEP CV 2 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_waituntil_sig+0xba lwp_park+0x157 syslwp_park+0x31 ffffff34603d7020 SLEEP CV 2 swtch+0x1e6 cv_timedwait_sig_hires+0x206 cv_waituntil_sig+0xba nanosleep+0x120 ffffff017090dc40 SLEEP CV 2 swtch+0x1e6 cv_wait+0x7f ill_taskq_dispatch+0x139 thread_start+8 ffffff01708fbc40 SLEEP CV 2 swtch+0x1e6 cv_wait+0x7f ipsec_loader+0xb2 thread_start+8 ffffff01719b1c40 SLEEP CV 2 swtch+0x1e6 cv_wait+0x7f progressbar_key_abort_thread+0x56 thread_start+8 ffffff01702b0c40 SLEEP CV 2 swtch+0x1e6 cv_wait+0x7f spa_thread+0x23b thread_start+8 ffffff01708bcc40 SLEEP CV 2 swtch+0x1e6 cv_wait+0x7f txg_thread_wait+0x70 txg_quiesce_thread+0x94 thread_start+8 ffffff346756dba0 SLEEP CV 2 swtch+0x1e6 cv_wait_sig+0x181 door_unref+0x89 doorfs32+0x82 sys_syscall32+0x1f7 ffffff348e279c40 SLEEP CV 2 swtch+0x1e6 cv_wait_sig+0x181 so_dequeue_msg+0x2bb so_recvmsg+0x1ca socket_recvmsg+0x3d socket_vop_read+0x58 fop_read+0x6b read+0x2b8 read32+0x22 _sys_sysenter_post_swapgs+0x237 ffffff348e705420 SLEEP CV 2 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c port_getn+0x23b portfs+0x282 portfs32+0x57 sys_syscall32+0x1f7 ffffff3466862040 SLEEP CV 2 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c sigtimedwait+0x1eb sys_syscall32+0x1f7 ffffff34687663e0 SLEEP CV 2 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 sigsuspend+0x125 sys_syscall32+0x1f7 ffffff346f708800 SLEEP SHUTTLE 2 swtch_to+0xf0 shuttle_resume+0x325 door_call+0x2f1 doorfs32+0x141 sys_syscall32+0x1f7 ffffff0170cc8c40 ONPROC 2 0x100f600fb877a6d cpu_idle_mwait+0xf1 cpu_acpi_idle+0xa2 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff01709a8c40 ONPROC 2 swtch+0x1e6 idle+0xc4 thread_start+8 ffffff0170afac40 FREE 1 0xffffff3430930780 cyclic_coverage+0x3b cyclic_softint+0x4fc dosoftint_epilog+0x10b dispatch_softint+0x3f ffffff0170e0bc40 FREE 1 0xffffff3430931080 cyclic_coverage+0x3b cyclic_softint+0x4fc dosoftint_epilog+0x10b dispatch_softint+0x3f ffffff0170bbdc40 FREE 1 0xffffff3470965300 sd_mapblockaddr_iodone+0xee sd_return_command+0x2a2 sdintr+0x82d scsi_hba_pkt_comp+0x179 vhci_intr+0x831 mptsas_pkt_comp+0x47 mptsas_doneq_empty+0xbb mptsas_intr+0x3cd intr_thread_epilog+0x110 dispatch_hardint+0x41 ffffff0170feec40 FREE 1 igb_check_acc_handle+0x33 igb_rx+0x23d apic_intr_exit+0x32 intr_thread_epilog+0x110 dispatch_hardint+0x41 ffffff0170f18c40 FREE 1 sd_media_watch_cb+0x299 scsi_watch_request_intr+0x353 atapi_complete+0x124 ata_hba_complete+0x38 ghd_doneq_process+0xb3 0x2fb8db0e1 dispatch_softint+0x3f ffffff0170d63c40 FREE 1 usb_dprintf4+0xae apic_intr_exit+0x60 intr_thread_epilog+0x110 dispatch_hardint+0x41 ffffff0170eb3c40 FREE 1 usb_dprintf4+0xae uhci_get_sw_frame_number+0x9e apic_send_directed_EOI+0x59 apic_intr_exit+0x60 intr_thread_epilog+0x110 dispatch_hardint+0x41 ffffff017171ac40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c idm_wd_thread+0x1d7 thread_start+8 ffffff0170185c40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c kcfpoold+0x107 thread_start+8 ffffff01711c1c40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c mod_uninstall_daemon+0xd3 thread_start+8 ffffff017017fc40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c page_capture_thread+0xf0 thread_start+8 ffffff0172404c40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c scsi_watch_thread+0x45b thread_start+8 ffffff01711c7c40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c seg_pasync_thread+0xdc thread_start+8 ffffff0171793c40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_reltimedwait+0x6c stmf_svc_timeout+0x2be stmf_svc+0x168 taskq_thread+0x29e thread_start+8 ffffff017018bc40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_timedwait+0x5a arc_reclaim_thread+0x14e thread_start+8 ffffff0170191c40 SLEEP CV 1 swtch+0x1e6 cv_timedwait_hires+0xee cv_timedwait+0x5a l2arc_feed_thread+0xb2 thread_start+8 ffffff0172ad8c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f crypto_bufcall_service+0x189 thread_start+8 ffffff01708e3c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f dld_taskq_dispatch+0x10d thread_start+8 ffffff01711bbc40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f fsflush+0x21d thread_start+8 ffffff0171525c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f ibcm_process_tlist+0x1a6 thread_start+8 ffffff0171e13c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f log_event_deliver+0x245 thread_start+8 ffffff01711afc40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f pageout+0x27f thread_start+8 ffffff01711b5c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f pageout_scanner+0x146 thread_start+8 ffffff0170197c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f pm_dep_thread+0xa6 thread_start+8 ffffff0170023c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f scsi_hba_barrier_daemon+0xb0 thread_start+8 ffffff0170029c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f scsi_lunchg1_daemon+0x23e thread_start+8 ffffff017002fc40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f scsi_lunchg2_daemon+0x12d thread_start+8 ffffff0170f4dc40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f softmac_taskq_dispatch+0x115 thread_start+8 ffffff017016dc40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f streams_bufcall_service+0x17d thread_start+8 ffffff0170173c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f streams_qbkgrnd_service+0xc4 thread_start+8 ffffff0170179c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f streams_sqbkgrnd_service+0xaa thread_start+8 ffffff0170598c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f task_commit+0xaa thread_start+8 ffffff017000bc40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f thread_reaper+0xc6 thread_start+8 ffffff01700efc40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f timeout_taskq_thread+0xbc thread_start+8 ffffff0171ed4c40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f ufs_thread_idle+0x163 thread_start+8 ffffff0171ecec40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f ufs_thread_run+0x15d ufs_thread_hlock+0x6f thread_start+8 ffffff01723fec40 SLEEP CV 1 swtch+0x1e6 cv_wait+0x7f zone_status_wait_cpr+0xcb zsched+0x7b5 thread_start+8 ffffff34602a2400 SLEEP CV 1 swtch+0x1e6 cv_wait_sig+0x181 sd_check_media+0x162 sdioctl+0x9d8 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e _sys_sysenter_post_swapgs+0x237 ffffff345fdb60c0 SLEEP CV 1 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 cv_relwaituntil_sig+0x2c poll_common+0x4c1 pollsys+0xea sys_syscall32+0x1f7 ffffff34605a3b80 SLEEP CV 1 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 pause+0x75 sys_syscall32+0x1f7 ffffff349ad68b80 SLEEP SHUTTLE 1 swtch_to+0xf0 shuttle_resume+0x325 door_call+0x2f1 doorfs32+0x141 _sys_sysenter_post_swapgs+0x237 ffffff01705b6c40 ONPROC 1 0x100ea00fb877a6d cpu_idle_mwait+0xf1 cpu_acpi_idle+0xa2 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff34600d24a0 ONPROC 1 apic_find_highest_isr+0x1c apic_find_highest_isr+0x1c apic_update_ppr+0x55 0x6c14 vcpu_enter_guest+0x33a __vcpu_run+0x77 kvm_arch_vcpu_ioctl_run+0x110 kvm_ioctl+0x119 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e ffffff0170bf2c40 ONPROC 1 apic_setspl+0x5c dosoftint_prolog+0x99 dosoftint+0x59 do_interrupt+0x15a tsc_gethrtimeunscaled+0x16 0x100f300fb877a6d cpu_idle_mwait+0xf1 cpu_acpi_idle+0xa2 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170ee8c40 ONPROC 1 apic_setspl+0x5c dosoftint_prolog+0x99 dosoftint+0x59 do_interrupt+0x15a tsc_gethrtimeunscaled+0x16 0x100fc00fb877a6d cpu_idle_mwait+0xf1 cpu_acpi_idle+0xa2 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff348e685840 ONPROC 1 swtch+0x1e6 cv_wait_sig_swap_core+0x183 cv_wait_sig_swap+0x18 kvm_vcpu_block+0x6b __vcpu_run+0x84 kvm_arch_vcpu_ioctl_run+0x110 kvm_ioctl+0x119 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e ffffff3464e2a060 ONPROC 1 x86pte_mapin+0x49 x86pte_release_pagetable+0x20 x86pte_get+0x3b uiomove+0xe9 mmio+0x145 mmrw+0x1a3 mmread+0x22 cdev_read+0x3d spec_read+0x279 fop_read+0x6b pread+0x22e ffffff0170b87c40 ONPROC 1 xc_serv+0x186 0xffffff34647ec580 apic_setspl+0x5c dosoftint_prolog+0x99 dosoftint+0x59 do_interrupt+0x15a tsc_gethrtimeunscaled+0x16 0x100f000fb877a6d cpu_idle_mwait+0xf1 cpu_acpi_idle+0xa2 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff017035ec40 ONPROC 1 xc_serv+0x186 apic_intr_exit+0x32 apic_setspl+0x5c dosoftint_prolog+0x99 dosoftint_prolog+0x99 dosoftint+0x59 do_interrupt+0x15a tsc_gethrtimeunscaled+0x16 0x100ea00fb877a6d cpu_idle_mwait+0xf1 cpu_acpi_idle+0xa2 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170c5dc40 ONPROC 1 xc_serv+0x186 apic_intr_exit+0x32 tsc_gethrtimeunscaled+0x16 apic_setspl+0x5c dosoftint_prolog+0x99 dosoftint+0x59 do_interrupt+0x15a tsc_gethrtimeunscaled+0x16 0x100f300fb877a6d cpu_idle_mwait+0xf1 cpu_acpi_idle+0xa2 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 fffffffffbc3d1e0 STOPPED 1 swtch+0x1e6 sched+0x446 main+0x4d2
estibi commented 12 years ago

::ps ! grep 100563 R 100563 1 100561 100561 0 0x42000000 ffffff34964250d8 qemu-kvm-system- ffffff34964250d8::walk thread | ::findstack -v stack pointer for thread ffffff34640ff860: ffffff01723b6c50 [ ffffff01723b6c50 _resume_from_idle+0xf1() ] ffffff01723b6c80 swtch+0x1e6() ffffff01723b6d10 cv_timedwait_sig_hires+0x206(ffffff3490f75b42, ffffff3490f75b08, 3b9aca00, f4240, 0) ffffff01723b6d70 cv_reltimedwait_sig+0x77(ffffff3490f75b42, ffffff3490f75b08, 64, 2) ffffff01723b6da0 cv_relwaituntil_sig+0x25(ffffff3490f75b42, ffffff3490f75b08, ffffff01723b6e08, 2) ffffff01723b6e60 poll_common+0x4c1(fffffd7fffdf9660, 4, ffffff01723b6e80, 0) ffffff01723b6ef0 pollsys+0xea(fffffd7fffdf9660, 4, fffffd7fffdf9760, 0) ffffff01723b6f00 sys_syscall+0x26e() stack pointer for thread ffffff3464343c00: ffffff017232ecd0 [ ffffff017232ecd0 _resume_from_idle+0xf1() ] ffffff017232ed00 swtch+0x1e6() ffffff017232ed60 cv_wait_sig_swap_core+0x183(ffffff3464343dee, ffffff3430528c40, 0) ffffff017232ed80 cv_wait_sig_swap+0x18(ffffff3464343dee, ffffff3430528c40) ffffff017232edf0 cv_waituntil_sig+0x13c(ffffff3464343dee, ffffff3430528c40, 0, 0) ffffff017232eef0 sigtimedwait+0x204(f75cc0, fffffd7ffee7ecb0, 0) ffffff017232ef00 sys_syscall+0x26e() stack pointer for thread ffffff346fab30c0: ffffff01723bc3a0 [ ffffff01723bc3a0 _resume_from_idle+0xf1() ] ffffff01723bc3d0 swtch+0x1e6() ffffff01723bc430 cv_wait_sig_swap_core+0x183(ffffff3496f57050, ffffff3496f57048, 0) ffffff01723bc450 cv_wait_sig_swap+0x18(ffffff3496f57050, ffffff3496f57048) ffffff01723bc480 kvm_vcpu_block+0x6b() ffffff01723bc4a0 __vcpu_run+0x84() ffffff01723bc4e0 kvm_arch_vcpu_ioctl_run+0x110() ffffff01723bccf0 kvm_ioctl+0x119() ffffff01723bcd30 cdev_ioctl+0x45(10e00000004, 2000ae80, 0, 202003, ffffff34936d7d28, ffffff01723bce14) ffffff01723bcd70 spec_ioctl+0x5a(ffffff34936d3000, 2000ae80, 0, 202003, ffffff34936d7d28, ffffff01723bce14, 0) ffffff01723bcdf0 fop_ioctl+0x7b(ffffff34936d3000, 2000ae80, 0, 202003, ffffff34936d7d28, ffffff01723bce14, 0) ffffff01723bcef0 ioctl+0x18e(b, 2000ae80, 0) ffffff01723bcf00 sys_syscall+0x26e() stack pointer for thread ffffff34600d24a0: ffffff01722dc370 [ ffffff01722dc370 _resume_from_idle+0xf1() ] ffffff01722dc480 vcpu_enter_guest+0x35e() ffffff01722dc4a0 __vcpu_run+0x77() ffffff01722dc4e0 kvm_arch_vcpu_ioctl_run+0x110() ffffff01722dccf0 kvm_ioctl+0x119() ffffff01722dcd30 cdev_ioctl+0x45(10e00000005, 2000ae80, 0, 202003, ffffff34936d7d28, ffffff01722dce14) ffffff01722dcd70 spec_ioctl+0x5a(ffffff3496a90600, 2000ae80, 0, 202003, ffffff34936d7d28, ffffff01722dce14, 0) ffffff01722dcdf0 fop_ioctl+0x7b(ffffff3496a90600, 2000ae80, 0, 202003, ffffff34936d7d28, ffffff01722dce14, 0) ffffff01722dcef0 ioctl+0x18e(c, 2000ae80, 0) ffffff01722dcf00 sys_syscall+0x26e() stack pointer for thread ffffff345fdb6c00: ffffff017242ecd0 [ ffffff017242ecd0 _resume_from_idle+0xf1() ] ffffff017242ed00 swtch+0x1e6() ffffff017242ed60 cv_wait_sig_swap_core+0x183(ffffff345fdb6dee, ffffff3430528c40, 0) ffffff017242ed80 cv_wait_sig_swap+0x18(ffffff345fdb6dee, ffffff3430528c40) ffffff017242edf0 cv_waituntil_sig+0x13c(ffffff345fdb6dee, ffffff3430528c40, 0, 0) ffffff017242eef0 sigtimedwait+0x1eb(2011b4ca0, fffffd7ffe540cb0, 0) ffffff017242ef00 sys_syscall+0x26e()

rmustacc commented 12 years ago

From the kernel stacks, it looks like you have threads which are making forward progress and are running the VM. Also, why do you have the -no-hpe option, are you trying to toggle it to be -no-hpet? If so, that shouldn't be necessary with the newest bits. As an added sanity check, what does kvmstat say is going on and does vnc show that the guest hasn't paniced?

estibi commented 12 years ago

Updates:

illumos-gate r13632 non debug

illumos-kvm 76be9d4 illumos-kvm-cmd 4f48eea

I have multiple kvm instances here.

PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 13540 root 0.2 100 0.0 0.0 0.0 0.0 0.0 0.0 0 0 5 0 qemu-kvm-sys/3 17412 root 26 63 0.0 0.0 0.0 0.0 11 0.0 48 0 806 0 prstat/1 394 root 3.6 6.4 0.0 0.0 0.0 0.0 90 0.0 48 0 288 0 nscd/6 16972 root 0.0 5.2 0.0 0.0 0.0 0.0 94 0.5 10 0 0 0 qemu-kvm-sys/3 16997 root 0.0 4.0 0.0 0.0 0.0 0.0 95 0.6 5 0 0 0 qemu-kvm-sys/3 17000 root 0.0 3.6 0.0 0.0 0.0 0.0 96 0.5 10 0 0 0 qemu-kvm-sys/3 12424 root 0.0 3.6 0.0 0.0 0.0 0.0 96 0.4 10 0 0 0 qemu-kvm-sys/3

As you can see 13540 consumes 100% SYS CPU time, but this VM actually works fine.. and it's idle.

pargs -l 13540

/usr/bin/amd64/qemu-kvm-system-x86_64 -enable-kvm -nographic -smp 1 -m 8192 -vnc 0.0.0.0:23 -no-hpet -no-acpi -net 'nic,vlan=0,name=net0,macaddr=2:8:20:7b:c8:6b,model=virtio' -net 'vnic,vlan=0,name=XXXvm03,ifname=XXXvm03,macaddr=2:8:20:7b:c8:6b' -drive 'file=/dev/zvol/rdsk/data/XXXvm03,index=0,media=disk,if=virtio' -cpu qemu64 -daemonize

Note that I replaced here the disk and NIC names.

This one hung at boot:

pargs -l 16991

/usr/bin/amd64/qemu-kvm-system-x86_64 -enable-kvm -nographic -smp 1 -m 10240 -vnc 0.0.0.0:51 -no-hpet -no-acpi -net 'nic,vlan=0,name=net0,macaddr=2:8:20:ec:65:7b,model=virtio' -net 'vnic,vlan=0,name=XXXvm01,ifname=XXXvm01,macaddr=2:8:20:ec:65:7b' -drive 'file=/dev/zvol/rdsk/data/XXXvm01,index=0,media=disk,if=virtio' -daemonize

kvmstat

pid vcpu | exits : haltx irqx irqwx iox mmiox | irqs emul eptv 12025 0 | 7035 : 2000 35 0 3000 0 | 2000 2000 0 12424 0 | 7015 : 1999 15 1 3000 0 | 2000 2000 0 12428 0 | 7037 : 2000 35 0 3001 0 | 2001 2001 0 13540 0 | 7017 : 1855 12 146 3003 0 | 2001 2001 0 14057 0 | 7050 : 1839 51 160 3000 0 | 2000 2000 0 14063 0 | 7025 : 2000 25 0 3000 0 | 2000 2000 0 16972 0 | 7009 : 1701 4 300 3003 0 | 2001 2001 0 16985 0 | 7071 : 2000 48 1 3005 0 | 2002 2002 8 16991 0 | 2008 : 1000 8 0 0 0 | 1000 1000 0 16994 0 | 7033 : 2003 17 0 3009 0 | 2004 2004 0 16997 0 | 7014 : 2000 14 0 3000 0 | 2000 2000 0 17000 0 | 7050 : 2003 22 0 3009 0 | 2004 2004 6 pid vcpu | exits : haltx irqx irqwx iox mmiox | irqs emul eptv 12025 0 | 7031 : 2001 24 0 3003 0 | 2001 2001 0 12424 0 | 7015 : 2000 15 0 3000 0 | 2000 2000 0 12428 0 | 7020 : 1694 17 307 3002 0 | 2000 2000 0 13540 0 | 7069 : 1879 69 121 3000 0 | 2000 2000 0 14057 0 | 7001 : 1999 1 1 3000 0 | 2000 2000 0 14063 0 | 7012 : 2000 12 0 3000 0 | 2000 2000 0 16972 0 | 7032 : 1999 16 1 3000 0 | 2000 2000 9 16985 0 | 7029 : 2002 16 0 3008 0 | 2003 2003 0 16991 0 | 2003 : 1000 3 0 0 0 | 1000 1000 0 16994 0 | 7015 : 2001 3 1 3007 0 | 2003 2003 0 16997 0 | 7014 : 2000 14 0 3000 0 | 2000 2000 0 17000 0 | 7032 : 2002 20 0 3007 0 | 2003 2003 0 pid vcpu | exits : haltx irqx irqwx iox mmiox | irqs emul eptv 12025 0 | 7114 : 2000 114 0 3000 0 | 2000 2000 0 12424 0 | 7013 : 1999 11 0 3000 0 | 2000 2000 1 12428 0 | 7005 : 2000 5 0 3000 0 | 2000 2000 0 13540 0 | 7010 : 2000 8 0 3000 0 | 2000 2000 0 14057 0 | 6980 : 1963 16 19 3000 0 | 1982 1982 0 14063 0 | 7061 : 1841 61 155 3003 0 | 2001 2001 0 16972 0 | 7005 : 2000 5 0 3000 0 | 2000 2000 0 16985 0 | 7124 : 1597 117 403 3005 0 | 2002 2002 0 16991 0 | 2005 : 1000 5 0 0 0 | 1000 1000 0 16994 0 | 7020 : 2002 4 1 3009 0 | 2004 2004 0 16997 0 | 7008 : 2000 8 0 3000 0 | 2000 2000 0 17000 0 | 7064 : 2003 48 0 3009 0 | 2004 2004 0 pid vcpu | exits : haltx irqx irqwx iox mmiox | irqs emul eptv 12025 0 | 7020 : 2000 20 0 3000 0 | 2000 2000 0 12424 0 | 7026 : 2001 17 0 3006 0 | 2002 2002 0 12428 0 | 7007 : 2000 7 0 3000 0 | 2000 2000 0 13540 0 | 7023 : 2000 23 0 3000 0 | 2000 2000 0 14057 0 | 7031 : 2000 31 0 3000 0 | 2000 2000 0 14063 0 | 7133 : 1698 136 299 3000 0 | 2000 2000 0 16972 0 | 7006 : 2000 6 0 3000 0 | 2000 2000 0 16985 0 | 7027 : 2001 19 0 3005 0 | 2002 2002 0 16991 0 | 2004 : 1000 4 0 0 0 | 1000 1000 0 16994 0 | 7015 : 2001 3 1 3007 0 | 2003 2003 0 16997 0 | 7008 : 2000 8 0 3000 0 | 2000 2000 0 17000 0 | 7032 : 2002 20 0 3007 0 | 2003 2003 0 ^C

mpstat 1 5

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 25 0 20 468 157 2467 1 37 75 0 415 0 4 0 96 1 14 0 16 2275 1111 1787 2 38 70 0 542 0 5 0 95 2 9 0 15 4040 1997 2252 2 27 64 0 299 0 5 0 95 3 9 0 16 2188 1068 2510 2 35 72 0 539 0 4 0 96 4 12 0 37 3980 1961 4479 7 43 67 0 414 0 10 0 90 5 9 0 33 2300 1123 2801 5 37 67 0 542 0 18 0 81 6 9 0 38 4228 2213 4470 5 36 111 0 468 0 9 0 91 7 10 0 35 7795 3864 5816 7 48 91 0 481 0 11 0 89 8 8 0 16 3996 1978 2589 3 29 65 0 331 0 5 0 95 9 6 0 17 3901 1923 2225 2 25 60 0 272 0 4 0 96 10 7 0 15 1954 958 2585 3 29 67 0 351 0 4 0 96 11 7 0 11 535 191 1308 1 20 58 0 275 0 2 0 98 12 8 0 34 2180 1066 3537 4 36 76 0 596 0 12 0 87 13 8 0 28 4146 2045 2861 4 30 63 0 435 0 20 0 80 14 9 0 38 4174 2057 4033 8 47 78 0 574 0 10 0 90 15 8 0 27 2033 989 4112 6 35 62 0 338 0 10 0 90 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 13 6131 3005 2162 0 50 4 0 720 0 4 0 96 1 0 0 12 4094 2026 1662 1 62 9 0 828 0 2 0 98 2 0 0 13 65 15 2953 2 35 6 0 102 0 8 0 92 3 0 0 13 2196 1068 1928 7 61 9 0 369 0 3 0 97 4 0 0 67 4103 1986 3340 22 106 0 0 730 1 8 0 91 5 0 0 0 2132 1039 0 0 0 0 0 0 0 100 0 0 6 0 0 58 306 185 4246 20 96 7 0 393 0 19 0 81 7 0 0 30 6102 3003 2932 12 55 1 0 437 0 47 0 53 8 0 0 25 240 88 1452 4 51 22 0 284 1 2 0 97 9 1 0 17 174 52 1550 1 40 3 0 589 1 2 0 97 10 0 0 12 2035 1005 3347 0 40 8 0 288 0 5 0 95 11 0 0 12 2036 1003 3384 3 30 1 0 196 1 4 0 95 12 0 0 74 4227 2038 4032 18 98 1 0 567 0 45 0 55 13 0 0 97 6153 3003 4104 20 137 29 0 372 0 16 0 84 14 0 0 48 4123 1994 2564 18 77 0 0 557 0 52 0 48 15 0 0 44 4078 1989 2331 8 64 31 0 362 0 47 0 53 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 22 6138 3007 2969 0 49 16 0 109 0 5 0 95 1 0 0 23 4040 1989 2497 4 81 18 0 1053 1 4 0 95 2 0 0 13 77 19 1947 3 44 9 0 183 0 7 0 93 3 0 0 25 2201 1078 1594 2 50 21 0 538 0 3 0 97 4 0 0 102 4298 2025 5394 29 140 3 0 517 0 6 0 94 5 0 0 8 2157 1055 878 3 17 0 0 171 0 77 0 23 6 0 0 30 268 166 4332 12 59 24 0 321 0 6 0 94 7 0 0 37 6010 2972 4083 8 64 1 0 242 1 8 0 91 8 0 0 12 263 97 1456 3 56 18 0 457 0 3 0 97 9 4 0 9 377 73 925 0 37 6 0 692 0 1 0 99 10 0 0 21 2029 994 3615 2 51 15 0 600 1 4 0 95 11 0 0 17 2047 1002 3687 3 32 13 0 156 0 4 0 96 12 0 0 62 4271 2073 4474 23 116 3 0 729 0 5 0 95 13 0 0 79 6192 3004 4393 14 116 5 0 621 0 28 0 72 14 0 0 20 4145 2019 1092 18 30 7 0 536 0 80 0 20 15 0 0 0 4106 2008 0 5 0 1 0 461 1 99 0 0 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 19 6129 2993 2112 3 64 179 0 593 1 4 0 95 1 0 0 29 4076 2007 3363 5 93 146 0 501 0 5 0 95 2 0 0 15 77 20 2947 4 55 127 0 125 0 9 0 91 3 0 0 10 2170 1054 1083 7 74 182 0 575 0 2 0 98 4 0 0 53 4097 1975 4537 23 108 108 0 545 0 6 0 94 5 0 0 41 2230 1062 3627 14 101 98 0 832 0 4 0 96 6 0 0 23 392 333 4240 15 53 174 0 188 0 7 0 93 7 0 0 34 6063 3002 3447 11 60 60 0 98 0 7 0 93 8 0 0 12 248 82 1294 5 68 136 0 698 0 1 0 99 9 0 0 8 282 68 1055 1 48 66 0 574 0 2 0 98 10 0 0 18 2064 1013 3544 1 49 93 0 145 0 4 0 96 11 0 0 19 2039 1002 3551 4 48 98 0 90 0 5 0 95 12 0 0 39 4182 2047 3941 16 91 102 0 679 0 5 0 95 13 0 0 45 6103 3001 4857 14 99 103 0 568 0 5 0 95 14 0 0 0 4115 2014 0 6 0 1 0 461 0 100 0 0 15 0 0 0 4102 2008 0 6 0 0 0 441 0 100 0 0 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 18 6039 2950 2299 5 51 43 0 255 0 4 0 96 1 0 0 25 4104 2025 2997 3 60 3 0 569 0 3 0 97 2 0 0 9 87 24 1742 2 33 2 0 274 0 6 0 94 3 0 0 13 2237 1081 1213 2 57 1 0 514 1 1 0 98 4 0 0 79 4184 2024 4798 16 118 0 0 484 0 5 0 95 5 0 0 30 2179 1060 4087 12 69 6 0 752 0 4 0 96 6 0 0 17 86 37 2546 3 39 15 0 175 0 5 0 95 7 0 0 17 6030 2999 4038 6 31 13 0 79 0 7 0 93 8 0 0 14 249 79 2029 4 52 7 0 586 0 3 0 97 9 0 0 2 234 66 460 1 30 1 0 904 0 0 0 100 10 0 0 13 2057 1004 3006 5 47 25 0 453 0 4 0 96 11 0 0 10 2058 1012 3655 2 24 4 0 117 0 5 0 95 12 0 0 33 4099 2018 2955 12 62 2 0 509 1 4 0 95 13 0 0 69 6112 2989 5552 13 100 2 0 640 1 6 0 93 14 0 0 0 4115 2013 0 5 0 0 0 464 0 100 0 0 15 0 0 0 4099 2005 2 5 0 0 0 465 0 100 0 0

mdb -k

Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs mpt mpt_sas sd ip hook neti sockfs arp usba uhci stmf stmf_sbd fctl md lofs random idm crypto cpc fcp fcip ufs logindmux ptm sppp nsmb smbsrv nfs kvm ]

::stacks -v mdb: stacks: processing kernel threads mdb: stacks: 127 unique stacks / 2190 threads mdb: stacks: done THREAD STATE SOBJ COUNT ffffff0170011c40 SLEEP CV 578 swtch+0x145 cv_wait+0x61 taskq_thread_wait+0x84 taskq_thread+0x308 thread_start+8

ffffff3289bb1420 SLEEP CV 200 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c lwp_park+0x157 syslwp_park+0x31 _sys_sysenter_post_swapgs+0x149

ffffff0171db7c40 SLEEP CV 174 swtch+0x145 cv_wait+0x61 mac_soft_ring_worker+0xb0 thread_start+8

ffffff3291ad9060 SLEEP CV 127 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_waituntil_sig+0xba nanosleep+0x120 _sys_sysenter_post_swapgs+0x149

ffffff0171dabc40 SLEEP CV 116 swtch+0x145 cv_wait+0x61 mac_srs_worker+0x1b4 thread_start+8

ffffff017009bc40 FREE 109

ffffff0171593c40 SLEEP CV 68 swtch+0x145 cv_wait+0x61 i_mac_notify_thread+0xc7 thread_start+8 ffffff328546cbc0 SLEEP SHUTTLE 67 swtch_to+0xe6 shuttle_resume+0x325 door_return+0x21a doorfs32+0x134 sys_syscall32+0xff ffffff3289bc2000 SLEEP SHUTTLE 65 swtch+0x145 shuttle_swtch+0x256 door_return+0x242 doorfs32+0x134 _sys_sysenter_post_swapgs+0x149 ffffff3291c55460 SLEEP SHUTTLE 55 swtch_to+0xe6 shuttle_resume+0x325 door_return+0x21a doorfs32+0x134 _sys_sysenter_post_swapgs+0x149 ffffff01705aac40 SLEEP CV 53 swtch+0x145 cv_wait+0x61 evch_delivery_hold+0x85 evch_delivery_thr+0x23e thread_start+8 ffffff3289ba1120 SLEEP CV 53 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_relwaituntil_sig+0x2c poll_common+0x483 pollsys+0xea _sys_sysenter_post_swapgs+0x149 ffffff3286679780 SLEEP SHUTTLE 35 swtch+0x145 shuttle_swtch+0x256 door_return+0x242 doorfs32+0x134 sys_syscall32+0xff ffffff3289b99860 SLEEP CV 29 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c lwp_park+0x157 syslwp_park+0x31 sys_syscall32+0xff ffffff3291ad9400 SLEEP CV 27 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 pause+0x48 _sys_sysenter_post_swapgs+0x149 ffffff3296e02460 SLEEP CV 24 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c sigtimedwait+0x1eb ffffff01725e9c40 SLEEP CV 21 swtch+0x145 cv_wait+0x61 squeue_polling_thread+0x90 thread_start+8 ffffff0172ba1c40 SLEEP CV 21 swtch+0x145 cv_wait+0x61 squeue_worker+0xc0 thread_start+8 ffffff3289bb00a0 SLEEP CV 18 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_waituntil_sig+0xba lwp_park+0x157 syslwp_park+0x31 _sys_sysenter_post_swapgs+0x149 ffffff0172f08c40 SLEEP CV 16 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f kcfpool_svc+0x83 thread_start+8 ffffff3291a63760 SLEEP CV 16 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_reltimedwait_sig+0x53 cv_relwaituntil_sig+0x25 poll_common+0x483 pollsys+0xea _sys_sysenter_post_swapgs+0x149 ffffff01705b6c40 ONPROC 16 swtch+0x145 cpu_pause+0x8c thread_start+8 ffffff32916f00a0 SLEEP CV 13 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_reltimedwait_sig+0x53 cv_relwaituntil_sig+0x25 poll_common+0x483 pollsys+0xea ffffff01700cbc40 FREE 12 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_broadcast+0x71 kvm_timer_fire+0x67 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff01731a8c40 SLEEP CV 12 swtch+0x145 cv_wait+0x61 md_daemon+0x13c start_daemon+0x1a thread_start+8 ffffff32b37aac40 SLEEP CV 12 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 kvm_vcpu_block+0x6b __vcpu_run+0x84 kvm_arch_vcpu_ioctl_run+0x110 kvm_ioctl+0x119 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e ffffff3289bb27a0 SLEEP CV 12 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 sigsuspend+0x107 _sys_sysenter_post_swapgs+0x149 ffffff32855594c0 SLEEP CV 10 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_waituntil_sig+0xba lwp_park+0x157 syslwp_park+0x31 sys_syscall32+0xff ffffff32843f7780 SLEEP CV 10 swtch+0x145 cv_wait_sig+0x14d cte_get_event+0xd7 ctfs_endpoint_ioctl+0xa7 ctfs_bu_ioctl+0x51 fop_ioctl+0x7b ioctl+0x18e sys_syscall32+0xff ffffff3291c58b60 SLEEP CV 10 swtch+0x145 cv_wait_sig+0x14d door_unref+0x89 doorfs32+0x82 _sys_sysenter_post_swapgs+0x149 ffffff0172a45c40 SLEEP CV 9 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f taskq_thread_wait+0x74 taskq_d_thread+0x144 thread_start+8 ffffff01701c7c40 SLEEP CV 8 swtch+0x145 cv_wait+0x61 mpt_doneq_thread+0x62 thread_start+8 ffffff017020fc40 SLEEP CV 8 swtch+0x145 cv_wait+0x61 mptsas_doneq_thread+0x60 thread_start+8 ffffff328551ab20 SLEEP CV 7 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_reltimedwait_sig+0x53 cv_relwaituntil_sig+0x25 poll_common+0x572 pollsys+0xea _sys_sysenter_post_swapgs+0x149 ffffff32916da140 SLEEP CV 7 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c sigtimedwait+0x1eb _sys_sysenter_post_swapgs+0x149 ffffff32855107c0 SLEEP CV 7 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 waitid+0x2a0 waitsys32+0x30 _sys_sysenter_post_swapgs+0x149 ffffff3291d65c00 SLEEP CV 6 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 fifo_read+0xca fop_read+0x6b read+0x2b8 read32+0x22 _sys_sysenter_post_swapgs+0x149 ffffff0171e8fc40 SLEEP CV 5 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a zone_status_timedwait+0x56 auto_do_unmount+0xba thread_start+8 ffffff329246bbe0 SLEEP CV 5 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_reltimedwait_sig+0x53 cv_relwaituntil_sig+0x25 poll_common+0x483 pollsys+0xea sys_syscall32+0xff ffffff32b3a18760 SLEEP CV 5 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_waituntil_sig+0xba lwp_park+0x157 syslwp_park+0x31 ffffff3291f8cae0 SLEEP CV 5 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_waituntil_sig+0xba nanosleep+0x120 ffffff0170913c40 SLEEP CV 5 swtch+0x145 cv_wait+0x61 ill_taskq_dispatch+0x128 thread_start+8 ffffff0170901c40 SLEEP CV 5 swtch+0x145 cv_wait+0x61 ipsec_loader+0x9d thread_start+8 ffffff3284d2db40 SLEEP CV 5 swtch+0x145 cv_wait_sig+0x14d door_unref+0x89 doorfs32+0x82 sys_syscall32+0xff ffffff3291c58420 SLEEP CV 5 swtch+0x145 cv_wait_sig+0x14d so_dequeue_msg+0x2a4 so_recvmsg+0x1af socket_recvmsg+0x3d socket_vop_read+0x52 fop_read+0x6b read+0x2b8 read32+0x22 _sys_sysenter_post_swapgs+0x149 ffffff331d9af740 SLEEP CV 5 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c port_getn+0x23b portfs+0x282 portfs32+0x57 sys_syscall32+0xff ffffff32843f7040 SLEEP CV 5 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_waituntil_sig+0x13c sigtimedwait+0x1eb sys_syscall32+0xff ffffff32fd8880a0 SLEEP CV 5 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 sigsuspend+0x125 sys_syscall32+0xff ffffff32e728e060 SLEEP SHUTTLE 5 swtch_to+0xe6 shuttle_resume+0x325 door_call+0x2b1 doorfs32+0x141 sys_syscall32+0xff ffffff01700c5c40 FREE 4 cstate_wakeup+0xd9 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch+0x405 callout_normal+0x121 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff0173274c40 SLEEP CV 4 swtch+0x145 cv_wait+0x61 zone_status_wait_cpr+0xa9 zsched+0x76e thread_start+8 ffffff328546c480 SLEEP CV 4 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 pause+0x48 sys_syscall32+0xff ffffff32b1a2a400 SLEEP SHUTTLE 4 swtch_to+0xe6 shuttle_resume+0x325 door_call+0x2b1 doorfs32+0x141 _sys_sysenter_post_swapgs+0x149 ffffff328a8dac40 SLEEP CV 3 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 pause+0x75 _sys_sysenter_post_swapgs+0x149 ffffff0170ce8c40 FREE 2 cstate_wakeup+0x43 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch+0x405 ehci_create_done_itd_list+0x27 apic_send_directed_EOI+0x59 apic_intr_exit+0x60 intr_thread_epilog+0xf4 dispatch_hardint+0x41 ffffff017037fc40 FREE 2 cstate_wakeup+0xd9 setbackdq+0x468 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 setrun+0x2b cv_wakeup+0x36 callout_list_expire+0x87 callout_expire+0x31 callout_realtime+0x2e cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff01708cbc40 SLEEP CV 2 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a txg_thread_wait+0x7c txg_sync_thread+0x118 thread_start+8 ffffff01702aac40 SLEEP CV 2 swtch+0x145 cv_wait+0x61 spa_thread+0x20b thread_start+8 ffffff01708c2c40 SLEEP CV 2 swtch+0x145 cv_wait+0x61 txg_thread_wait+0x5f txg_quiesce_thread+0x94 thread_start+8 ffffff0170349c40 ONPROC 2 cpu_idle_dtrace_enter+0x24 cpu_idle_enter+0xc5 cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170aa1c40 ONPROC 2 swtch+0x145 idle+0xc4 thread_start+8 ffffff0170379c40 FREE 1 0x38 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch+0x405 callout_normal+0x121 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff0170badc40 FREE 1 0xffffff0170badc40 intr_thread_epilog+0xf4 dispatch_hardint+0x41 ffffff0170f0ec40 FREE 1 0xffffff3283c5d000 intr_get_time+0x8b 0x13fb8bbb1b dispatch_hardint+0x41 ffffff0170e9dc40 FREE 1 0xffffff3283c5e500 cmt_balance+0x153 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch+0x405 callout_normal+0x121 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff0170c7dc40 FREE 1 0xffffff3283c70500 cmt_balance+0x153 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch+0x405 callout_normal+0x121 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff0170df5c40 FREE 1 apic_send_directed_EOI+0x59 apic_intr_exit+0x60 intr_thread_epilog+0xf4 dispatch_hardint+0x41 ffffff01702f8c40 FREE 1 bitset_find+0x59 cstate_wakeup+0x89 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch+0x405 callout_normal+0x121 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff0170c83c40 FREE 1 cstate_wakeup+0x43 setbackdq+0x468 ts_setrun+0x124 cv_unsleep+0x8c do_splx+0x62 disp_lock_exit+0x55 sigtoproc+0x457 sigaddqa+0x5f timer_fire+0xcd clock_highres_fire+0x46 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff0170ba7c40 FREE 1 cstate_wakeup+0xd9 setbackdq+0x468 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch_ent+0x145 zio_taskq_dispatch+0xb5 zio_interrupt+0x1c vdev_disk_io_intr+0x6b biodone+0x84 sd_buf_iodone+0x67 sd_mapblockaddr_iodone+0x4f sd_return_command+0x1e5 sdintr+0x5ab scsi_hba_pkt_comp+0x15c vhci_intr+0x66b mptsas_pkt_comp+0x47 mptsas_doneq_empty+0xbb mptsas_intr+0x388 0x33fb8bbb1b dispatch_hardint+0x41 ffffff0170ad1c40 FREE 1 igb_tx_recycle_head_wb+0x169 0xffffff0170ad1c40 0x3cfb8bbb1b dispatch_hardint+0x41 ffffff0170f73c40 FREE 1 igb_tx_recycle_head_wb+0x169 0xffffff0170f73c40 0x36fb8bbb1b dispatch_hardint+0x41 ffffff0170fdec40 FREE 1 ip_xmit+0x7eb cstate_wakeup+0xd9 setbackdq+0x468 apic_setspl+0x5c do_splx+0x62 cbe_restore_level+0x17 cyclic_reprogram_cyclic+0xfd cyclic_reprogram+0xac callout_heap_insert+0x6f timeout_generic+0x3bd tcp_timeout+0xa5 tcp_input_data+0x3f50 squeue_enter+0x546 ip_fanout_v4+0x48d ire_recv_local_v4+0x366 ill_input_short_v4+0x6ce ip_input+0x2c2 mac_rx_soft_ring_process+0x267 mac_rx_srs_proto_fanout+0x46f mac_rx_srs_drain+0x276 mac_rx_srs_process+0x1bc mac_rx_classify+0x1a2 mac_rx_flow+0x54 mac_rx_common+0x1f6 mac_rx+0xac mac_rx_ring+0x4c 0xffffff0170fdec40 0x37fb8bbb1b dispatch_hardint+0x41 ffffff01703a3c40 FREE 1 ire_recv_forward_v4+0x99b ill_input_short_v4+0x6ce ip_input+0x2c2 i_dls_link_rx+0x2f3 mac_rx_deliver+0x5d mac_rx_soft_ring_process+0x267 mac_rx_srs_proto_fanout+0x4e5 0xffffff32837a5580 cstate_wakeup+0xd9 apic_setspl+0x5c do_splx+0x62 disp_lock_exit+0x55 cv_signal+0x96 taskq_dispatch+0x405 callout_normal+0x121 cyclic_softint+0x129 dosoftint_epilog+0xfc dispatch_softint+0x3f ffffff01702fec40 FREE 1 ire_recv_forward_v4+0x99b ill_input_short_v4+0x6ce ip_input+0x2c2 i_dls_link_rx+0x2f3 mac_rx_deliver+0x5d mac_rx_soft_ring_process+0x267 mac_rx_srs_proto_fanout+0x4e5 mac_rx_srs_drain+0x276 mac_rx_srs_process+0x1bc mac_rx_classify+0x1a2 mac_rx_flow+0x54 mac_rx_common+0x1f6 mac_rx+0xac mac_rx_ring+0x4c 0xffffff01702fec40 0x39fb8bbb1b dispatch_hardint+0x41 ffffff0170b42c40 FREE 1 mac_rx_ring+0x4c 0xffffff0170b42c40 0x3dfb8bbb1b dispatch_hardint+0x41 ffffff0173babc40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f idm_wd_thread+0x1d7 thread_start+8 ffffff0170185c40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f kcfpoold+0xf6 thread_start+8 ffffff017017fc40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f page_capture_thread+0x130 thread_start+8 ffffff01736f5c40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f scsi_watch_thread+0x330 thread_start+8 ffffff017119fc40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f seg_pasync_thread+0xcb thread_start+8 ffffff0171996c40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_reltimedwait+0x4f stmf_svc_timeout+0x2aa stmf_svc+0x168 taskq_thread+0x285 thread_start+8 ffffff017018bc40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a arc_reclaim_thread+0x13d thread_start+8 ffffff0170191c40 SLEEP CV 1 swtch+0x145 cv_timedwait_hires+0xe0 cv_timedwait+0x5a l2arc_feed_thread+0xa1 thread_start+8 ffffff3289fdbbe0 SLEEP CV 1 swtch+0x145 cv_timedwait_sig_hires+0x1e9 cv_waituntil_sig+0xba nanosleep+0x15a _sys_sysenter_post_swapgs+0x149 ffffff0173722c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 crypto_bufcall_service+0x152 thread_start+8 ffffff01708e9c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 dld_taskq_dispatch+0xfc thread_start+8 ffffff0171193c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 fsflush+0x201 thread_start+8 ffffff0171861c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 ibcm_process_tlist+0x14f thread_start+8 ffffff0173b9fc40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 log_event_deliver+0x20e thread_start+8 ffffff0171199c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 mod_uninstall_daemon+0x99 thread_start+8 ffffff0171187c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 pageout+0x26a thread_start+8 ffffff017118dc40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 pageout_scanner+0x132 thread_start+8 ffffff0170197c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 pm_dep_thread+0x95 thread_start+8 ffffff0170023c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 scsi_hba_barrier_daemon+0x9f thread_start+8 ffffff0170029c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 scsi_lunchg1_daemon+0x1f7 thread_start+8 ffffff017002fc40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 scsi_lunchg2_daemon+0x120 thread_start+8 ffffff0172ef6c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 smb_thread_continue_timedwait_locked+0x60 smb_thread_continue+0x2a smb_kshare_unexport_thread+0x20 smb_thread_entry_point+0x69 thread_start+8 ffffff017292ec40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 smb_thread_continue_timedwait_locked+0x60 smb_thread_continue+0x2a smb_notify_change_daemon+0x3f smb_thread_entry_point+0x69 thread_start+8 ffffff017346ec40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 smb_thread_continue_timedwait_locked+0x60 smb_thread_continue+0x2a smb_oplock_break_thread+0x1c smb_thread_entry_point+0x69 thread_start+8 ffffff01711bbc40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 softmac_taskq_dispatch+0x104 thread_start+8 ffffff017016dc40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 streams_bufcall_service+0x146 thread_start+8 ffffff0170173c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 streams_qbkgrnd_service+0xaa thread_start+8 ffffff0170179c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 streams_sqbkgrnd_service+0x99 thread_start+8 ffffff01705a4c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 task_commit+0x99 thread_start+8 ffffff017000bc40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 thread_reaper+0xb1 thread_start+8 ffffff01700efc40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 timeout_taskq_thread+0xa7 thread_start+8 ffffff017298ac40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 ufs_thread_idle+0x152 thread_start+8 ffffff0172990c40 SLEEP CV 1 swtch+0x145 cv_wait+0x61 ufs_thread_run+0x12d ufs_thread_hlock+0x6f thread_start+8 ffffff3286665b60 SLEEP CV 1 swtch+0x145 cv_wait_sig+0x14d sd_check_media+0x10d sdioctl+0x843 cdev_ioctl+0x45 spec_ioctl+0x5a fop_ioctl+0x7b ioctl+0x18e _sys_sysenter_post_swapgs+0x149 ffffff3289b02800 SLEEP CV 1 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 cv_relwaituntil_sig+0x2c poll_common+0x483 pollsys+0xea sys_syscall32+0xff ffffff32854857e0 SLEEP CV 1 swtch+0x145 cv_wait_sig_swap_core+0x174 cv_wait_sig_swap+0x18 pause+0x75 sys_syscall32+0xff ffffff32b38c9ae0 SLEEP SHUTTLE 1 swtch_to+0xe6 shuttle_resume+0x325 door_return+0x21a doorfs32+0x11c sys_syscall32+0xff ffffff01705c2c40 ONPROC 1 0x67681200fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170322c40 ONPROC 1 0x67681500fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170e6dc40 ONPROC 1 0x67682100fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170ed8c40 ONPROC 1 0x67682400fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170d23c40 ONPROC 1 apic_intr_exit+0x32 apic_intr_exit+0x32 apic_setspl+0x5c 0xffffff3283c6dac0 dosoftint_prolog+0x98 dosoftint+0x59 do_interrupt+0x114 tsc_gethrtimeunscaled+0x16 0x67681e00fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170faec40 ONPROC 1 apic_setspl+0x5c apic_intr_exit+0x32 apic_intr_exit+0x32 apic_setspl+0x5c dosoftint_prolog+0x98 dosoftint_prolog+0x98 dosoftint+0x59 do_interrupt+0x114 tsc_gethrtimeunscaled+0x16 0x67682700fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170f43c40 ONPROC 1 apic_setspl+0x5c apic_intr_exit+0x32 apic_setspl+0x5c dosoftint_prolog+0x98 apic_intr_exit+0x32 dosoftint_prolog+0x98 dosoftint+0x59 do_interrupt+0x114 tsc_gethrtimeunscaled+0x16 0x67682400fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170dc5c40 ONPROC 1 apic_setspl+0x5c apic_setspl+0x5c dosoftint_prolog+0x98 dosoftint+0x59 do_interrupt+0x114 tsc_gethrtimeunscaled+0x16 0x67682100fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170005c40 ONPROC 1 apic_setspl+0x5c cpus apic_setspl+0x5c apic_setspl+0x5c dosoftint_prolog+0x98 dosoftint+0x59 do_interrupt+0x114 tsc_gethrtimeunscaled+0x16 0x6767eb00fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170b0cc40 ONPROC 1 apic_setspl+0x5c dosoftint_prolog+0x98 dosoftint+0x59 do_interrupt+0x114 tsc_gethrtimeunscaled+0x16 0x67681800fb86c1fd cpu_idle_mwait+0xf1 cpu_acpi_idle+0x8d cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff0170be2c40 ONPROC 1 cpu_decay+0x2f cpu_update_pct+0xe7 restore_mstate+0x129 swtch+0x145 idle+0xc4 thread_start+8 ffffff0170cb8c40 ONPROC 1 cpu_idle_dtrace_enter+0x24 0x815 acpi_cpu_cstate+0x223 cpu_acpi_idle+0x82 cpu_idle_adaptive+0x19 idle+0x114 thread_start+8 ffffff3313a1c060 ONPROC 1 page_io_unlock+0x50 pvn_plist_init+0xb2 x86pte_mapin+0x3f x86pte_release_pagetable+0x20 x86pte_set+0x136 hati_pte_map+0x39c hati_load_common+0x167 hat_memload+0x81 segvn_faultpage+0x450 segvn_fault+0xc13 as_fault+0x6c1 new_mstate+0xf0 trap+0x16b8 _sys_rtt_ints_disabled+8 0xfffffffffb85a6da uiomove+0xe9 mmio+0x145 mmrw+0x1a3 mmread+0x22 cdev_read+0x3d spec_read+0x233 fop_read+0x6b pread+0x22c fffffffffbc2f260 STOPPED 1 swtch+0x145 sched+0x3fd main+0x483 > ::ps ! grep 13540 > R 13540 1 13539 13539 0 0x42000000 ffffff3316f160a8 qemu-kvm-system- > > ffffff3316f160a8::walk thread | ::findstack -v > stack pointer for thread ffffff32b3d237e0: ffffff017315ac90 > [ ffffff017315ac90 _resume_from_idle+0xf1() ] > ffffff017315acc0 swtch+0x145() > ffffff017315ad50 cv_timedwait_sig_hires+0x1e9(ffffff3294c6e04a, ffffff3294c6e010, 3b9aca00, f4240, 0) > ffffff017315ad80 cv_reltimedwait_sig+0x53(ffffff3294c6e04a, ffffff3294c6e010, 64, 2) > ffffff017315adb0 cv_relwaituntil_sig+0x25(ffffff3294c6e04a, ffffff3294c6e010, ffffff017315ae18, 2) > ffffff017315ae70 poll_common+0x483(fffffd7fffdf9640, 5, ffffff017315ae90, 0) > ffffff017315af00 pollsys+0xea(fffffd7fffdf9640, 5, fffffd7fffdf9730, 0) > ffffff017315af10 sys_syscall+0x17a() > stack pointer for thread ffffff32fc849000: ffffff0173bb7ce0 > [ ffffff0173bb7ce0 _resume_from_idle+0xf1() ] > ffffff0173bb7d10 swtch+0x145() > ffffff0173bb7d70 cv_wait_sig_swap_core+0x174(ffffff32fc8491ee, ffffff3256aeb280, 0) > ffffff0173bb7d90 cv_wait_sig_swap+0x18(ffffff32fc8491ee, ffffff3256aeb280) > ffffff0173bb7e00 cv_waituntil_sig+0x13c(ffffff32fc8491ee, ffffff3256aeb280, 0, 0) > ffffff0173bb7f00 sigtimedwait+0x1eb(f75cc0, fffffd7ffee9ecb0, 0) > ffffff0173bb7f10 sys_syscall+0x17a() > stack pointer for thread ffffff3291a56420: ffffff01728ec3b0 > [ ffffff01728ec3b0 _resume_from_idle+0xf1() ] > ffffff01728ec3e0 swtch+0x145() > ffffff01728ec440 cv_wait_sig_swap_core+0x174(ffffff38a0fac050, ffffff38a0fac048, 0) > ffffff01728ec470 kvm_inject_pit_timer_irqs+0x76() > ffffff38a0fac000 1() > stack pointer for thread ffffff3289b20840: ffffff0170d9ece0 > [ ffffff0170d9ece0 _resume_from_idle+0xf1() ] > ffffff0170d9ed10 swtch+0x145() > ffffff0170d9ed70 cv_wait_sig_swap_core+0x174(ffffff3289b20a2e, ffffff3256aeb280, 0) > ffffff0170d9ed90 cv_wait_sig_swap+0x18(ffffff3289b20a2e, ffffff3256aeb280) > ffffff0170d9ee00 cv_waituntil_sig+0x13c(ffffff3289b20a2e, ffffff3256aeb280, 0, 0) > ffffff0170d9ef00 sigtimedwait+0x1eb(2011a5340, fffffd7ffe75fcb0, 0) > ffffff0170d9ef10 sys_syscall+0x17a() > > ::ps ! grep 16991 > R 16991 1 16990 16990 0 0x42000000 ffffff32a907f078 qemu-kvm-system- > > ffffff32a907f078::walk thread | ::findstack -v > stack pointer for thread ffffff3297ec00e0: ffffff0172bd1c90 > [ ffffff0172bd1c90 _resume_from_idle+0xf1() ] > ffffff0172bd1cc0 swtch+0x145() > ffffff0172bd1d50 cv_timedwait_sig_hires+0x1e9(ffffff329219f0ea, ffffff329219f0b0, 3b9aca00, f4240, 0) > ffffff0172bd1d80 cv_reltimedwait_sig+0x53(ffffff329219f0ea, ffffff329219f0b0, 64, 2) > ffffff0172bd1db0 cv_relwaituntil_sig+0x25(ffffff329219f0ea, ffffff329219f0b0, ffffff0172bd1e18, 2) > ffffff0172bd1e70 poll_common+0x483(fffffd7fffdf9650, 5, ffffff0172bd1e90, 0) > ffffff0172bd1f00 pollsys+0xea(fffffd7fffdf9650, 5, fffffd7fffdf9750, 0) > ffffff0172bd1f10 sys_syscall+0x17a() > stack pointer for thread ffffff32917d5480: ffffff0170a08ce0 > [ ffffff0170a08ce0 _resume_from_idle+0xf1() ] > ffffff0170a08d10 swtch+0x145() > ffffff0170a08d70 cv_wait_sig_swap_core+0x174(ffffff32917d566e, ffffff3256aead40, 0) > ffffff0170a08d90 cv_wait_sig_swap+0x18(ffffff32917d566e, ffffff3256aead40) > ffffff0170a08e00 cv_waituntil_sig+0x13c(ffffff32917d566e, ffffff3256aead40, 0, 0) > ffffff0170a08f00 sigtimedwait+0x1eb(f75cc0, fffffd7ffee9ecb0, 0) > ffffff0170a08f10 sys_syscall+0x17a() > stack pointer for thread ffffff3291c45be0: ffffff01737403b0 > [ ffffff01737403b0 _resume_from_idle+0xf1() ] > ffffff01737403e0 swtch+0x145() > ffffff0173740440 cv_wait_sig_swap_core+0x174(ffffff3294dfd050, ffffff3294dfd048, 0) > ffffff0173740460 cv_wait_sig_swap+0x18(ffffff3294dfd050, ffffff3294dfd048) > ffffff0173740490 kvm_vcpu_block+0x6b() > ffffff01737404b0 __vcpu_run+0x84() > ffffff01737404f0 kvm_arch_vcpu_ioctl_run+0x110() > ffffff0173740d00 kvm_ioctl+0x119() > ffffff0173740d40 cdev_ioctl+0x45(10e0000000e, 2000ae80, 0, 202003, ffffff38a0f88d80, ffffff0173740e24) > ffffff0173740d80 spec_ioctl+0x5a(ffffff3317602640, 2000ae80, 0, 202003, ffffff38a0f88d80, ffffff0173740e24, 0) > ffffff0173740e00 fop_ioctl+0x7b(ffffff3317602640, 2000ae80, 0, 202003, ffffff38a0f88d80, ffffff0173740e24, 0) > ffffff0173740f00 ioctl+0x18e(b, 2000ae80, 0) > ffffff0173740f10 sys_syscall+0x17a() > stack pointer for thread ffffff32e77fdba0: ffffff0171e95ce0 > [ ffffff0171e95ce0 _resume_from_idle+0xf1() ] > ffffff0171e95d10 swtch+0x145() > ffffff0171e95d70 cv_wait_sig_swap_core+0x174(ffffff32e77fdd8e, ffffff3256aead40, 0) > ffffff0171e95d90 cv_wait_sig_swap+0x18(ffffff32e77fdd8e, ffffff3256aead40) > ffffff0171e95e00 cv_waituntil_sig+0x13c(ffffff32e77fdd8e, ffffff3256aead40, 0, 0) > ffffff0171e95f00 sigtimedwait+0x1eb(281206b20, fffffd7ffe75fcb0, 0) > ffffff0171e95f10 sys_syscall+0x17a() Guest OS kernel is 2.6.18-308.1.1.el5.
rmustacc commented 12 years ago

Okay, I should have noticed this earlier, but one of the things that we have fixed a little while back and is hopefully what is causing you problems is related to the presence of -no-hpet and -no-acpi. There have been known boot and timer issues with those, which is what inspired us to fix those in QEMU as well as get multiticks there. For the systems that fail to boot, it'd still be extremely helpful if you could get a VNC screenshot or something like that, so we know where they are failing to boot.

Unrelated, I do notice cstates seem to be on the scene, which have been known to cause general illumos problems, you may be happier with them disabled in the bios.

estibi commented 12 years ago

Screenshot attached.

http://www.dropbox.com/gallery/7506135/1/illumos-kvm-cmd%20%238?h=9ee6be

On Tue, Mar 13, 2012 at 4:33 PM, Robert Mustacchi reply@reply.github.com wrote:

Okay, I should have noticed this earlier, but one of the things that we have fixed a little while back and is hopefully what is causing you problems is related to the presence of -no-hpet and -no-acpi. There have been known boot and timer issues with those, which is what inspired us to fix those in QEMU as well as get multiticks there. For the systems that fail to boot, it'd still be extremely helpful if you could get a VNC screenshot or something like that, so we know where they are failing to boot.

Unrelated, I do notice cstates seem to be on the scene, which have been known to cause general illumos problems, you may be happier with them disabled in the bios.

Reply to this email directly or view it on GitHub: https://github.com/joyent/illumos-kvm-cmd/issues/8#issuecomment-4475736

Piotr Jasiukajtis

estibi commented 12 years ago

I'm able to reproduce guest's IO-APIC panic without -no-hpet flag using simple shell script (was paranoid about SMF in the first place).

I can see some symbols added by 'HVM-711 need an interval timer-based alarm timer backend' in the live image of running qemu-kvm process, so I think we have at least right bits in the right place.

http://pastebin.com/ZPvaC1md

mdb -p 21574

Loading modules: [ ld.so.1 libc.so.1 ]

::nm -D ! grep multitick 0x0000000000541b20|0x00000000000002b4|FUNC |LOCL |0x0 |17 |multiticks_start_timer 0x0000000000541ad0|0x000000000000004e|FUNC |LOCL |0x0 |17 |multiticks_stop_timer 0x0000000000541de0|0x0000000000000492|FUNC |LOCL |0x0 |17 |multiticks_rearm_timer 0x0000000000754918|0x0000000000000004|OBJT |GLOB |0x0 |30 |multiticks_tolerance_jitter 0x000000000075491c|0x0000000000000004|OBJT |GLOB |0x0 |30 |multiticks_enabled 0x0000000000754910|0x0000000000000008|OBJT |GLOB |0x0 |30 |multiticks_tolerance_interval 0x0000000000754900|0x0000000000000004|OBJT |GLOB |0x0 |30 |multiticks_reap_multiplier 0x0000000000754908|0x0000000000000008|OBJT |GLOB |0x0 |30 |multiticks_reap_threshold

estibi commented 12 years ago

Do I need to enable something on a client side (linux guest)? I do have IO-APIC panics if HPET KVM is used. I traced all system calls from qemu-kvm:

grep -i timer_create kvm_calls

4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C788) = 0 4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C7B0) = 0 4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C7D8) = 0 4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C800) = 0 4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C828) = 0 4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C850) = 0 4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C878) = 0 4461: timer_create(4, 0xFFFFFD7FFFDFF6E0, 0x00F8C8A0) = 0

clockid_t argument in timer_create is 4, which means it's using CLOCK_MONOTONIC (CLOCK_HIGHRES)

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/sys/time_impl.h#126

So at least we know CLOCK_HIGHRES is used here.

estibi commented 12 years ago

Ok, I figured out the issue was an old guest kernel version. I'm unable to reproduce it on the latest guest kernel version (I don't use -no-hpet option now).