search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.63k stars 1.75k forks source link

Kernel panic on sys_read/sys_epoll_wait (zfs 0.6.5.2+?) #4301

Closed aktau closed 8 years ago

aktau commented 8 years ago

Running zol on Debian Jessie. No special configuration. The machine has only 1.7GB RAM, so low memory situations happen. It's used as a media center (running transmission, kodi and openvpn, mostly).

$ apt-cache policy spl spl-dkms zfs-dkms | grep -i 'installed' -B1 | grep -v -- '--'
spl:
  Installed: 0.6.5.3-1
spl-dkms:
  Installed: 0.6.5-1
zfs-dkms:
  Installed: 0.6.5.2-2

$ name -srv
Linux 4.3.0-1-amd64 #1 SMP Debian 4.3.3-7 (2016-01-19)

I tried to deduplicate this bug report but couldn't find any that matched exactly, although zio_vdev_io_start occurs in a lot of these reports.

[ 8833.270509] perf interrupt took too long (5011 > 5000), lowering kernel.perf_event_max_sample_rate to 25000
[25640.570027] VERIFY3((arc_stats.arcstat_c.value.ui64) >= 2ULL << 24) failed (33447699 >= 33554432)
[25640.570037] PANIC at arc.c:3760:arc_adapt()
[25640.570041] Showing stack for process 1534
[25640.570048] CPU: 1 PID: 1534 Comm: DVDPlayer Tainted: P           O    4.3.0-1-amd64 #1 Debian 4.3.3-7
[25640.570051] Hardware name: GIADA ChiefRiver Platform/Emerald Lake 2, BIOS W6561014 02/05/2013
[25640.570055]  0000000000000000 0000000001d28f43 ffffffff812ddcf9 ffffffffa09990a3
[25640.570063]  ffffffffa06e2adb ffffffff810dcb17 0000000000000030 ffff88003ec03950
[25640.570068]  ffff88003ec038f0 2833594649524556 6174735f63726128 74736372612e7374
[25640.570073] Call Trace:
[25640.570090]  [<ffffffff812ddcf9>] ? dump_stack+0x40/0x57
[25640.570112]  [<ffffffffa06e2adb>] ? spl_panic+0xbb/0xf0 [spl]
[25640.570122]  [<ffffffff810dcb17>] ? getrawmonotonic64+0x27/0xb0
[25640.570213]  [<ffffffffa0975eec>] ? zio_vdev_io_start+0x16c/0x2c0 [zfs]
[25640.570296]  [<ffffffffa0979145>] ? zio_nowait+0xa5/0x160 [zfs]
[25640.570332]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[25640.570344]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[25640.570420]  [<ffffffffa0976ecc>] ? zio_execute+0xbc/0x170 [zfs]
[25640.570455]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[25640.570489]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[25640.570522]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[25640.570559]  [<ffffffffa08d9d2d>] ? dbuf_read+0x66d/0x850 [zfs]
[25640.570597]  [<ffffffffa08da691>] ? dbuf_prefetch+0x191/0x2e0 [zfs]
[25640.570644]  [<ffffffffa08f0ae3>] ? dmu_zfetch_dofetch.isra.6+0xe3/0x150 [zfs]
[25640.570690]  [<ffffffffa08f1199>] ? dmu_zfetch+0x429/0xcd0 [zfs]
[25640.570728]  [<ffffffffa08d9a87>] ? dbuf_read+0x3c7/0x850 [zfs]
[25640.570769]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[25640.570809]  [<ffffffffa08e25a0>] ? dmu_read_uio_dnode+0x50/0xf0 [zfs]
[25640.570882]  [<ffffffffa095eab1>] ? zfs_range_lock+0x411/0x5f0 [zfs]
[25640.570924]  [<ffffffffa08e3559>] ? dmu_read_uio_dbuf+0x39/0x50 [zfs]
[25640.570995]  [<ffffffffa09639bc>] ? zfs_read+0x13c/0x3d0 [zfs]
[25640.571065]  [<ffffffffa097e1e4>] ? zpl_read_common_iovec+0x84/0xd0 [zfs]
[25640.571134]  [<ffffffffa097e428>] ? zpl_read+0x88/0xc0 [zfs]
[25640.571141]  [<ffffffff811cd36e>] ? vfs_read+0x7e/0x120
[25640.571146]  [<ffffffff811ce072>] ? SyS_read+0x52/0xc0
[25640.571156]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[25645.428288] VERIFY3((arc_stats.arcstat_c.value.ui64) >= 2ULL << 24) failed (33447699 >= 33554432)
[25645.428300] PANIC at arc.c:3760:arc_adapt()
[25645.428303] Showing stack for process 781
[25645.428312] CPU: 1 PID: 781 Comm: transmission-da Tainted: P           O    4.3.0-1-amd64 #1 Debian 4.3.3-7
[25645.428316] Hardware name: GIADA ChiefRiver Platform/Emerald Lake 2, BIOS W6561014 02/05/2013
[25645.428320]  0000000000000000 0000000055283119 ffffffff812ddcf9 ffffffffa09990a3
[25645.428328]  ffffffffa06e2adb ffff88006ee9a720 ffff880000000030 ffff8800662afb00
[25645.428334]  ffff8800662afaa0 2833594649524556 6174735f63726128 74736372612e7374
[25645.428340] Call Trace:
[25645.428358]  [<ffffffff812ddcf9>] ? dump_stack+0x40/0x57
[25645.428380]  [<ffffffffa06e2adb>] ? spl_panic+0xbb/0xf0 [spl]
[25645.428391]  [<ffffffff8109a234>] ? ttwu_do_wakeup+0x14/0xc0
[25645.428399]  [<ffffffff8109ac43>] ? try_to_wake_up+0x43/0x330
[25645.428445]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[25645.428458]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[25645.428464]  [<ffffffff810b0e8e>] ? __wake_up_common+0x4e/0x90
[25645.428479]  [<ffffffffa06e448d>] ? __cv_init+0x3d/0x60 [spl]
[25645.428518]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[25645.428557]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[25645.428594]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[25645.428635]  [<ffffffffa08d952a>] ? dbuf_create+0x39a/0x500 [zfs]
[25645.428675]  [<ffffffffa08d8020>] ? dbuf_rele_and_unlock+0x440/0x440 [zfs]
[25645.428716]  [<ffffffffa08d9980>] ? dbuf_read+0x2c0/0x850 [zfs]
[25645.428763]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[25645.428808]  [<ffffffffa08e2313>] ? dmu_read+0xa3/0x1a0 [zfs]
[25645.428816]  [<ffffffff812e30c7>] ? __radix_tree_create+0x77/0x1f0
[25645.428904]  [<ffffffffa0963d6a>] ? zfs_getpage+0x11a/0x1e0 [zfs]
[25645.428988]  [<ffffffffa097e690>] ? zpl_writepages+0x150/0x150 [zfs]
[25645.429067]  [<ffffffffa097e6e3>] ? zpl_readpage+0x53/0xa0 [zfs]
[25645.429074]  [<ffffffff8116c981>] ? read_cache_pages+0x91/0xd0
[25645.429080]  [<ffffffff8116cb5d>] ? __do_page_cache_readahead+0x19d/0x220
[25645.429086]  [<ffffffff8116cef2>] ? force_page_cache_readahead+0x42/0x60
[25645.429092]  [<ffffffff8119df57>] ? SyS_fadvise64+0x1e7/0x230
[25645.429100]  [<ffffffff81213213>] ? SyS_epoll_wait+0x93/0xd0
[25645.429110]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[25792.307936] INFO: task transmission-da:781 blocked for more than 120 seconds.
[25792.307950]       Tainted: P           O    4.3.0-1-amd64 #1
[25792.307953] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[25792.307957] transmission-da D ffffffff810a08c0     0   781      1 0x00000000
[25792.307967]  ffff88006ee9a6c0 0000000000000082 ffff8800758c8dc0 ffff8800662b0000
[25792.307974]  ffff8800662afaf0 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[25792.307980]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[25792.307986] Call Trace:
[25792.308006]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[25792.308026]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[25792.308038]  [<ffffffff8109a234>] ? ttwu_do_wakeup+0x14/0xc0
[25792.308046]  [<ffffffff8109ac43>] ? try_to_wake_up+0x43/0x330
[25792.308100]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[25792.308116]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[25792.308122]  [<ffffffff810b0e8e>] ? __wake_up_common+0x4e/0x90
[25792.308137]  [<ffffffffa06e448d>] ? __cv_init+0x3d/0x60 [spl]
[25792.308179]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[25792.308221]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[25792.308262]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[25792.308307]  [<ffffffffa08d952a>] ? dbuf_create+0x39a/0x500 [zfs]
[25792.308350]  [<ffffffffa08d8020>] ? dbuf_rele_and_unlock+0x440/0x440 [zfs]
[25792.308395]  [<ffffffffa08d9980>] ? dbuf_read+0x2c0/0x850 [zfs]
[25792.308446]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[25792.308494]  [<ffffffffa08e2313>] ? dmu_read+0xa3/0x1a0 [zfs]
[25792.308503]  [<ffffffff812e30c7>] ? __radix_tree_create+0x77/0x1f0
[25792.308602]  [<ffffffffa0963d6a>] ? zfs_getpage+0x11a/0x1e0 [zfs]
[25792.308693]  [<ffffffffa097e690>] ? zpl_writepages+0x150/0x150 [zfs]
[25792.308778]  [<ffffffffa097e6e3>] ? zpl_readpage+0x53/0xa0 [zfs]
[25792.308788]  [<ffffffff8116c981>] ? read_cache_pages+0x91/0xd0
[25792.308794]  [<ffffffff8116cb5d>] ? __do_page_cache_readahead+0x19d/0x220
[25792.308800]  [<ffffffff8116cef2>] ? force_page_cache_readahead+0x42/0x60
[25792.308806]  [<ffffffff8119df57>] ? SyS_fadvise64+0x1e7/0x230
[25792.308814]  [<ffffffff81213213>] ? SyS_epoll_wait+0x93/0xd0
[25792.308824]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[25792.308840] INFO: task DVDPlayer:1534 blocked for more than 120 seconds.
[25792.308845]       Tainted: P           O    4.3.0-1-amd64 #1
[25792.308847] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[25792.308851] DVDPlayer       D ffffffff810a08c0     0  1534    814 0x00000000
[25792.308858]  ffff88006b5eb300 0000000000000086 ffff8800758c8dc0 ffff88003ec04000
[25792.308864]  ffff88003ec03940 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[25792.308869]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[25792.308875] Call Trace:
[25792.308883]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[25792.308900]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[25792.308990]  [<ffffffffa0975eec>] ? zio_vdev_io_start+0x16c/0x2c0 [zfs]
[25792.309078]  [<ffffffffa0979145>] ? zio_nowait+0xa5/0x160 [zfs]
[25792.309122]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[25792.309136]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[25792.309222]  [<ffffffffa0976ecc>] ? zio_execute+0xbc/0x170 [zfs]
[25792.309263]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[25792.309302]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[25792.309343]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[25792.309388]  [<ffffffffa08d9d2d>] ? dbuf_read+0x66d/0x850 [zfs]
[25792.309433]  [<ffffffffa08da691>] ? dbuf_prefetch+0x191/0x2e0 [zfs]
[25792.309490]  [<ffffffffa08f0ae3>] ? dmu_zfetch_dofetch.isra.6+0xe3/0x150 [zfs]
[25792.309547]  [<ffffffffa08f1199>] ? dmu_zfetch+0x429/0xcd0 [zfs]
[25792.309594]  [<ffffffffa08d9a87>] ? dbuf_read+0x3c7/0x850 [zfs]
[25792.309643]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[25792.309693]  [<ffffffffa08e25a0>] ? dmu_read_uio_dnode+0x50/0xf0 [zfs]
[25792.309779]  [<ffffffffa095eab1>] ? zfs_range_lock+0x411/0x5f0 [zfs]
[25792.309831]  [<ffffffffa08e3559>] ? dmu_read_uio_dbuf+0x39/0x50 [zfs]
[25792.309915]  [<ffffffffa09639bc>] ? zfs_read+0x13c/0x3d0 [zfs]
[25792.310000]  [<ffffffffa097e1e4>] ? zpl_read_common_iovec+0x84/0xd0 [zfs]
[25792.310083]  [<ffffffffa097e428>] ? zpl_read+0x88/0xc0 [zfs]
[25792.310093]  [<ffffffff811cd36e>] ? vfs_read+0x7e/0x120
[25792.310098]  [<ffffffff811ce072>] ? SyS_read+0x52/0xc0
[25792.310108]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[25912.270028] INFO: task transmission-da:781 blocked for more than 120 seconds.
[25912.270042]       Tainted: P           O    4.3.0-1-amd64 #1
[25912.270045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[25912.270049] transmission-da D ffffffff810a08c0     0   781      1 0x00000000
[25912.270060]  ffff88006ee9a6c0 0000000000000082 ffff8800758c8dc0 ffff8800662b0000
[25912.270067]  ffff8800662afaf0 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[25912.270073]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[25912.270079] Call Trace:
[25912.270099]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[25912.270119]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[25912.270132]  [<ffffffff8109a234>] ? ttwu_do_wakeup+0x14/0xc0
[25912.270140]  [<ffffffff8109ac43>] ? try_to_wake_up+0x43/0x330
[25912.270194]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[25912.270209]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[25912.270216]  [<ffffffff810b0e8e>] ? __wake_up_common+0x4e/0x90
[25912.270231]  [<ffffffffa06e448d>] ? __cv_init+0x3d/0x60 [spl]
[25912.270274]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[25912.270316]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[25912.270358]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[25912.270405]  [<ffffffffa08d952a>] ? dbuf_create+0x39a/0x500 [zfs]
[25912.270449]  [<ffffffffa08d8020>] ? dbuf_rele_and_unlock+0x440/0x440 [zfs]
[25912.270494]  [<ffffffffa08d9980>] ? dbuf_read+0x2c0/0x850 [zfs]
[25912.270545]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[25912.270596]  [<ffffffffa08e2313>] ? dmu_read+0xa3/0x1a0 [zfs]
[25912.270605]  [<ffffffff812e30c7>] ? __radix_tree_create+0x77/0x1f0
[25912.270701]  [<ffffffffa0963d6a>] ? zfs_getpage+0x11a/0x1e0 [zfs]
[25912.270794]  [<ffffffffa097e690>] ? zpl_writepages+0x150/0x150 [zfs]
[25912.270880]  [<ffffffffa097e6e3>] ? zpl_readpage+0x53/0xa0 [zfs]
[25912.270889]  [<ffffffff8116c981>] ? read_cache_pages+0x91/0xd0
[25912.270895]  [<ffffffff8116cb5d>] ? __do_page_cache_readahead+0x19d/0x220
[25912.270901]  [<ffffffff8116cef2>] ? force_page_cache_readahead+0x42/0x60
[25912.270907]  [<ffffffff8119df57>] ? SyS_fadvise64+0x1e7/0x230
[25912.270915]  [<ffffffff81213213>] ? SyS_epoll_wait+0x93/0xd0
[25912.270925]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[25912.270941] INFO: task DVDPlayer:1534 blocked for more than 120 seconds.
[25912.270945]       Tainted: P           O    4.3.0-1-amd64 #1
[25912.270948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[25912.270952] DVDPlayer       D ffffffff810a08c0     0  1534    814 0x00000000
[25912.270959]  ffff88006b5eb300 0000000000000086 ffff8800758c8dc0 ffff88003ec04000
[25912.270965]  ffff88003ec03940 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[25912.270971]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[25912.270976] Call Trace:
[25912.270984]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[25912.271002]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[25912.271092]  [<ffffffffa0975eec>] ? zio_vdev_io_start+0x16c/0x2c0 [zfs]
[25912.271177]  [<ffffffffa0979145>] ? zio_nowait+0xa5/0x160 [zfs]
[25912.271218]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[25912.271236]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[25912.271320]  [<ffffffffa0976ecc>] ? zio_execute+0xbc/0x170 [zfs]
[25912.271362]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[25912.271402]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[25912.271442]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[25912.271487]  [<ffffffffa08d9d2d>] ? dbuf_read+0x66d/0x850 [zfs]
[25912.271533]  [<ffffffffa08da691>] ? dbuf_prefetch+0x191/0x2e0 [zfs]
[25912.271591]  [<ffffffffa08f0ae3>] ? dmu_zfetch_dofetch.isra.6+0xe3/0x150 [zfs]
[25912.271646]  [<ffffffffa08f1199>] ? dmu_zfetch+0x429/0xcd0 [zfs]
[25912.271693]  [<ffffffffa08d9a87>] ? dbuf_read+0x3c7/0x850 [zfs]
[25912.271744]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[25912.271794]  [<ffffffffa08e25a0>] ? dmu_read_uio_dnode+0x50/0xf0 [zfs]
[25912.271880]  [<ffffffffa095eab1>] ? zfs_range_lock+0x411/0x5f0 [zfs]
[25912.271932]  [<ffffffffa08e3559>] ? dmu_read_uio_dbuf+0x39/0x50 [zfs]
[25912.272019]  [<ffffffffa09639bc>] ? zfs_read+0x13c/0x3d0 [zfs]
[25912.272104]  [<ffffffffa097e1e4>] ? zpl_read_common_iovec+0x84/0xd0 [zfs]
[25912.272186]  [<ffffffffa097e428>] ? zpl_read+0x88/0xc0 [zfs]
[25912.272194]  [<ffffffff811cd36e>] ? vfs_read+0x7e/0x120
[25912.272200]  [<ffffffff811ce072>] ? SyS_read+0x52/0xc0
[25912.272209]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[26032.232024] INFO: task transmission-da:781 blocked for more than 120 seconds.
[26032.232038]       Tainted: P           O    4.3.0-1-amd64 #1
[26032.232041] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[26032.232046] transmission-da D ffffffff810a08c0     0   781      1 0x00000000
[26032.232056]  ffff88006ee9a6c0 0000000000000082 ffff8800758c8dc0 ffff8800662b0000
[26032.232063]  ffff8800662afaf0 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[26032.232068]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[26032.232074] Call Trace:
[26032.232093]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[26032.232113]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[26032.232125]  [<ffffffff8109a234>] ? ttwu_do_wakeup+0x14/0xc0
[26032.232132]  [<ffffffff8109ac43>] ? try_to_wake_up+0x43/0x330
[26032.232186]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[26032.232201]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[26032.232208]  [<ffffffff810b0e8e>] ? __wake_up_common+0x4e/0x90
[26032.232223]  [<ffffffffa06e448d>] ? __cv_init+0x3d/0x60 [spl]
[26032.232270]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[26032.232312]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[26032.232352]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[26032.232397]  [<ffffffffa08d952a>] ? dbuf_create+0x39a/0x500 [zfs]
[26032.232443]  [<ffffffffa08d8020>] ? dbuf_rele_and_unlock+0x440/0x440 [zfs]
[26032.232485]  [<ffffffffa08d9980>] ? dbuf_read+0x2c0/0x850 [zfs]
[26032.232537]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[26032.232586]  [<ffffffffa08e2313>] ? dmu_read+0xa3/0x1a0 [zfs]
[26032.232595]  [<ffffffff812e30c7>] ? __radix_tree_create+0x77/0x1f0
[26032.232691]  [<ffffffffa0963d6a>] ? zfs_getpage+0x11a/0x1e0 [zfs]
[26032.232783]  [<ffffffffa097e690>] ? zpl_writepages+0x150/0x150 [zfs]
[26032.232866]  [<ffffffffa097e6e3>] ? zpl_readpage+0x53/0xa0 [zfs]
[26032.232875]  [<ffffffff8116c981>] ? read_cache_pages+0x91/0xd0
[26032.232881]  [<ffffffff8116cb5d>] ? __do_page_cache_readahead+0x19d/0x220
[26032.232887]  [<ffffffff8116cef2>] ? force_page_cache_readahead+0x42/0x60
[26032.232893]  [<ffffffff8119df57>] ? SyS_fadvise64+0x1e7/0x230
[26032.232901]  [<ffffffff81213213>] ? SyS_epoll_wait+0x93/0xd0
[26032.232911]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[26032.232928] INFO: task DVDPlayer:1534 blocked for more than 120 seconds.
[26032.232932]       Tainted: P           O    4.3.0-1-amd64 #1
[26032.232935] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[26032.232939] DVDPlayer       D ffffffff810a08c0     0  1534    814 0x00000000
[26032.232946]  ffff88006b5eb300 0000000000000086 ffff8800758c8dc0 ffff88003ec04000
[26032.232951]  ffff88003ec03940 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[26032.232957]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[26032.232962] Call Trace:
[26032.232971]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[26032.232988]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[26032.233079]  [<ffffffffa0975eec>] ? zio_vdev_io_start+0x16c/0x2c0 [zfs]
[26032.233165]  [<ffffffffa0979145>] ? zio_nowait+0xa5/0x160 [zfs]
[26032.233208]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[26032.233223]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[26032.233308]  [<ffffffffa0976ecc>] ? zio_execute+0xbc/0x170 [zfs]
[26032.233351]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[26032.233390]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[26032.233430]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[26032.233475]  [<ffffffffa08d9d2d>] ? dbuf_read+0x66d/0x850 [zfs]
[26032.233519]  [<ffffffffa08da691>] ? dbuf_prefetch+0x191/0x2e0 [zfs]
[26032.233578]  [<ffffffffa08f0ae3>] ? dmu_zfetch_dofetch.isra.6+0xe3/0x150 [zfs]
[26032.233634]  [<ffffffffa08f1199>] ? dmu_zfetch+0x429/0xcd0 [zfs]
[26032.233680]  [<ffffffffa08d9a87>] ? dbuf_read+0x3c7/0x850 [zfs]
[26032.233730]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[26032.233779]  [<ffffffffa08e25a0>] ? dmu_read_uio_dnode+0x50/0xf0 [zfs]
[26032.233863]  [<ffffffffa095eab1>] ? zfs_range_lock+0x411/0x5f0 [zfs]
[26032.233916]  [<ffffffffa08e3559>] ? dmu_read_uio_dbuf+0x39/0x50 [zfs]
[26032.233998]  [<ffffffffa09639bc>] ? zfs_read+0x13c/0x3d0 [zfs]
[26032.234084]  [<ffffffffa097e1e4>] ? zpl_read_common_iovec+0x84/0xd0 [zfs]
[26032.234166]  [<ffffffffa097e428>] ? zpl_read+0x88/0xc0 [zfs]
[26032.234176]  [<ffffffff811cd36e>] ? vfs_read+0x7e/0x120
[26032.234181]  [<ffffffff811ce072>] ? SyS_read+0x52/0xc0
[26032.234190]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[26152.194098] INFO: task transmission-da:781 blocked for more than 120 seconds.
[26152.194111]       Tainted: P           O    4.3.0-1-amd64 #1
[26152.194115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[26152.194119] transmission-da D ffffffff810a08c0     0   781      1 0x00000000
[26152.194129]  ffff88006ee9a6c0 0000000000000082 ffff8800758c8dc0 ffff8800662b0000
[26152.194136]  ffff8800662afaf0 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[26152.194142]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[26152.194149] Call Trace:
[26152.194168]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[26152.194188]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[26152.194200]  [<ffffffff8109a234>] ? ttwu_do_wakeup+0x14/0xc0
[26152.194208]  [<ffffffff8109ac43>] ? try_to_wake_up+0x43/0x330
[26152.194265]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[26152.194282]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[26152.194288]  [<ffffffff810b0e8e>] ? __wake_up_common+0x4e/0x90
[26152.194303]  [<ffffffffa06e448d>] ? __cv_init+0x3d/0x60 [spl]
[26152.194347]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[26152.194390]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[26152.194431]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[26152.194475]  [<ffffffffa08d952a>] ? dbuf_create+0x39a/0x500 [zfs]
[26152.194521]  [<ffffffffa08d8020>] ? dbuf_rele_and_unlock+0x440/0x440 [zfs]
[26152.194566]  [<ffffffffa08d9980>] ? dbuf_read+0x2c0/0x850 [zfs]
[26152.194617]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[26152.194668]  [<ffffffffa08e2313>] ? dmu_read+0xa3/0x1a0 [zfs]
[26152.194677]  [<ffffffff812e30c7>] ? __radix_tree_create+0x77/0x1f0
[26152.194775]  [<ffffffffa0963d6a>] ? zfs_getpage+0x11a/0x1e0 [zfs]
[26152.194868]  [<ffffffffa097e690>] ? zpl_writepages+0x150/0x150 [zfs]
[26152.194954]  [<ffffffffa097e6e3>] ? zpl_readpage+0x53/0xa0 [zfs]
[26152.194963]  [<ffffffff8116c981>] ? read_cache_pages+0x91/0xd0
[26152.194969]  [<ffffffff8116cb5d>] ? __do_page_cache_readahead+0x19d/0x220
[26152.194975]  [<ffffffff8116cef2>] ? force_page_cache_readahead+0x42/0x60
[26152.194982]  [<ffffffff8119df57>] ? SyS_fadvise64+0x1e7/0x230
[26152.194991]  [<ffffffff81213213>] ? SyS_epoll_wait+0x93/0xd0
[26152.195000]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[26152.195016] INFO: task DVDPlayer:1534 blocked for more than 120 seconds.
[26152.195022]       Tainted: P           O    4.3.0-1-amd64 #1
[26152.195024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[26152.195028] DVDPlayer       D ffffffff810a08c0     0  1534    814 0x00000000
[26152.195034]  ffff88006b5eb300 0000000000000086 ffff8800758c8dc0 ffff88003ec04000
[26152.195040]  ffff88003ec03940 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[26152.195046]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[26152.195051] Call Trace:
[26152.195059]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[26152.195076]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[26152.195169]  [<ffffffffa0975eec>] ? zio_vdev_io_start+0x16c/0x2c0 [zfs]
[26152.195257]  [<ffffffffa0979145>] ? zio_nowait+0xa5/0x160 [zfs]
[26152.195298]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[26152.195314]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[26152.195402]  [<ffffffffa0976ecc>] ? zio_execute+0xbc/0x170 [zfs]
[26152.195444]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[26152.195484]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[26152.195526]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[26152.195569]  [<ffffffffa08d9d2d>] ? dbuf_read+0x66d/0x850 [zfs]
[26152.195615]  [<ffffffffa08da691>] ? dbuf_prefetch+0x191/0x2e0 [zfs]
[26152.195674]  [<ffffffffa08f0ae3>] ? dmu_zfetch_dofetch.isra.6+0xe3/0x150 [zfs]
[26152.195729]  [<ffffffffa08f1199>] ? dmu_zfetch+0x429/0xcd0 [zfs]
[26152.195774]  [<ffffffffa08d9a87>] ? dbuf_read+0x3c7/0x850 [zfs]
[26152.195824]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[26152.195875]  [<ffffffffa08e25a0>] ? dmu_read_uio_dnode+0x50/0xf0 [zfs]
[26152.195962]  [<ffffffffa095eab1>] ? zfs_range_lock+0x411/0x5f0 [zfs]
[26152.196012]  [<ffffffffa08e3559>] ? dmu_read_uio_dbuf+0x39/0x50 [zfs]
[26152.196097]  [<ffffffffa09639bc>] ? zfs_read+0x13c/0x3d0 [zfs]
[26152.196181]  [<ffffffffa097e1e4>] ? zpl_read_common_iovec+0x84/0xd0 [zfs]
[26152.196262]  [<ffffffffa097e428>] ? zpl_read+0x88/0xc0 [zfs]
[26152.196271]  [<ffffffff811cd36e>] ? vfs_read+0x7e/0x120
[26152.196276]  [<ffffffff811ce072>] ? SyS_read+0x52/0xc0
[26152.196286]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[26272.156095] INFO: task transmission-da:781 blocked for more than 120 seconds.
[26272.156108]       Tainted: P           O    4.3.0-1-amd64 #1
[26272.156112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[26272.156116] transmission-da D ffffffff810a08c0     0   781      1 0x00000000
[26272.156126]  ffff88006ee9a6c0 0000000000000082 ffff8800758c8dc0 ffff8800662b0000
[26272.156133]  ffff8800662afaf0 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[26272.156139]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[26272.156145] Call Trace:
[26272.156165]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[26272.156185]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[26272.156197]  [<ffffffff8109a234>] ? ttwu_do_wakeup+0x14/0xc0
[26272.156206]  [<ffffffff8109ac43>] ? try_to_wake_up+0x43/0x330
[26272.156261]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[26272.156277]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[26272.156283]  [<ffffffff810b0e8e>] ? __wake_up_common+0x4e/0x90
[26272.156299]  [<ffffffffa06e448d>] ? __cv_init+0x3d/0x60 [spl]
[26272.156343]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[26272.156384]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[26272.156426]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[26272.156470]  [<ffffffffa08d952a>] ? dbuf_create+0x39a/0x500 [zfs]
[26272.156515]  [<ffffffffa08d8020>] ? dbuf_rele_and_unlock+0x440/0x440 [zfs]
[26272.156559]  [<ffffffffa08d9980>] ? dbuf_read+0x2c0/0x850 [zfs]
[26272.156610]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[26272.156659]  [<ffffffffa08e2313>] ? dmu_read+0xa3/0x1a0 [zfs]
[26272.156668]  [<ffffffff812e30c7>] ? __radix_tree_create+0x77/0x1f0
[26272.156767]  [<ffffffffa0963d6a>] ? zfs_getpage+0x11a/0x1e0 [zfs]
[26272.156856]  [<ffffffffa097e690>] ? zpl_writepages+0x150/0x150 [zfs]
[26272.156942]  [<ffffffffa097e6e3>] ? zpl_readpage+0x53/0xa0 [zfs]
[26272.156951]  [<ffffffff8116c981>] ? read_cache_pages+0x91/0xd0
[26272.156957]  [<ffffffff8116cb5d>] ? __do_page_cache_readahead+0x19d/0x220
[26272.156963]  [<ffffffff8116cef2>] ? force_page_cache_readahead+0x42/0x60
[26272.156970]  [<ffffffff8119df57>] ? SyS_fadvise64+0x1e7/0x230
[26272.156978]  [<ffffffff81213213>] ? SyS_epoll_wait+0x93/0xd0
[26272.156988]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67
[26272.157004] INFO: task DVDPlayer:1534 blocked for more than 120 seconds.
[26272.157009]       Tainted: P           O    4.3.0-1-amd64 #1
[26272.157011] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[26272.157014] DVDPlayer       D ffffffff810a08c0     0  1534    814 0x00000000
[26272.157021]  ffff88006b5eb300 0000000000000086 ffff8800758c8dc0 ffff88003ec04000
[26272.157027]  ffff88003ec03940 0000000000000eb0 ffffffffa098c298 ffffffffa0999148
[26272.157033]  0000000000000000 ffffffff8158217f ffffffffa09990a3 ffffffffa06e2b00
[26272.157038] Call Trace:
[26272.157046]  [<ffffffff8158217f>] ? schedule+0x2f/0x70
[26272.157064]  [<ffffffffa06e2b00>] ? spl_panic+0xe0/0xf0 [spl]
[26272.157156]  [<ffffffffa0975eec>] ? zio_vdev_io_start+0x16c/0x2c0 [zfs]
[26272.157243]  [<ffffffffa0979145>] ? zio_nowait+0xa5/0x160 [zfs]
[26272.157287]  [<ffffffffa08ce8a6>] ? buf_cons+0x66/0x70 [zfs]
[26272.157301]  [<ffffffffa06df359>] ? spl_kmem_cache_alloc+0x129/0x790 [spl]
[26272.157388]  [<ffffffffa0976ecc>] ? zio_execute+0xbc/0x170 [zfs]
[26272.157430]  [<ffffffffa08ce765>] ? arc_get_data_buf.isra.21+0x395/0x3d0 [zfs]
[26272.157471]  [<ffffffffa08d19b5>] ? arc_buf_alloc+0x105/0x170 [zfs]
[26272.157511]  [<ffffffffa08d21b7>] ? arc_read+0x1b7/0xa80 [zfs]
[26272.157553]  [<ffffffffa08d9d2d>] ? dbuf_read+0x66d/0x850 [zfs]
[26272.157600]  [<ffffffffa08da691>] ? dbuf_prefetch+0x191/0x2e0 [zfs]
[26272.157658]  [<ffffffffa08f0ae3>] ? dmu_zfetch_dofetch.isra.6+0xe3/0x150 [zfs]
[26272.157712]  [<ffffffffa08f1199>] ? dmu_zfetch+0x429/0xcd0 [zfs]
[26272.157759]  [<ffffffffa08d9a87>] ? dbuf_read+0x3c7/0x850 [zfs]
[26272.157809]  [<ffffffffa08e196b>] ? dmu_buf_hold_array_by_dnode+0x11b/0x480 [zfs]
[26272.157859]  [<ffffffffa08e25a0>] ? dmu_read_uio_dnode+0x50/0xf0 [zfs]
[26272.157945]  [<ffffffffa095eab1>] ? zfs_range_lock+0x411/0x5f0 [zfs]
[26272.157997]  [<ffffffffa08e3559>] ? dmu_read_uio_dbuf+0x39/0x50 [zfs]
[26272.158081]  [<ffffffffa09639bc>] ? zfs_read+0x13c/0x3d0 [zfs]
[26272.158166]  [<ffffffffa097e1e4>] ? zpl_read_common_iovec+0x84/0xd0 [zfs]
[26272.158248]  [<ffffffffa097e428>] ? zpl_read+0x88/0xc0 [zfs]
[26272.158256]  [<ffffffff811cd36e>] ? vfs_read+0x7e/0x120
[26272.158261]  [<ffffffff811ce072>] ? SyS_read+0x52/0xc0
[26272.158270]  [<ffffffff81585df2>] ? system_call_fast_compare_end+0xc/0x67

Anything that tries to do anything to the filesystem in question just hangs. Luckily, my root isn't ZFS at the moment, so I could login and gather some data :). Anything I should try?

behlendorf commented 8 years ago

Sorry about that. This is a duplicate of #3904, there was a fix recently merged to master,. You can cherry-pick 1b8951b319b7bcddfe0cc2408b7717f9ce3d758b if you need a fix now. It's also already been merged to the release branch for the next point release.

aktau commented 8 years ago

Sorry about that. This is a duplicate of #3904, there was a fix recently merged to master,. You can cherry-pick 1b8951b if you need a fix now. It's also already been merged to the release branch for the next point release.

Nothing to be sorry about, these things happens. I should apologize for the noise generated by not finding the older issue.

I've just upgraded the RAM on this little unit from 2GB to 8GB, which I think should keep me out of the water until the repo maintainer decides to update the ZoL repo to a version with the fix. (Or the Debian maintainers upload zfs-dkms et al. to the main Debian repo, whichever comes first).

Thanks for the prompt reply, great maintainership there.