search

openzfs / spl

A shim layer which adds the core interfaces required for OpenZFS.
https://zfsonlinux.org/
GNU General Public License v2.0
282 stars 181 forks source link

general protection fault: 0000 [#1] SMP arc_adapt #285

Closed gentoouser closed 8 years ago

gentoouser commented 10 years ago

My system keeps locking up and I think it might be something with zfs.

After the error happens I cannot do any zpool or zfs command. Other times the load keeps increasing and network, input devices, and video stops to function.

When I do a hard reboot it will work for a while, but will lock up after a indeterminate amount of time.

Here are some log output.

/var/log/syslog (Just general protection fault)

general protection fault: 0000 [#1] SMP
Sep  1 11:16:23 zeus kernel: [11078.008280] CPU 7
Sep  1 11:16:23 zeus kernel: [11078.008281] Modules linked in: asus_atk0110 dm_crypt bnep rfcomm bluetooth nfsd nfs binfmt_misc lockd fscache auth_rpcgss nfs_acl sunrpc mt2131 s5h1409 tda18271 tda8290 tuner nls_iso8859_1 nls_cp437 vfat fat cx25840 ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder ir_rc6_decoder ir_rc5_decoder sp5100_tco eeepc_wmi asus_wmi sparse_keymap snd_hda_codec_hdmi edac_core edac_mce_amd rc_rc6_mce snd_hda_codec_realtek fam15h_power psmouse serio_raw mceusb k10temp joydev i2c_piix4 fglrx(P) snd_hda_intel snd_hda_codec ir_nec_decoder snd_hwdep cx23885 rc_core videobuf_dma_sg altera_stapl snd_pcm cx2341x videobuf_dvb videobuf_core snd_seq_midi v4l2_common videodev v4l2_compat_ioctl32 snd_rawmidi altera_ci dvb_core btcx_risc tveeprom snd_seq_midi_event snd_seq snd_timer snd_seq_device snd soundcore snd_page_alloc mac_hid adm1021 it87 hwmon_vid lp parport zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl(O) zlib_deflate hid_logitech_dj usbhid hid mxm_wmi firewire_oh
Sep  1 11:16:23 zeus kernel: ci sata_sil firewire_core crc_itu_t r8169 wmi usb_storage [last unloaded: cpuid]
Sep  1 11:16:23 zeus kernel: [11078.008347]
Sep  1 11:16:23 zeus kernel: [11078.008349] Pid: 444, comm: arc_adapt Tainted: P           O 3.2.0-52-generic #78-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX
Sep  1 11:16:23 zeus kernel: [11078.008355] RIP: 0010:[<ffffffffa015e944>]  [<ffffffffa015e944>] dbuf_destroy+0x104/0x170 [zfs]
Sep  1 11:16:23 zeus kernel: [11078.008385] RSP: 0018:ffff88040243de10  EFLAGS: 00010286
Sep  1 11:16:23 zeus kernel: [11078.008387] RAX: 4cff880364b21a80 RBX: ffff880364b21a80 RCX: 0000000000000000
Sep  1 11:16:23 zeus kernel: [11078.008389] RDX: ffff88040243dfd8 RSI: 000000000000002f RDI: ffffffffa0234410
Sep  1 11:16:23 zeus kernel: [11078.008391] RBP: ffff88040243de30 R08: 00df3f4d940a7bfa R09: 0000000000000000
Sep  1 11:16:23 zeus kernel: [11078.008394] R10: ffff880402a2b420 R11: ffff880364f13300 R12: 4cff880364b21ab8
Sep  1 11:16:23 zeus kernel: [11078.008396] R13: ffffffffa0234410 R14: 0000000000000000 R15: 0000000000000000
Sep  1 11:16:23 zeus kernel: [11078.008399] FS:  00007f5866627700(0000) GS:ffff88043edc0000(0000) knlGS:00007fffff768000
Sep  1 11:16:23 zeus kernel: [11078.008401] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep  1 11:16:23 zeus kernel: [11078.008403] CR2: 0000000018a47ff8 CR3: 0000000299852000 CR4: 00000000000406e0
Sep  1 11:16:23 zeus kernel: [11078.008406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep  1 11:16:23 zeus kernel: [11078.008408] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep  1 11:16:23 zeus kernel: [11078.008411] Process arc_adapt (pid: 444, threadinfo ffff88040243c000, task ffff88041eaf8000)
Sep  1 11:16:23 zeus kernel: [11078.008413] Stack:
Sep  1 11:16:23 zeus kernel: [11078.008414]  ffff880364b21a80 ffff880364b21a80 ffff880365af47c0 0000000000000000
Sep  1 11:16:23 zeus kernel: [11078.008419]  ffff88040243de50 ffffffffa015fb6e ffff880364b21a80 ffff880364b21a80
Sep  1 11:16:23 zeus kernel: [11078.008423]  ffff88040243de70 ffffffffa015fbc3 ffff880365af47b0 ffff880365af47b0
Sep  1 11:16:23 zeus kernel: [11078.008428] Call Trace:
Sep  1 11:16:23 zeus kernel: [11078.008445]  [<ffffffffa015fb6e>] dbuf_evict+0x1e/0x30 [zfs]
Sep  1 11:16:23 zeus kernel: [11078.008462]  [<ffffffffa015fbc3>] dbuf_do_evict+0x43/0x70 [zfs]
Sep  1 11:16:23 zeus kernel: [11078.008478]  [<ffffffffa0155790>] arc_do_user_evicts+0x50/0xd0 [zfs]
Sep  1 11:16:23 zeus kernel: [11078.008495]  [<ffffffffa015a510>] ? arc_adjust_meta+0x1e0/0x1e0 [zfs]
Sep  1 11:16:23 zeus kernel: [11078.008511]  [<ffffffffa015a58b>] arc_adapt_thread+0x7b/0x170 [zfs]
Sep  1 11:16:23 zeus kernel: [11078.008522]  [<ffffffffa00d2cb8>] thread_generic_wrapper+0x78/0x90 [spl]
Sep  1 11:16:23 zeus kernel: [11078.008531]  [<ffffffffa00d2c40>] ? __thread_create+0x310/0x310 [spl]
Sep  1 11:16:23 zeus kernel: [11078.008536]  [<ffffffff8108b78c>] kthread+0x8c/0xa0
Sep  1 11:16:23 zeus kernel: [11078.008540]  [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10
Sep  1 11:16:23 zeus kernel: [11078.008544]  [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0
Sep  1 11:16:23 zeus kernel: [11078.008547]  [<ffffffff8166ae70>] ? gs_change+0x13/0x13
Sep  1 11:16:23 zeus kernel: [11078.008549] Code: 05 49 81 c5 d0 43 23 a0 4c 89 ef e8 77 08 50 e1 48 8b 05 98 5a 0d 00 4e 8d 24 e0 49 8b 04 24 48 39 c3 74 10 0f 1f 00 4c 8d 60 38 <48> 8b 40 38 48 39 c3 75 f3 48 8b 43 38 4c 89 ef 49 89 04 24 48
Sep  1 11:16:23 zeus kernel: [11078.008581] RIP  [<ffffffffa015e944>] dbuf_destroy+0x104/0x170 [zfs]
Sep  1 11:16:23 zeus kernel: [11078.008597]  RSP <ffff88040243de10>
Sep  1 11:16:23 zeus kernel: [11078.008604] ---[ end trace 3f6476c14c7646e3 ]---

Running Mythbuntu 12.04 uname -a

Linux zeus 3.2.0-52-generic #78-Ubuntu SMP Fri Jul 26 16:21:44 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/modprobe.d/zfs.conf

options zfs l2arc_noprefetch=0 zfs_prefetch_disable=0 zfs_arc_max=12884901888 zfs_arc_meta_limit=6442450944

cat /proc/spl/kmem/slab


--------------------- cache -------------------------------------------------------  ----- slab ------  ---- object -----  --- emergency ---
name                                    flags      size     alloc slabsize  objsize  total alloc   max  total alloc   max  dlock alloc   max
spl_vn_cache                          0x00020      8192      3696     4096       88      2     2     2     42    42    42      0     0     0
spl_vn_file_cache                     0x00020         0         0     4096       96      0     0     0      0     0     0      0     0     0
spl_zlib_workspace_cache              0x00140  16777216  16622448  8388608   268104      2     2     2     62    62    62      0     0     0
space_seg_cache                       0x00020   2621440   1458176     4096       64    640   640   676  24320 22784 25688      0     0     0
zio_cache                             0x00020   1409024    659880    32768     1080     43    43   237   1247   611  6873      0     0     0
zio_link_cache                        0x00020    221184     61344     4096       48     54    54   164   2430  1278  7380      0     0     0
zio_vdev_cache                        0x00040  29360128  22284960  4194304   131088      7     7     8    210   170   218      0     0     0
zio_buf_512                           0x00020   4685824   2138624    32768      512    143   143   143   4433  4177  4433      0     0     0
zio_data_buf_512                      0x00020    360448    174592    32768      512     11    11    11    341   341   341      0     0     0
zio_buf_1024                          0x00040   1048576    557056    32768     1024     32    32    87    672   544  1827      0     0     0
zio_data_buf_1024                     0x00040    360448    236544    32768     1024     11    11    11    231   231   231      0     0     0
zio_buf_1536                          0x00040   1376256    901632    65536     1536     21    21    21    651   587   651      0     0     0
zio_data_buf_1536                     0x00040    458752    333312    65536     1536      7     7     7    217   217   217      0     0     0
zio_buf_2048                          0x00040   2031616   1062912    65536     2048     31    31    31    775   519   775      0     0     0
zio_data_buf_2048                     0x00040    458752    358400    65536     2048      7     7     7    175   175   175      0     0     0
zio_buf_2560                          0x00040   1638400   1016320    65536     2560     25    25    27    525   397   567      0     0     0
zio_data_buf_2560                     0x00040    589824    483840    65536     2560      9     9     9    189   189   189      0     0     0
zio_buf_3072                          0x00040    917504    774144    65536     3072     14    14    14    252   252   252      0     0     0
zio_data_buf_3072                     0x00040   1245184    854016    65536     3072     19    19    19    342   278   342      0     0     0
zio_buf_3584                          0x00040    917504    777728   131072     3584      7     7     7    217   217   217      0     0     0
zio_data_buf_3584                     0x00040    786432    666624   131072     3584      6     6     6    186   186   186      0     0     0
zio_buf_4096                          0x00040   3932160   1511424   262144     4096     15    15    15    465   369   465      0     0     0
zio_data_buf_4096                     0x00040   2883584   1003520   262144     4096     11    11    11    341   245   341      0     0     0
zio_buf_5120                          0x00040   1703936   1233920   131072     5120     13    13    13    273   241   273      0     0     0
zio_data_buf_5120                     0x00040    917504    752640   131072     5120      7     7     7    147   147   147      0     0     0
zio_buf_6144                          0x00040   3145728   1830912   131072     6144     24    24    29    432   298   522      0     0     0
zio_data_buf_6144                     0x00040    655360    552960   131072     6144      5     5     5     90    90    90      0     0     0
zio_buf_7168                          0x00040   2359296   1770496   262144     7168      9     9     9    279   247   279      0     0     0
zio_data_buf_7168                     0x00040   1048576    888832   262144     7168      4     4     4    124   124   124      0     0     0
zio_buf_8192                          0x00040   3145728   2064384   262144     8192     12    12    12    252   252   252      0     0     0
zio_data_buf_8192                     0x00040   2097152   1376256   262144     8192      8     8     8    168   168   168      0     0     0
zio_buf_10240                         0x00040   3145728   1925120   262144    10240     12    12    12    252   188   252      0     0     0
zio_data_buf_10240                    0x00040   2097152   1720320   262144    10240      8     8     8    168   168   168      0     0     0
zio_buf_12288                         0x00040   4718592   2641920   524288    12288      9     9     9    279   215   279      0     0     0
zio_data_buf_12288                    0x00040   2097152   1523712   524288    12288      4     4     4    124   124   124      0     0     0
zio_buf_14336                         0x00040   4718592   3540992   524288    14336      9     9    10    279   247   310      0     0     0
zio_data_buf_14336                    0x00040   2097152   1777664   524288    14336      4     4     4    124   124   124      0     0     0
zio_buf_16384                         0x00040  51380224  38043648   524288    16384     98    98    98   2450  2322  2450      0     0     0
zio_data_buf_16384                    0x00040   4194304   1998848   524288    16384      8     8    10    200   122   250      0     0     0
zio_buf_20480                         0x00040   6815744   4935680   524288    20480     13    13    16    273   241   336      0     0     0
zio_data_buf_20480                    0x00040   4194304   3440640   524288    20480      8     8     8    168   168   168      0     0     0
zio_buf_24576                         0x00040   7864320   5849088   524288    24576     15    15    15    270   238   270      0     0     0
zio_data_buf_24576                    0x00040   1048576    884736   524288    24576      2     2     2     36    36    36      0     0     0
zio_buf_28672                         0x00040   9437184   6164480  1048576    28672      9     9     9    279   215   279      0     0     0
zio_data_buf_28672                    0x00040   6291456   5332992  1048576    28672      6     6     6    186   186   186      0     0     0
zio_buf_32768                         0x00040  10485760   7077888  1048576    32768     10    10    10    280   216   280      0     0     0
zio_data_buf_32768                    0x00040   7340032   6422528  1048576    32768      7     7     7    196   196   196      0     0     0
zio_buf_36864                         0x00040  11534336   8957952  1048576    36864     11    11    11    275   243   275      0     0     0
zio_data_buf_36864                    0x00040   8388608   7372800  1048576    36864      8     8     8    200   200   200      0     0     0
zio_buf_40960                         0x00040  14680064   9256960  1048576    40960     14    14    14    322   226   322      0     0     0
zio_data_buf_40960                    0x00040   8388608   6594560  1048576    40960      8     7     7    184   161   161      0     0     0
zio_buf_45056                         0x00040  26214400  13516800  1048576    45056     25    25    35    525   300   735      0     0     0
zio_data_buf_45056                    0x00040   5242880   4730880  1048576    45056      5     5     5    105   105   105      0     0     0
zio_buf_49152                         0x00040  18874368  12091392  1048576    49152     18    18    18    342   246   342      0     0     0
zio_data_buf_49152                    0x00040   3145728   2801664  1048576    49152      3     3     3     57    57    57      0     0     0
zio_buf_53248                         0x00040  19922944  14802944  1048576    53248     19    19    19    342   278   342      0     0     0
zio_data_buf_53248                    0x00040   7340032   6709248  1048576    53248      7     7     7    126   126   126      0     0     0
zio_buf_57344                         0x00040  18874368  17547264  1048576    57344     18    18    18    306   306   306      0     0     0
zio_data_buf_57344                    0x00040   5242880   4874240  1048576    57344      5     5     5     85    85    85      0     0     0
zio_buf_61440                         0x00040  23068672  15052800  2097152    61440     11    11    11    341   245   341      0     0     0
zio_data_buf_61440                    0x00040  10485760   9523200  2097152    61440      5     5     5    155   155   155      0     0     0
zio_buf_65536                         0x00040  20971520  15466496  2097152    65536     10    10    10    300   236   300      0     0     0
zio_data_buf_65536                    0x00040  12582912  11796480  2097152    65536      6     6     6    180   180   180      0     0     0
zio_buf_69632                         0x00040  14680064  13647872  2097152    69632      7     7     7    196   196   196      0     0     0
zio_data_buf_69632                    0x00040  12582912  11698176  2097152    69632      6     6     6    168   168   168      0     0     0
zio_buf_73728                         0x00040  10485760   9584640  2097152    73728      5     5     5    130   130   130      0     0     0
zio_data_buf_73728                    0x00040  10485760   9584640  2097152    73728      5     5     5    130   130   130      0     0     0
zio_buf_77824                         0x00040  10485760   9728000  2097152    77824      5     5     5    125   125   125      0     0     0
zio_data_buf_77824                    0x00040   6291456   5836800  2097152    77824      3     3     3     75    75    75      0     0     0
zio_buf_81920                         0x00040  12582912  11796480  2097152    81920      6     6     6    144   144   144      0     0     0
zio_data_buf_81920                    0x00040  12582912  11796480  2097152    81920      6     6     6    144   144   144      0     0     0
zio_buf_86016                         0x00040  14680064  13848576  2097152    86016      7     7     7    161   161   161      0     0     0
zio_data_buf_86016                    0x00040  14680064  13848576  2097152    86016      7     7     7    161   161   161      0     0     0
zio_buf_90112                         0x00040   8388608   7929856  2097152    90112      4     4     4     88    88    88      0     0     0
zio_data_buf_90112                    0x00040   4194304   3964928  2097152    90112      2     2     2     44    44    44      0     0     0
zio_buf_94208                         0x00040  12582912  11870208  2097152    94208      6     6     6    126   126   126      0     0     0
zio_data_buf_94208                    0x00040  12582912  11870208  2097152    94208      6     6     6    126   126   126      0     0     0
zio_buf_98304                         0x00040  14680064  13762560  2097152    98304      7     7     7    140   140   140      0     0     0
zio_data_buf_98304                    0x00040  10485760   9830400  2097152    98304      5     5     5    100   100   100      0     0     0
zio_buf_102400                        0x00040  14680064  13619200  2097152   102400      7     7     7    133   133   133      0     0     0
zio_data_buf_102400                   0x00040  14680064  13619200  2097152   102400      7     7     7    133   133   133      0     0     0
zio_buf_106496                        0x00040  14680064  13418496  2097152   106496      7     7     7    126   126   126      0     0     0
zio_data_buf_106496                   0x00040   2097152   1916928  2097152   106496      1     1     1     18    18    18      0     0     0
zio_buf_110592                        0x00040  14680064  13934592  2097152   110592      7     7     7    126   126   126      0     0     0
zio_data_buf_110592                   0x00040  12582912  11943936  2097152   110592      6     6     6    108   108   108      0     0     0
zio_buf_114688                        0x00040  14680064  13647872  2097152   114688      7     7     7    119   119   119      0     0     0
zio_data_buf_114688                   0x00040   4194304   3899392  2097152   114688      2     2     2     34    34    34      0     0     0
zio_buf_118784                        0x00040   2097152   2019328  2097152   118784      1     1     1     17    17    17      0     0     0
zio_data_buf_118784                   0x00040  12582912  12115968  2097152   118784      6     6     6    102   102   102      0     0     0
zio_buf_122880                        0x00040         0         0  2097152   122880      0     0     0      0     0     0      0     0     0
zio_data_buf_122880                   0x00040   6291456   5898240  2097152   122880      3     3     3     48    48    48      0     0     0
zio_buf_126976                        0x00040         0         0  4194304   126976      0     0     0      0     0     0      0     0     0
zio_data_buf_126976                   0x00040  16777216  15745024  4194304   126976      4     4     4    124   124   124      0     0     0
zio_buf_131072                        0x00040  20971520  11927552  4194304   131072      5     5    15    155    91   450      0     0     0
zio_data_buf_131072                   0x00040 13161725952 12749373440  4194304   131072   3138  3138  3139  97278 97270 97302      0     0     0
lz4_cache                             0x00040   4194304   4063232   524288    16384      8     8     8    248   248   248      0     0     0
sa_cache                              0x00020    294912    190080     4096       80     72    72    72   2376  2376  2376      0     0     0
spill_cache                           0x00040         0         0  4194304   131072      0     0     0      0     0     0      0     0     0
dnode_t                               0x00020   4308992   3905024    16384      928    263   263   263   4208  4208  4208      0     0     0
dmu_buf_impl_t                        0x00020  34127872  29155000     8192      280   4166  4165  4165  104150 104125 104125      0     0     0
arc_buf_hdr_t                         0x00020  87908352  73650720     8192      264  10731 10730 10730  279006 278980 278980      0     0     0
arc_buf_t                             0x00020  11546624   7306848     4096       72   2819  2819  2821  101484 101484 101556      0     0     0
zil_lwb_cache                         0x00020    266240    160000     4096      200     65    65    93   1040   800  1488      0     0     0
zfs_znode_cache                       0x00020   2523136   2367904    32768      992     77    77    77   2387  2387  2387      0     0     0

cat /proc/spl/kstat/zfs/arcstats

4 1 0x01 84 4032 6767837588 11881194361060
name                            type data
hits                            4    2772590
misses                          4    87073
demand_data_hits                4    880044
demand_data_misses              4    15189
demand_metadata_hits            4    1857771
demand_metadata_misses          4    1865
prefetch_data_hits              4    1367
prefetch_data_misses            4    69541
prefetch_metadata_hits          4    33408
prefetch_metadata_misses        4    478
mru_hits                        4    306266
mru_ghost_hits                  4    1069
mfu_hits                        4    2439897
mfu_ghost_hits                  4    6
deleted                         4    10816
recycle_miss                    4    57
mutex_miss                      4    4
evict_skip                      4    453164
evict_l2_cached                 4    8497478144
evict_l2_eligible               4    2200793600
evict_l2_ineligible             4    163301376
hash_elements                   4    278664
hash_elements_max               4    278664
hash_collisions                 4    200761
hash_chains                     4    75344
hash_chain_max                  4    7
p                               4    805552128
c                               4    12884901888
c_min                           4    3221225472
c_max                           4    12884901888
size                            4    12885016384
hdr_size                        4    79753464
data_size                       4    12765022720
other_size                      4    33439248
anon_size                       4    18795520
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    106665984
mru_evict_data                  4    79439872
mru_evict_metadata              4    13602816
mru_ghost_size                  4    10025191936
mru_ghost_evict_data            4    9689464320
mru_ghost_evict_metadata        4    335727616
mfu_size                        4    12639561216
mfu_evict_data                  4    12628735488
mfu_evict_metadata              4    1333248
mfu_ghost_size                  4    2479090176
mfu_ghost_evict_data            4    2477538304
mfu_ghost_evict_metadata        4    1551872
l2_hits                         4    109
l2_misses                       4    86935
l2_feeds                        4    12091
l2_rw_clash                     4    0
l2_read_bytes                   4    10841088
l2_write_bytes                  4    12616475648
l2_writes_sent                  4    3820
l2_writes_done                  4    3820
l2_writes_error                 4    0
l2_writes_hdr_miss              4    1
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    223
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    12409737216
l2_asize                        4    11813787136
l2_hdr_size                     4    6800952
l2_compress_successes           4    105490
l2_compress_zeros               4    0
l2_compress_failures            4    85786
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    0
memory_indirect_count           4    19343
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    158094656
arc_meta_limit                  4    6442450944
arc_meta_max                    4    160304280

cat /proc/spl/kstat/zfs/zil

5 1 0x01 13 624 6768062103 11904571572435
name                            type data
zil_commit_count                4    15417
zil_commit_writer_count         4    15417
zil_itx_count                   4    469118
zil_itx_indirect_count          4    0
zil_itx_indirect_bytes          4    0
zil_itx_copied_count            4    0
zil_itx_copied_bytes            4    0
zil_itx_needcopy_count          4    459727
zil_itx_needcopy_bytes          4    15448977895
zil_itx_metaslab_normal_count   4    0
zil_itx_metaslab_normal_bytes   4    0
zil_itx_metaslab_slog_count     4    181655
zil_itx_metaslab_slog_bytes     4    15572150176

cat /proc/spl/kstat/zfs/zfetchstats

2 1 0x01 11 528 6767422065 11924866530432
name                            type data
hits                            4    7184339
misses                          4    1114900
colinear_hits                   4    253
colinear_misses                 4    1114647
stride_hits                     4    6971255
stride_misses                   4    5
reclaim_successes               4    2273
reclaim_failures                4    1112374
streams_resets                  4    25
streams_noresets                4    213086
bogus_streams                   4    0

cat /proc/spl/kstat/zfs/vdev_cache_stats

6 1 0x01 3 144 6768063600 11941050308150
name                            type data
delegations                     4    0
hits                            4    0
misses                          4    0

cat /proc/spl/kstat/zfs/fm

0 1 0x01 4 192 6760957118 11953987818825
name                            type data
erpt-dropped                    4    0
erpt-set-failed                 4    0
fmri-set-failed                 4    0
payload-set-failed              4    0

cat /proc/spl/kstat/zfs/txgs-data

9 5 0x01 60 4320 7462936770 11969002122747
txg      state birth         nread        nwritten     reads    writes   otime        qtime        stime
7570088  C     10852780056754 0            32595968     0        694        4999998330         1467   1092368171
7570089  C     10857780056551 94720        27654656     2        681        5000011531         1466   1617363767
7570090  C     10862780069549 775680       51465216     11       1308       5000031576          977   2284654369
7570091  C     10867780102592 466432       65698816     9        1596       5000012021         1466   2087159141
7570092  C     10872780116080 611840       70736384     12       1715       3082310917          977   2645168387
7570093  C     10875862427975 466944       67017728     10       1800       3669156203         1467   2389136230
7570094  C     10879531586134 584704       67476992     9        1679       2857259093         1955   3144787857
7570095  C     10882388846205 376320       62651904     8        1603       3587724782         1467   2723808697
7570096  C     10885976571965 222720       64940032     5        1592       4270865863          977   3103423869
7570097  C     10890247438806 459264       62130688     10       1590       3695277397         1955   2991718082
7570098  C     10893942717670 2631680      62732288     40       1652       4087966123         1467   3208871795
7570099  C     10898030685260 286720       61335040     6        1569       4112268634          978   2692591253
7570100  C     10902142954872 280576       63868416     6        1581       3772890545         1467   2886554204
7570101  C     10905915846884 278528       57519104     6        1611       1189020858         1466   3113835873
7570102  C     10907104869698 472576       55715840     11       1532       5205826281         1467   3138892263
7570103  C     10912310696956 451072       62454272     10       1559       3273875838         1467   2791920643
7570104  C     10915584573772 387584       54577152     9        1438       3542492644         1466   3273241734
7570105  C     10919127067883 327680       58512896     7        1528       2237339541         1955   2251916449
7570106  C     10921364408891 500224       52611072     11       1565       3913443609         2444   3025930496
7570107  C     10925277854455 496640       53208576     12       1478       1172368099         1467   2793372180
7570108  C     10926450224020 463360       58214912     10       1509       6336670521         1467   2809248088
7570109  C     10932786896007 584192       54789632     16       1478       2706229975         1467   2733635004
7570110  C     10935493127449 496128       60676096     12       1587       4016844038          978   2998409590
7570111  C     10939509972464 313344       53281792     9        1460       3343521446          978   2543490811
7570112  C     10942853494888 344576       50854400     8        1465       2899873182         1955   3180152594
7570113  C     10945753370026 236544       56373760     5        1523       3862465972         1466   3092052161
7570114  C     10949615836976 368128       49421824     9        1432        487629311         1955   2675834349
7570115  C     10950103467754 446464       60403200     10       1540       6484437241         1466   2569662374
7570116  C     10956587906462 502784       52225536     11       1469        733136029         1467   3318142873
7570117  C     10957321043957 179200       69088768     6        1578       5897296176          978   2955519756
7570118  C     10963218341111 7835648      54591488     69       1460       3856098603          978   3391020577
7570119  C     10967074440691 113657344    52925952     880      1445       1360490214         1467   2525802575
7570120  C     10968434932371 86128128     50567680     670      1373       4912536369         1956   2670059025
7570121  C     10973347470207 89646080     53710848     694      1559       2847249437          977   2946964574
7570122  C     10976194721111 78028800     48708096     603      1412        109121976          978   2764574756
7570123  C     10976303844065 123380736    60773888     953      1523       6480936255         1467   2740757228
7570124  C     10982784781298 101584384    50917376     790      1406       1255214378         1467   3150810060
7570125  C     10984039996654 124446208    51698176     964      1383       4698611315         1956   2687284771
7570126  C     10988738608947 113592832    60365824     877      1519       2725762805         1467   3270229641
7570127  C     10991464372730 79625728     49154048     619      1372       2801131922          978   3000420416
7570128  C     10994265505629 84507648     49219072     658      1393       5435894639         1466   3200170423
7570129  C     10999701402224 102387200    50267648     795      1434       3809250182        16623   2824949442
7570130  C     11003510653384 109849600    51161088     853      1451        591381742          978   2700422858
7570131  C     11004102036104 103356928    55085056     801      1477       5515696083         1466   3376775623
7570132  C     11009617733654 97490944     47449600     759      1402       3538130217          978   3156243158
7570133  C     11013155865827 69249536     56827392     537      1516       3556017443         1466   2792295134
7570134  C     11016711885226 82781696     60024832     637      1551       3197600782         1466   2794356806
7570135  C     11019909487475 62027776     65214464     478      1554        341161091         1466   2614906711
7570136  C     11020250649544 44326912     67223040     342      1300       7449401649         1467   2836855030
7570137  C     11027700052170 44982272     28075520     347      658        5000070170         1955    977232283
7570138  C     11032700123318 79750656     42108928     614      1025       4999950877         1956   2086864323
7570139  C     11037700075662 36427264     34177536     286      823        4999987543         1955   1135374344
7570140  C     11042700064672 42579968     31678464     332      751        5000003188         1466   1475840716
7570141  C     11047700069326 65135104     30135296     505      695        4999989494         1467   1326443038
7570142  C     11052700060286 42716672     25909760     330      622        5000005629         1467   1108333046
7570143  C     11057700068360 73153536     22586880     565      599        4999997316         1466    842506848
7570144  C     11062700067143 47794176     34252800     384      785        5003984750         1467   1813098460
7570145  C     11067704053360 50170368     29467136     400      681        4999993403         1466   1429785775
7570146  S     11072704048230 0            0            0        0          5000020290         2444            0
7570147  O     11077704070964 0            0            0        0                   0            0            0

cat /proc/spl/kstat/zfs/dmu_tx

3 1 0x01 12 576 6767426273 11997397357319
name                            type data
dmu_tx_assigned                 4    1364076
dmu_tx_delay                    4    0
dmu_tx_error                    4    0
dmu_tx_suspended                4    0
dmu_tx_group                    4    0
dmu_tx_how                      4    0
dmu_tx_memory_reserve           4    0
dmu_tx_memory_reclaim           4    0
dmu_tx_memory_inflight          4    0
dmu_tx_dirty_throttle           4    0
dmu_tx_write_limit              4    102
dmu_tx_quota                    4    0

cat /proc/spl/kstat/zfs/dmu_tx_assign-data

10 1 0x01 32 1536 7462944496 12009513920581
name                            type data
1 us                            4    972722
2 us                            4    291133
4 us                            4    88477
8 us                            4    5554
16 us                           4    2456
32 us                           4    3440
64 us                           4    184
128 us                          4    43
256 us                          4    11
512 us                          4    16
1024 us                         4    14
2048 us                         4    4
4096 us                         4    7
8192 us                         4    10
16384 us                        4    3
32768 us                        4    1
65536 us                        4    1
131072 us                       4    0
262144 us                       4    0
524288 us                       4    0
1048576 us                      4    0
2097152 us                      4    0
4194304 us                      4    0
8388608 us                      4    0
16777216 us                     4    0
33554432 us                     4    0
67108864 us                     4    0
134217728 us                    4    0
268435456 us                    4    0
536870912 us                    4    0
1073741824 us                   4    0
2147483648 us                   4    0

cat /proc/meminfo

MemTotal:       16379928 kB
MemFree:         1421820 kB
Buffers:           19716 kB
Cached:            82992 kB
SwapCached:        28516 kB
Active:           542352 kB
Inactive:         156544 kB
Active(anon):     510804 kB
Inactive(anon):    89860 kB
Active(file):      31548 kB
Inactive(file):    66684 kB
Unevictable:        3832 kB
Mlocked:               0 kB
SwapTotal:      46874996 kB
SwapFree:       46698140 kB
Dirty:                44 kB
Writeback:             0 kB
AnonPages:        580312 kB
Mapped:            24016 kB
Shmem:               624 kB
Slab:              85516 kB
SReclaimable:      25236 kB
SUnreclaim:        60280 kB
KernelStack:        5096 kB
PageTables:        19768 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    55064960 kB
Committed_AS:    2139684 kB
VmallocTotal:   34359738367 kB
VmallocUsed:    14110280 kB
VmallocChunk:   34345423564 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      158716 kB
DirectMap2M:     7131136 kB
DirectMap1G:     9437184 kB
behlendorf commented 10 years ago

This does appear to be zfs related, can you include the version of zfs you were running. The Ubuntu PPA was recently updated to include packages for spl/zfs-0.6.2, it would be best if you could update to the latest stable and report if the issue still occurs.

gentoouser commented 10 years ago

This issue did happen on the latest build from the PPA included at the bottom. Please let me know if there is any other information you would like from me.

dpkg -s zfs-dkms

Package: zfs-dkms
Status: install ok installed
Priority: extra
Section: kernel
Installed-Size: 10232
Maintainer: Darik Horn <dajhorn@vanadac.com>
Architecture: amd64
Source: zfs-linux
Version: 0.6.2-1~precise
Replaces: lzfs, lzfs-dkms
Provides: lustre-backend-fs, lzfs, lzfs-dkms
Depends: dkms (>= 2.2.0.2)
Pre-Depends: spl-dkms (>= 0.6.2)

dpkg -s spl-dkms

Package: spl-dkms
Status: install ok installed
Priority: extra
Section: kernel
Installed-Size: 2964
Maintainer: Darik Horn <dajhorn@vanadac.com>
Architecture: all
Source: spl-linux
Version: 0.6.2-1~precise
Depends: dkms (>= 2.2.0.2), lsb-release
Recommends: spl

/etc/apt/sources.list.d/zfs-native-stable-precise.list

deb http://ppa.launchpad.net/zfs-native/stable/ubuntu precise main
deb-src http://ppa.launchpad.net/zfs-native/stable/ubuntu precise main
devsk commented 10 years ago

This is happening very regularly for me too. I think since the time I moved to 3.11 kernel, its been very frequent.

AndrewKvalheim commented 10 years ago

I'm hitting something like this consistently during heavy IO. General protection fault, then txg_sync sits in the D state. I'm not sure when the problem started as this system is usually fairly idle.

behlendorf commented 10 years ago

Thanks for the additional debug, it should help.

tuxoko commented 10 years ago

Looks the same as https://github.com/zfsonlinux/zfs/issues/470

behlendorf commented 8 years ago

This was a duplicate of zfsonlinux/zfs#470 which was resolved.