search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.69k stars 1.75k forks source link

System Hangs/Kernel Panic during rsync #597

Closed comm closed 12 years ago

comm commented 12 years ago

Hi,

I'm running Ubuntu 10.4, Kernel 2.6.32-39, zfs from ppa, and while trying to rsync from another machine over NFS (about 1TB in total) the system hangs. Section of kern.log:

Mar 12 04:20:25 hermes kernel: [42184.724176] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Mar 12 04:20:25 hermes kernel: [42184.724185] IP: [<ffffffff81138131>] __kmalloc+0x91/0x1d0
Mar 12 04:20:25 hermes kernel: [42184.724195] PGD 0 
Mar 12 04:20:25 hermes kernel: [42184.724197] Oops: 0000 [#1] SMP 
Mar 12 04:20:25 hermes kernel: [42184.724200] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host5/target5:0:0/5:0:0:0/block/sdc/uevent
Mar 12 04:20:25 hermes kernel: [42184.724204] CPU 0 
Mar 12 04:20:25 hermes kernel: [42184.724206] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd snd_seq_midi_event exportfs snd_seq lirc_mceusb zfs(P) nfs zcommon(P) znvpair(P) lockd nfs_acl auth_rpcgss lirc_dev snd_timer snd_seq_device zavl(P) zunicode(P) fbcon tileblit sunrpc spl zlib_deflate font bitblit softcursor ppdev nvidia(P) vga16fb vgastate snd edac_core edac_mce_amd parport_pc k8temp psmouse serio_raw i2c_viapro soundcore shpchp lp parport ohci1394 pata_via floppy sata_sil via_velocity crc_ccitt sata_via ieee1394
Mar 12 04:20:25 hermes kernel: [42184.724248] Pid: 1291, comm: z_wr_iss/0 Tainted: P           2.6.32-39-generic #86-Ubuntu  
Mar 12 04:20:25 hermes kernel: [42184.724250] RIP: 0010:[<ffffffff81138131>]  [<ffffffff81138131>] __kmalloc+0x91/0x1d0
Mar 12 04:20:25 hermes kernel: [42184.724255] RSP: 0018:ffff88007b43bcd0  EFLAGS: 00010002
Mar 12 04:20:25 hermes kernel: [42184.724257] RAX: 0000000000000000 RBX: ffffffff817b7be0 RCX: ffffffffa0c899b3
Mar 12 04:20:25 hermes kernel: [42184.724259] RDX: 0000000000000000 RSI: 00000000000080d0 RDI: 0000000000000006
Mar 12 04:20:25 hermes kernel: [42184.724261] RBP: ffff88007b43bd10 R08: ffff880001c120e0 R09: 0000000000000246
Mar 12 04:20:25 hermes kernel: [42184.724263] R10: 0000000000000040 R11: 0000000000000000 R12: 00000000000080d0
Mar 12 04:20:25 hermes kernel: [42184.724266] R13: 0000000000000028 R14: 0000000000000001 R15: 00000000000080d0
Mar 12 04:20:25 hermes kernel: [42184.724269] FS:  00007f3410383700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 12 04:20:25 hermes kernel: [42184.724271] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Mar 12 04:20:25 hermes kernel: [42184.724273] CR2: 0000000000000001 CR3: 000000007c33b000 CR4: 00000000000006f0
Mar 12 04:20:25 hermes kernel: [42184.724275] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 12 04:20:25 hermes kernel: [42184.724277] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 12 04:20:25 hermes kernel: [42184.724280] Process z_wr_iss/0 (pid: 1291, threadinfo ffff88007b43a000, task ffff88007c752de0)
Mar 12 04:20:25 hermes kernel: [42184.724282] Stack:
Mar 12 04:20:25 hermes kernel: [42184.724283]  0000000000000003 ffffffffa0c899b3 ffff88007b43bd00 0000000000000028
Mar 12 04:20:25 hermes kernel: [42184.724287] <0> 0000000000000010 00000000000080d0 ffff88007c0ec000 0000000000000001
Mar 12 04:20:25 hermes kernel: [42184.724291] <0> ffff88007b43bd50 ffffffffa0c899b3 ffffc90000b11328 ffff88003286c580
Mar 12 04:20:25 hermes kernel: [42184.724295] Call Trace:
Mar 12 04:20:25 hermes kernel: [42184.724314]  [<ffffffffa0c899b3>] ? kmem_alloc_debug+0xb3/0x130 [spl]
Mar 12 04:20:25 hermes kernel: [42184.724323]  [<ffffffffa0c899b3>] kmem_alloc_debug+0xb3/0x130 [spl]
Mar 12 04:20:25 hermes kernel: [42184.724391]  [<ffffffffa0e9c706>] vdev_mirror_io_start+0x2e6/0x3d0 [zfs]
Mar 12 04:20:25 hermes kernel: [42184.724427]  [<ffffffffa0e8fa6b>] ? spa_config_enter+0x11b/0x190 [zfs]
Mar 12 04:20:25 hermes kernel: [42184.724464]  [<ffffffffa0ed2720>] zio_vdev_io_start+0x220/0x2e0 [zfs]
Mar 12 04:20:25 hermes kernel: [42184.724498]  [<ffffffffa0ed4fa9>] zio_execute+0x99/0xf0 [zfs]
Mar 12 04:20:25 hermes kernel: [42184.724508]  [<ffffffffa0c8aee6>] taskq_thread+0x1b6/0x310 [spl]
Mar 12 04:20:25 hermes kernel: [42184.724512]  [<ffffffff8105cd70>] ? default_wake_function+0x0/0x20
Mar 12 04:20:25 hermes kernel: [42184.724521]  [<ffffffffa0c8ad30>] ? taskq_thread+0x0/0x310 [spl]
Mar 12 04:20:25 hermes kernel: [42184.724525]  [<ffffffff81084ac6>] kthread+0x96/0xa0
Mar 12 04:20:25 hermes kernel: [42184.724529]  [<ffffffff810131ea>] child_rip+0xa/0x20
Mar 12 04:20:25 hermes kernel: [42184.724532]  [<ffffffff81084a30>] ? kthread+0x0/0xa0
Mar 12 04:20:25 hermes kernel: [42184.724535]  [<ffffffff810131e0>] ? child_rip+0x0/0x20
Mar 12 04:20:25 hermes kernel: [42184.724536] Code: 89 c1 fa 66 66 90 66 66 90 65 8b 04 25 18 e4 00 00 48 98 4c 8b 84 c3 f0 02 00 00 4d 8b 30 45 8b 50 18 4d 85 f6 74 65 41 8b 40 14 <49> 8b 04 c6 49 89 00 4c 89 cf 57 9d 66 66 90 66 90 4d 85 f6 0f 
Mar 12 04:20:25 hermes kernel: [42184.724562] RIP  [<ffffffff81138131>] __kmalloc+0x91/0x1d0
Mar 12 04:20:25 hermes kernel: [42184.724565]  RSP <ffff88007b43bcd0>
Mar 12 04:20:25 hermes kernel: [42184.724567] CR2: 0000000000000001
Mar 12 04:20:25 hermes kernel: [42184.724570] ---[ end trace a6ce2ea61a6c40fb ]---
Mar 12 04:20:56 hermes kernel: [42215.260249] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Mar 12 04:20:56 hermes kernel: [42215.260262] IP: [<ffffffff81120dc3>] remove_vm_area+0x83/0xa0
Mar 12 04:20:56 hermes kernel: [42215.260276] PGD 7c4ef067 PUD 7c4c3067 PMD 0 
Mar 12 04:20:56 hermes kernel: [42215.260284] Oops: 0000 [#2] SMP 
Mar 12 04:20:56 hermes kernel: [42215.260290] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host5/target5:0:0/5:0:0:0/block/sdc/uevent
Mar 12 04:20:56 hermes kernel: [42215.260296] CPU 0 
Mar 12 04:20:56 hermes kernel: [42215.260300] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd snd_seq_midi_event exportfs snd_seq lirc_mceusb zfs(P) nfs zcommon(P) znvpair(P) lockd nfs_acl auth_rpcgss lirc_dev snd_timer snd_seq_device zavl(P) zunicode(P) fbcon tileblit sunrpc spl zlib_deflate font bitblit softcursor ppdev nvidia(P) vga16fb vgastate snd edac_core edac_mce_amd parport_pc k8temp psmouse serio_raw i2c_viapro soundcore shpchp lp parport ohci1394 pata_via floppy sata_sil via_velocity crc_ccitt sata_via ieee1394
Mar 12 04:20:56 hermes kernel: [42215.260383] Pid: 6, comm: events/0 Tainted: P      D    2.6.32-39-generic #86-Ubuntu  
Mar 12 04:20:56 hermes kernel: [42215.260389] RIP: 0010:[<ffffffff81120dc3>]  [<ffffffff81120dc3>] remove_vm_area+0x83/0xa0
Mar 12 04:20:56 hermes kernel: [42215.260399] RSP: 0018:ffff88007dbd1d00  EFLAGS: 00010296
Mar 12 04:20:56 hermes kernel: [42215.260403] RAX: ffff88002bd02780 RBX: ffff880061d87600 RCX: 0000000000000001
Mar 12 04:20:56 hermes kernel: [42215.260408] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff817ccb80
Mar 12 04:20:56 hermes kernel: [42215.260413] RBP: ffff88007dbd1d20 R08: 0000000000000001 R09: ffff88007dbd1d90
Mar 12 04:20:56 hermes kernel: [42215.260417] R10: ffffc90013621018 R11: ffff88007c9366f8 R12: ffff88007dbd1da0
Mar 12 04:20:56 hermes kernel: [42215.260422] R13: 0000000000000001 R14: ffff88007dbd1d78 R15: 0000000000000000
Mar 12 04:20:56 hermes kernel: [42215.260428] FS:  00007f6a24d57700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 12 04:20:56 hermes kernel: [42215.260433] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Mar 12 04:20:56 hermes kernel: [42215.260438] CR2: 0000000000000001 CR3: 0000000037c6a000 CR4: 00000000000006f0
Mar 12 04:20:56 hermes kernel: [42215.260443] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 12 04:20:56 hermes kernel: [42215.260448] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 12 04:20:56 hermes kernel: [42215.260453] Process events/0 (pid: 6, threadinfo ffff88007dbd0000, task ffff88007dbc8000)
Mar 12 04:20:56 hermes kernel: [42215.260457] Stack:
Mar 12 04:20:56 hermes kernel: [42215.260460]  ffff88007dbd1d10 ffff88002bd02780 ffff88007dbd1d30 ffffc90013545000
Mar 12 04:20:56 hermes kernel: [42215.260468] <0> ffff88007dbd1d50 ffffffff81120ea2 ffff88007dbd1d80 ffff88007c936400
Mar 12 04:20:56 hermes kernel: [42215.260476] <0> ffff88007dbd1da0 ffff88007dbd1d90 ffff88007dbd1d60 ffffffff81120ffa
Mar 12 04:20:56 hermes kernel: [42215.260485] Call Trace:
Mar 12 04:20:56 hermes kernel: [42215.260494]  [<ffffffff81120ea2>] __vunmap+0x32/0x110
Mar 12 04:20:56 hermes kernel: [42215.260500]  [<ffffffff81120ffa>] vfree+0x2a/0x30
Mar 12 04:20:56 hermes kernel: [42215.260522]  [<ffffffffa0c88795>] kv_free+0x45/0x50 [spl]
Mar 12 04:20:56 hermes kernel: [42215.260541]  [<ffffffffa0c889c1>] spl_slab_reclaim+0x221/0x2a0 [spl]
Mar 12 04:20:56 hermes kernel: [42215.260559]  [<ffffffffa0c894d0>] ? spl_cache_age+0x0/0x50 [spl]
Mar 12 04:20:56 hermes kernel: [42215.260576]  [<ffffffffa0c894f2>] spl_cache_age+0x22/0x50 [spl]
Mar 12 04:20:56 hermes kernel: [42215.260586]  [<ffffffff81080267>] run_workqueue+0xc7/0x1a0
Mar 12 04:20:56 hermes kernel: [42215.260594]  [<ffffffff810803e3>] worker_thread+0xa3/0x110
Mar 12 04:20:56 hermes kernel: [42215.260602]  [<ffffffff81084e40>] ? autoremove_wake_function+0x0/0x40
Mar 12 04:20:56 hermes kernel: [42215.260609]  [<ffffffff81080340>] ? worker_thread+0x0/0x110
Mar 12 04:20:56 hermes kernel: [42215.260616]  [<ffffffff81084ac6>] kthread+0x96/0xa0
Mar 12 04:20:56 hermes kernel: [42215.260623]  [<ffffffff810131ea>] child_rip+0xa/0x20
Mar 12 04:20:56 hermes kernel: [42215.260630]  [<ffffffff81084a30>] ? kthread+0x0/0xa0
Mar 12 04:20:56 hermes kernel: [42215.260635]  [<ffffffff810131e0>] ? child_rip+0x0/0x20
Mar 12 04:20:56 hermes kernel: [42215.260639] Code: 48 89 45 e8 48 c7 c7 80 cb 7c 81 e8 58 70 42 00 48 8b 15 61 af 8d 00 48 c7 c1 10 bd 9f 81 48 8b 45 e8 48 39 d3 74 0c 90 48 89 d1 <48> 8b 12 48 39 d3 75 f5 48 8b 13 48 89 11 3e 81 05 a4 bd 6a 00 
Mar 12 04:20:56 hermes kernel: [42215.260695] RIP  [<ffffffff81120dc3>] remove_vm_area+0x83/0xa0
Mar 12 04:20:56 hermes kernel: [42215.260702]  RSP <ffff88007dbd1d00>
Mar 12 04:20:56 hermes kernel: [42215.260705] CR2: 0000000000000001
Mar 12 04:20:56 hermes kernel: [42215.260710] ---[ end trace a6ce2ea61a6c40fc ]---
Mar 12 04:21:00 hermes kernel: [42219.236629] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Mar 12 04:21:00 hermes kernel: [42219.236641] IP: [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 12 04:21:00 hermes kernel: [42219.236656] PGD 7c4ef067 PUD 7c4c3067 PMD 0 
Mar 12 04:21:00 hermes kernel: [42219.236664] Oops: 0000 [#3] SMP 
Mar 12 04:21:00 hermes kernel: [42219.236670] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host5/target5:0:0/5:0:0:0/block/sdc/uevent
Mar 12 04:21:00 hermes kernel: [42219.236676] CPU 0 
Mar 12 04:21:00 hermes kernel: [42219.236679] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd snd_seq_midi_event exportfs snd_seq lirc_mceusb zfs(P) nfs zcommon(P) znvpair(P) lockd nfs_acl auth_rpcgss lirc_dev snd_timer snd_seq_device zavl(P) zunicode(P) fbcon tileblit sunrpc spl zlib_deflate font bitblit softcursor ppdev nvidia(P) vga16fb vgastate snd edac_core edac_mce_amd parport_pc k8temp psmouse serio_raw i2c_viapro soundcore shpchp lp parport ohci1394 pata_via floppy sata_sil via_velocity crc_ccitt sata_via ieee1394
Mar 12 04:21:00 hermes kernel: [42219.236763] Pid: 2225, comm: gam_server Tainted: P      D    2.6.32-39-generic #86-Ubuntu  
Mar 12 04:21:00 hermes kernel: [42219.236768] RIP: 0010:[<ffffffff81137202>]  [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 12 04:21:00 hermes kernel: [42219.236779] RSP: 0018:ffff88007c4c1b58  EFLAGS: 00010002
Mar 12 04:21:00 hermes kernel: [42219.236784] RAX: 0000000000000000 RBX: ffffffff817b7be0 RCX: ffffffffa0dde55d
Mar 12 04:21:00 hermes kernel: [42219.236788] RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff817b7be0
Mar 12 04:21:00 hermes kernel: [42219.236793] RBP: ffff88007c4c1b98 R08: ffff880001c120e0 R09: 0000000000000040
Mar 12 04:21:00 hermes kernel: [42219.236798] R10: ffff880075afcc60 R11: 0000000000000000 R12: 0000000000000001
Mar 12 04:21:00 hermes kernel: [42219.236802] R13: 00000000000000d0 R14: 00000000000000d0 R15: 0000000000000246
Mar 12 04:21:00 hermes kernel: [42219.236809] FS:  00007f6a24d57700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 12 04:21:00 hermes kernel: [42219.236814] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 12 04:21:00 hermes kernel: [42219.236818] CR2: 0000000000000001 CR3: 0000000037c6a000 CR4: 00000000000006f0
Mar 12 04:21:00 hermes kernel: [42219.236823] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 12 04:21:00 hermes kernel: [42219.236828] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 12 04:21:00 hermes kernel: [42219.236834] Process gam_server (pid: 2225, threadinfo ffff88007c4c0000, task ffff88007c4ddbc0)
Mar 12 04:21:00 hermes kernel: [42219.236838] Stack:
Mar 12 04:21:00 hermes kernel: [42219.236840]  0000000000000007 ffffffffa0dde55d ffff880075afceb8 ffff88007c4c1c08
Mar 12 04:21:00 hermes kernel: [42219.236848] <0> ffff880075afce08 ffff88007b717680 0000000000000001 ffff88007c4c1d18
Mar 12 04:21:00 hermes kernel: [42219.236856] <0> ffff88007c4c1be8 ffffffffa0dde55d ffff88007c4c1ac8 000000000000001f
Mar 12 04:21:00 hermes kernel: [42219.236866] Call Trace:
Mar 12 04:21:00 hermes kernel: [42219.236894]  [<ffffffffa0dde55d>] ? nfs_access_add_cache+0x2d/0x250 [nfs]
Mar 12 04:21:00 hermes kernel: [42219.236915]  [<ffffffffa0dde55d>] nfs_access_add_cache+0x2d/0x250 [nfs]
Mar 12 04:21:00 hermes kernel: [42219.236935]  [<ffffffffa0dde86b>] nfs_do_access+0xeb/0xf0 [nfs]
Mar 12 04:21:00 hermes kernel: [42219.236968]  [<ffffffffa0cbe855>] ? generic_lookup_cred+0x15/0x20 [sunrpc]
Mar 12 04:21:00 hermes kernel: [42219.236992]  [<ffffffffa0cbd970>] ? rpcauth_lookupcred+0x70/0xc0 [sunrpc]
Mar 12 04:21:00 hermes kernel: [42219.237012]  [<ffffffffa0dde915>] nfs_permission+0xa5/0x1e0 [nfs]
Mar 12 04:21:00 hermes kernel: [42219.237022]  [<ffffffff81150869>] __link_path_walk+0x99/0xf80
Mar 12 04:21:00 hermes kernel: [42219.237029]  [<ffffffff811567b0>] ? pollwake+0x0/0x60
Mar 12 04:21:00 hermes kernel: [42219.237036]  [<ffffffff811567b0>] ? pollwake+0x0/0x60
Mar 12 04:21:00 hermes kernel: [42219.237043]  [<ffffffff811519ca>] path_walk+0x6a/0xe0
Mar 12 04:21:00 hermes kernel: [42219.237050]  [<ffffffff81151b9b>] do_path_lookup+0x5b/0xa0
Mar 12 04:21:00 hermes kernel: [42219.237056]  [<ffffffff81152867>] user_path_at+0x57/0xa0
Mar 12 04:21:00 hermes kernel: [42219.237063]  [<ffffffff81148c44>] ? cp_new_stat+0xe4/0x100
Mar 12 04:21:00 hermes kernel: [42219.237069]  [<ffffffff81148e6c>] vfs_fstatat+0x3c/0x80
Mar 12 04:21:00 hermes kernel: [42219.237077]  [<ffffffff8108f63b>] ? ktime_get_ts+0xab/0xe0
Mar 12 04:21:00 hermes kernel: [42219.237083]  [<ffffffff81148fdb>] vfs_stat+0x1b/0x20
Mar 12 04:21:00 hermes kernel: [42219.237088]  [<ffffffff81149004>] sys_newstat+0x24/0x50
Mar 12 04:21:00 hermes kernel: [42219.237095]  [<ffffffff8108f3aa>] ? do_gettimeofday+0x1a/0x50
Mar 12 04:21:00 hermes kernel: [42219.237102]  [<ffffffff8106c160>] ? sys_gettimeofday+0x40/0x90
Mar 12 04:21:00 hermes kernel: [42219.237111]  [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
Mar 12 04:21:00 hermes kernel: [42219.237115] Code: 89 c7 fa 66 66 90 66 66 90 65 8b 04 25 18 e4 00 00 48 98 4c 8b 84 c3 f0 02 00 00 4d 8b 20 45 8b 48 18 4d 85 e4 74 64 41 8b 40 14 <49> 8b 04 c4 49 89 00 4c 89 ff 57 9d 66 66 90 66 90 4d 85 e4 0f 
Mar 12 04:21:00 hermes kernel: [42219.237171] RIP  [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 12 04:21:00 hermes kernel: [42219.237179]  RSP <ffff88007c4c1b58>
Mar 12 04:21:00 hermes kernel: [42219.237182] CR2: 0000000000000001
Mar 12 04:21:00 hermes kernel: [42219.237188] ---[ end trace a6ce2ea61a6c40fd ]---```

This log section can be repeated several times until panic/hang. Any thoughts?

Thanks in advance!

Comm
behlendorf commented 12 years ago

Are you running the stable of daily ppa? Please make sure your running the daily for now, the stable should be updated fairly soon one I tag a new release candidate.

comm commented 12 years ago

I changed to the daily just now (now at 0.6.0.53) and ran the rsync again. It lasted longer this time, but still died with similar errors:

Mar 13 00:44:14 hermes kernel: [24695.476469] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Mar 13 00:44:14 hermes kernel: [24695.476478] IP: [<ffffffff81138131>] __kmalloc+0x91/0x1d0
Mar 13 00:44:14 hermes kernel: [24695.476488] PGD 62ab3067 PUD 62ab4067 PMD 0 
Mar 13 00:44:14 hermes kernel: [24695.476492] Oops: 0000 [#1] SMP 
Mar 13 00:44:14 hermes kernel: [24695.476495] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host0/target0:0:0/0:0:0:0/block/sda/uevent
Mar 13 00:44:14 hermes kernel: [24695.476499] CPU 0 
Mar 13 00:44:14 hermes kernel: [24695.476500] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd exportfs snd_seq_midi_event nfs snd_seq lirc_mceusb zfs(P) lockd nfs_acl zcommon(P) lirc_dev auth_rpcgss snd_timer snd_seq_device znvpair(P) zavl(P) zunicode(P) sunrpc spl zlib_deflate fbcon tileblit font bitblit softcursor nvidia(P) ppdev snd vga16fb vgastate parport_pc psmouse serio_raw edac_core shpchp i2c_viapro soundcore edac_mce_amd k8temp lp parport ohci1394 floppy sata_via pata_via ieee1394 sata_sil via_velocity crc_ccitt
Mar 13 00:44:14 hermes kernel: [24695.476542] Pid: 1302, comm: z_wr_iss/0 Tainted: P           2.6.32-39-generic #86-Ubuntu  
Mar 13 00:44:14 hermes kernel: [24695.476544] RIP: 0010:[<ffffffff81138131>]  [<ffffffff81138131>] __kmalloc+0x91/0x1d0
Mar 13 00:44:14 hermes kernel: [24695.476549] RSP: 0018:ffff880064375cd0  EFLAGS: 00010002
Mar 13 00:44:14 hermes kernel: [24695.476551] RAX: 0000000000000000 RBX: ffffffff817b7be0 RCX: ffffffffa0c959b3
Mar 13 00:44:14 hermes kernel: [24695.476553] RDX: 0000000000000000 RSI: 00000000000080d0 RDI: 0000000000000006
Mar 13 00:44:14 hermes kernel: [24695.476555] RBP: ffff880064375d10 R08: ffff880001c120e0 R09: 0000000000000246
Mar 13 00:44:14 hermes kernel: [24695.476558] R10: 0000000000000040 R11: 0000000000000000 R12: 00000000000080d0
Mar 13 00:44:14 hermes kernel: [24695.476560] R13: 0000000000000028 R14: 0000000000000001 R15: 00000000000080d0
Mar 13 00:44:14 hermes kernel: [24695.476563] FS:  00007f45fbaf8700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 13 00:44:14 hermes kernel: [24695.476565] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Mar 13 00:44:14 hermes kernel: [24695.476567] CR2: 0000000000000001 CR3: 0000000062ab2000 CR4: 00000000000006f0
Mar 13 00:44:14 hermes kernel: [24695.476570] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 13 00:44:14 hermes kernel: [24695.476572] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 13 00:44:14 hermes kernel: [24695.476574] Process z_wr_iss/0 (pid: 1302, threadinfo ffff880064374000, task ffff880064378000)
Mar 13 00:44:14 hermes kernel: [24695.476576] Stack:
Mar 13 00:44:14 hermes kernel: [24695.476578]  0000000000000003 ffffffffa0c959b3 ffff880064375d00 0000000000000028
Mar 13 00:44:14 hermes kernel: [24695.476581] <0> 0000000000000010 00000000000080d0 ffff88006744b000 0000000000000001
Mar 13 00:44:14 hermes kernel: [24695.476585] <0> ffff880064375d50 ffffffffa0c959b3 ffffc900095a8c18 ffff88001bcc1bd0
Mar 13 00:44:14 hermes kernel: [24695.476590] Call Trace:
Mar 13 00:44:14 hermes kernel: [24695.476608]  [<ffffffffa0c959b3>] ? kmem_alloc_debug+0xb3/0x130 [spl]
Mar 13 00:44:14 hermes kernel: [24695.476616]  [<ffffffffa0c959b3>] kmem_alloc_debug+0xb3/0x130 [spl]
Mar 13 00:44:14 hermes kernel: [24695.476686]  [<ffffffffa0e21706>] vdev_mirror_io_start+0x2e6/0x3d0 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.476721]  [<ffffffffa0e14a6b>] ? spa_config_enter+0x11b/0x190 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.476758]  [<ffffffffa0e57720>] zio_vdev_io_start+0x220/0x2e0 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.476792]  [<ffffffffa0e59fa9>] zio_execute+0x99/0xf0 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.476802]  [<ffffffffa0c96ee6>] taskq_thread+0x1b6/0x310 [spl]
Mar 13 00:44:14 hermes kernel: [24695.476806]  [<ffffffff8105cd70>] ? default_wake_function+0x0/0x20
Mar 13 00:44:14 hermes kernel: [24695.476814]  [<ffffffffa0c96d30>] ? taskq_thread+0x0/0x310 [spl]
Mar 13 00:44:14 hermes kernel: [24695.476819]  [<ffffffff81084ac6>] kthread+0x96/0xa0
Mar 13 00:44:14 hermes kernel: [24695.476823]  [<ffffffff810131ea>] child_rip+0xa/0x20
Mar 13 00:44:14 hermes kernel: [24695.476826]  [<ffffffff81084a30>] ? kthread+0x0/0xa0
Mar 13 00:44:14 hermes kernel: [24695.476828]  [<ffffffff810131e0>] ? child_rip+0x0/0x20
Mar 13 00:44:14 hermes kernel: [24695.476830] Code: 89 c1 fa 66 66 90 66 66 90 65 8b 04 25 18 e4 00 00 48 98 4c 8b 84 c3 f0 02 00 00 4d 8b 30 45 8b 50 18 4d 85 f6 74 65 41 8b 40 14 <49> 8b 04 c6 49 89 00 4c 89 cf 57 9d 66 66 90 66 90 4d 85 f6 0f 
Mar 13 00:44:14 hermes kernel: [24695.476855] RIP  [<ffffffff81138131>] __kmalloc+0x91/0x1d0
Mar 13 00:44:14 hermes kernel: [24695.476859]  RSP <ffff880064375cd0>
Mar 13 00:44:14 hermes kernel: [24695.476861] CR2: 0000000000000001
Mar 13 00:44:14 hermes kernel: [24695.476864] ---[ end trace 91526fd8d79f024e ]---
Mar 13 00:44:14 hermes kernel: [24695.946174] BUG: unable to handle kernel paging request at ffff88007fef0008
Mar 13 00:44:14 hermes kernel: [24695.946182] IP: [<ffffffffa0e21142>] vdev_mirror_io_done+0x52/0x330 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.946242] PGD 1002063 PUD 10067 PMD 11067 PTE 0
Mar 13 00:44:14 hermes kernel: [24695.946246] Oops: 0000 [#2] SMP 
Mar 13 00:44:14 hermes kernel: [24695.946249] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host0/target0:0:0/0:0:0:0/block/sda/uevent
Mar 13 00:44:14 hermes kernel: [24695.946252] CPU 0 
Mar 13 00:44:14 hermes kernel: [24695.946254] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd exportfs snd_seq_midi_event nfs snd_seq lirc_mceusb zfs(P) lockd nfs_acl zcommon(P) lirc_dev auth_rpcgss snd_timer snd_seq_device znvpair(P) zavl(P) zunicode(P) sunrpc spl zlib_deflate fbcon tileblit font bitblit softcursor nvidia(P) ppdev snd vga16fb vgastate parport_pc psmouse serio_raw edac_core shpchp i2c_viapro soundcore edac_mce_amd k8temp lp parport ohci1394 floppy sata_via pata_via ieee1394 sata_sil via_velocity crc_ccitt
Mar 13 00:44:14 hermes kernel: [24695.946294] Pid: 1313, comm: z_wr_int/5 Tainted: P      D    2.6.32-39-generic #86-Ubuntu  
Mar 13 00:44:14 hermes kernel: [24695.946296] RIP: 0010:[<ffffffffa0e21142>]  [<ffffffffa0e21142>] vdev_mirror_io_done+0x52/0x330 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.946332] RSP: 0018:ffff8800643a3cb0  EFLAGS: 00010287
Mar 13 00:44:14 hermes kernel: [24695.946334] RAX: 0000000055ac21e8 RBX: ffff88002a42de00 RCX: 00000003f6449a00
Mar 13 00:44:14 hermes kernel: [24695.946336] RDX: 0000000000030646 RSI: 000000002a42dbc0 RDI: ffffc90000b01570
Mar 13 00:44:14 hermes kernel: [24695.946338] RBP: ffff8800643a3d20 R08: 0000000000000008 R09: 0000000000000000
Mar 13 00:44:14 hermes kernel: [24695.946340] R10: 0000000000000001 R11: 0000000000000000 R12: ffffc90000b01570
Mar 13 00:44:14 hermes kernel: [24695.946343] R13: 00000000003cf520 R14: ffffc90000bf3c30 R15: ffff8800643996f0
Mar 13 00:44:14 hermes kernel: [24695.946346] FS:  00007fa46fca0700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 13 00:44:14 hermes kernel: [24695.946348] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Mar 13 00:44:14 hermes kernel: [24695.946350] CR2: ffff88007fef0008 CR3: 000000006765a000 CR4: 00000000000006f0
Mar 13 00:44:14 hermes kernel: [24695.946352] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 13 00:44:14 hermes kernel: [24695.946355] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 13 00:44:14 hermes kernel: [24695.946357] Process z_wr_int/5 (pid: 1313, threadinfo ffff8800643a2000, task ffff8800643996f0)
Mar 13 00:44:14 hermes kernel: [24695.946359] Stack:
Mar 13 00:44:14 hermes kernel: [24695.946360]  ffff8800643a3cf0 ffffffff810590a6 ffff8800643a3d00 ffff8800643996f0
Mar 13 00:44:14 hermes kernel: [24695.946364] <0> ffff8800643a3cf0 ffffffff815467de ffffc90000b01570 ffffc90000b01878
Mar 13 00:44:14 hermes kernel: [24695.946368] <0> ffff8800643a3d20 0000000000000000 ffffc90000b01570 ffffffffa0e84240
Mar 13 00:44:14 hermes kernel: [24695.946372] Call Trace:
Mar 13 00:44:14 hermes kernel: [24695.946380]  [<ffffffff810590a6>] ? update_curr+0xe6/0x1e0
Mar 13 00:44:14 hermes kernel: [24695.946385]  [<ffffffff815467de>] ? mutex_lock+0x1e/0x50
Mar 13 00:44:14 hermes kernel: [24695.946422]  [<ffffffffa0e56844>] zio_vdev_io_done+0x64/0x190 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.946457]  [<ffffffffa0e5af0e>] zio_done+0x68e/0xc00 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.946491]  [<ffffffffa0e5af0e>] zio_done+0x68e/0xc00 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.946525]  [<ffffffffa0e59fa9>] zio_execute+0x99/0xf0 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.946539]  [<ffffffffa0c96ee6>] taskq_thread+0x1b6/0x310 [spl]
Mar 13 00:44:14 hermes kernel: [24695.946543]  [<ffffffff8105cd70>] ? default_wake_function+0x0/0x20
Mar 13 00:44:14 hermes kernel: [24695.946551]  [<ffffffffa0c96d30>] ? taskq_thread+0x0/0x310 [spl]
Mar 13 00:44:14 hermes kernel: [24695.946555]  [<ffffffff81084ac6>] kthread+0x96/0xa0
Mar 13 00:44:14 hermes kernel: [24695.946559]  [<ffffffff810131ea>] child_rip+0xa/0x20
Mar 13 00:44:14 hermes kernel: [24695.946562]  [<ffffffff81084a30>] ? kthread+0x0/0xa0
Mar 13 00:44:14 hermes kernel: [24695.946565]  [<ffffffff810131e0>] ? child_rip+0x0/0x20
Mar 13 00:44:14 hermes kernel: [24695.946566] Code: 85 f6 7e 42 8d 46 ff 48 8d 4c 40 03 31 c0 48 c1 e1 03 eb 17 0f 1f 44 00 00 80 7c 03 25 01 41 83 d5 00 48 83 c0 18 48 39 c8 74 1b <44> 8b 44 03 20 45 85 c0 75 e4 80 7c 03 24 01 83 da ff 48 83 c0 
Mar 13 00:44:14 hermes kernel: [24695.946592] RIP  [<ffffffffa0e21142>] vdev_mirror_io_done+0x52/0x330 [zfs]
Mar 13 00:44:14 hermes kernel: [24695.946626]  RSP <ffff8800643a3cb0>
Mar 13 00:44:14 hermes kernel: [24695.946628] CR2: ffff88007fef0008
Mar 13 00:44:14 hermes kernel: [24695.946631] ---[ end trace 91526fd8d79f024f ]---
Mar 13 00:44:51 hermes kernel: [24732.731727] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Mar 13 00:44:51 hermes kernel: [24732.731740] IP: [<ffffffff81120dc3>] remove_vm_area+0x83/0xa0
Mar 13 00:44:51 hermes kernel: [24732.731756] PGD 5f567067 PUD 68d9e067 PMD 0 
Mar 13 00:44:51 hermes kernel: [24732.731764] Oops: 0000 [#3] SMP 
Mar 13 00:44:51 hermes kernel: [24732.731770] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host0/target0:0:0/0:0:0:0/block/sda/uevent
Mar 13 00:44:51 hermes kernel: [24732.731776] CPU 0 
Mar 13 00:44:51 hermes kernel: [24732.731779] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd exportfs snd_seq_midi_event nfs snd_seq lirc_mceusb zfs(P) lockd nfs_acl zcommon(P) lirc_dev auth_rpcgss snd_timer snd_seq_device znvpair(P) zavl(P) zunicode(P) sunrpc spl zlib_deflate fbcon tileblit font bitblit softcursor nvidia(P) ppdev snd vga16fb vgastate parport_pc psmouse serio_raw edac_core shpchp i2c_viapro soundcore edac_mce_amd k8temp lp parport ohci1394 floppy sata_via pata_via ieee1394 sata_sil via_velocity crc_ccitt
Mar 13 00:44:51 hermes kernel: [24732.731863] Pid: 6, comm: events/0 Tainted: P      D    2.6.32-39-generic #86-Ubuntu  
Mar 13 00:44:51 hermes kernel: [24732.731868] RIP: 0010:[<ffffffff81120dc3>]  [<ffffffff81120dc3>] remove_vm_area+0x83/0xa0
Mar 13 00:44:51 hermes kernel: [24732.731878] RSP: 0018:ffff88007dbd1d00  EFLAGS: 00010292
Mar 13 00:44:51 hermes kernel: [24732.731882] RAX: ffff880013a78200 RBX: ffff880017e3c180 RCX: 0000000000000001
Mar 13 00:44:51 hermes kernel: [24732.731887] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff817ccb80
Mar 13 00:44:51 hermes kernel: [24732.731892] RBP: ffff88007dbd1d20 R08: 0000000000000002 R09: ffffc9001b95b018
Mar 13 00:44:51 hermes kernel: [24732.731897] R10: ffffc9001b91f018 R11: ffff880069e712f8 R12: ffff88007dbd1da0
Mar 13 00:44:51 hermes kernel: [24732.731901] R13: 0000000000000001 R14: ffff88007dbd1d78 R15: 0000000000000000
Mar 13 00:44:51 hermes kernel: [24732.731908] FS:  00007fa632c5d700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 13 00:44:51 hermes kernel: [24732.731913] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Mar 13 00:44:51 hermes kernel: [24732.731917] CR2: 0000000000000001 CR3: 0000000069e30000 CR4: 00000000000006f0
Mar 13 00:44:51 hermes kernel: [24732.731922] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 13 00:44:51 hermes kernel: [24732.731927] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 13 00:44:51 hermes kernel: [24732.731933] Process events/0 (pid: 6, threadinfo ffff88007dbd0000, task ffff88007dbc8000)
Mar 13 00:44:51 hermes kernel: [24732.731937] Stack:
Mar 13 00:44:51 hermes kernel: [24732.731940]  ffff88007dbd1d10 ffff880013a78200 ffff88007dbd1d30 ffffc9001b82d000
Mar 13 00:44:51 hermes kernel: [24732.731947] <0> ffff88007dbd1d50 ffffffff81120ea2 ffff88007dbd1d80 ffff880069e71000
Mar 13 00:44:51 hermes kernel: [24732.731956] <0> ffff88007dbd1da0 ffffc9001b92f000 ffff88007dbd1d60 ffffffff81120ffa
Mar 13 00:44:51 hermes kernel: [24732.731965] Call Trace:
Mar 13 00:44:51 hermes kernel: [24732.731973]  [<ffffffff81120ea2>] __vunmap+0x32/0x110
Mar 13 00:44:51 hermes kernel: [24732.731979]  [<ffffffff81120ffa>] vfree+0x2a/0x30
Mar 13 00:44:51 hermes kernel: [24732.732004]  [<ffffffffa0c94795>] kv_free+0x45/0x50 [spl]
Mar 13 00:44:51 hermes kernel: [24732.732022]  [<ffffffffa0c949c1>] spl_slab_reclaim+0x221/0x2a0 [spl]
Mar 13 00:44:51 hermes kernel: [24732.732040]  [<ffffffffa0c954d0>] ? spl_cache_age+0x0/0x50 [spl]
Mar 13 00:44:51 hermes kernel: [24732.732057]  [<ffffffffa0c954f2>] spl_cache_age+0x22/0x50 [spl]
Mar 13 00:44:51 hermes kernel: [24732.732067]  [<ffffffff81080267>] run_workqueue+0xc7/0x1a0
Mar 13 00:44:51 hermes kernel: [24732.732075]  [<ffffffff810803e3>] worker_thread+0xa3/0x110
Mar 13 00:44:51 hermes kernel: [24732.732083]  [<ffffffff81084e40>] ? autoremove_wake_function+0x0/0x40
Mar 13 00:44:51 hermes kernel: [24732.732091]  [<ffffffff81080340>] ? worker_thread+0x0/0x110
Mar 13 00:44:51 hermes kernel: [24732.732097]  [<ffffffff81084ac6>] kthread+0x96/0xa0
Mar 13 00:44:51 hermes kernel: [24732.732105]  [<ffffffff810131ea>] child_rip+0xa/0x20
Mar 13 00:44:51 hermes kernel: [24732.732111]  [<ffffffff81084a30>] ? kthread+0x0/0xa0
Mar 13 00:44:51 hermes kernel: [24732.732117]  [<ffffffff810131e0>] ? child_rip+0x0/0x20
Mar 13 00:44:51 hermes kernel: [24732.732120] Code: 48 89 45 e8 48 c7 c7 80 cb 7c 81 e8 58 70 42 00 48 8b 15 61 af 8d 00 48 c7 c1 10 bd 9f 81 48 8b 45 e8 48 39 d3 74 0c 90 48 89 d1 <48> 8b 12 48 39 d3 75 f5 48 8b 13 48 89 11 3e 81 05 a4 bd 6a 00 
Mar 13 00:44:51 hermes kernel: [24732.732176] RIP  [<ffffffff81120dc3>] remove_vm_area+0x83/0xa0
Mar 13 00:44:51 hermes kernel: [24732.732183]  RSP <ffff88007dbd1d00>
Mar 13 00:44:51 hermes kernel: [24732.732186] CR2: 0000000000000001
Mar 13 00:44:51 hermes kernel: [24732.732192] ---[ end trace 91526fd8d79f0250 ]---
Mar 13 00:53:14 hermes kernel: [25235.106891] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Mar 13 00:53:14 hermes kernel: [25235.106904] IP: [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 13 00:53:14 hermes kernel: [25235.106920] PGD 69c0a067 PUD 5e243067 PMD 0 
Mar 13 00:53:14 hermes kernel: [25235.106928] Oops: 0000 [#4] SMP 
Mar 13 00:53:14 hermes kernel: [25235.106934] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host0/target0:0:0/0:0:0:0/block/sda/uevent
Mar 13 00:53:14 hermes kernel: [25235.106941] CPU 0 
Mar 13 00:53:14 hermes kernel: [25235.106944] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd exportfs snd_seq_midi_event nfs snd_seq lirc_mceusb zfs(P) lockd nfs_acl zcommon(P) lirc_dev auth_rpcgss snd_timer snd_seq_device znvpair(P) zavl(P) zunicode(P) sunrpc spl zlib_deflate fbcon tileblit font bitblit softcursor nvidia(P) ppdev snd vga16fb vgastate parport_pc psmouse serio_raw edac_core shpchp i2c_viapro soundcore edac_mce_amd k8temp lp parport ohci1394 floppy sata_via pata_via ieee1394 sata_sil via_velocity crc_ccitt
Mar 13 00:53:14 hermes kernel: [25235.107027] Pid: 2249, comm: udisks-daemon Tainted: P      D    2.6.32-39-generic #86-Ubuntu  
Mar 13 00:53:14 hermes kernel: [25235.107033] RIP: 0010:[<ffffffff81137202>]  [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 13 00:53:14 hermes kernel: [25235.107044] RSP: 0018:ffff88005bc4dd90  EFLAGS: 00010002
Mar 13 00:53:14 hermes kernel: [25235.107048] RAX: 0000000000000000 RBX: ffffffff817b7be0 RCX: ffffffff8116c710
Mar 13 00:53:14 hermes kernel: [25235.107053] RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff817b7be0
Mar 13 00:53:14 hermes kernel: [25235.107058] RBP: ffff88005bc4ddd0 R08: ffff880001c120e0 R09: 0000000000000040
Mar 13 00:53:14 hermes kernel: [25235.107063] R10: 0000000000000001 R11: 8000000000000000 R12: 0000000000000001
Mar 13 00:53:14 hermes kernel: [25235.107067] R13: 00000000000000d0 R14: 00000000000000d0 R15: 0000000000000246
Mar 13 00:53:14 hermes kernel: [25235.107073] FS:  00007f5d9bd147a0(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 13 00:53:14 hermes kernel: [25235.107079] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 13 00:53:14 hermes kernel: [25235.107083] CR2: 0000000000000001 CR3: 0000000069c08000 CR4: 00000000000006f0
Mar 13 00:53:14 hermes kernel: [25235.107088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 13 00:53:14 hermes kernel: [25235.107093] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 13 00:53:14 hermes kernel: [25235.107099] Process udisks-daemon (pid: 2249, threadinfo ffff88005bc4c000, task ffff88005e1644d0)
Mar 13 00:53:14 hermes kernel: [25235.107103] Stack:
Mar 13 00:53:14 hermes kernel: [25235.107106]  ffff880000000000 ffffffff8116c710 0000000000000008 ffff88005f641840
Mar 13 00:53:14 hermes kernel: [25235.107113] <0> ffff88007b2944d0 0000000000000000 ffff88005bc4df58 00007f5d9bd14a70
Mar 13 00:53:14 hermes kernel: [25235.107122] <0> ffff88005bc4ddf0 ffffffff8116c710 0000000001200011 ffff88007b2944d0
Mar 13 00:53:14 hermes kernel: [25235.107131] Call Trace:
Mar 13 00:53:14 hermes kernel: [25235.107142]  [<ffffffff8116c710>] ? copy_fs_struct+0x20/0x90
Mar 13 00:53:14 hermes kernel: [25235.107149]  [<ffffffff8116c710>] copy_fs_struct+0x20/0x90
Mar 13 00:53:14 hermes kernel: [25235.107158]  [<ffffffff81064f3b>] copy_process+0x5cb/0xe70
Mar 13 00:53:14 hermes kernel: [25235.107165]  [<ffffffff811372ce>] ? kmem_cache_alloc+0x13e/0x150
Mar 13 00:53:14 hermes kernel: [25235.107171]  [<ffffffff81065874>] do_fork+0x94/0x440
Mar 13 00:53:14 hermes kernel: [25235.107179]  [<ffffffff8115e8ca>] ? alloc_fd+0x10a/0x150
Mar 13 00:53:14 hermes kernel: [25235.107188]  [<ffffffff81547dee>] ? _spin_lock+0xe/0x20
Mar 13 00:53:14 hermes kernel: [25235.107195]  [<ffffffff811420c7>] ? fd_install+0x67/0x90
Mar 13 00:53:14 hermes kernel: [25235.107202]  [<ffffffff8114dbdf>] ? do_pipe_flags+0xcf/0x130
Mar 13 00:53:14 hermes kernel: [25235.107211]  [<ffffffff81010568>] sys_clone+0x28/0x30
Mar 13 00:53:14 hermes kernel: [25235.107218]  [<ffffffff810124d3>] stub_clone+0x13/0x20
Mar 13 00:53:14 hermes kernel: [25235.107225]  [<ffffffff810121b2>] ? system_call_fastpath+0x16/0x1b
Mar 13 00:53:14 hermes kernel: [25235.107230] Code: 89 c7 fa 66 66 90 66 66 90 65 8b 04 25 18 e4 00 00 48 98 4c 8b 84 c3 f0 02 00 00 4d 8b 20 45 8b 48 18 4d 85 e4 74 64 41 8b 40 14 <49> 8b 04 c4 49 89 00 4c 89 ff 57 9d 66 66 90 66 90 4d 85 e4 0f 
Mar 13 00:53:14 hermes kernel: [25235.107286] RIP  [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 13 00:53:14 hermes kernel: [25235.107294]  RSP <ffff88005bc4dd90>
Mar 13 00:53:14 hermes kernel: [25235.107297] CR2: 0000000000000001
Mar 13 00:53:14 hermes kernel: [25235.107302] ---[ end trace 91526fd8d79f0251 ]---
Mar 13 00:53:14 hermes kernel: [25235.150738] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Mar 13 00:53:14 hermes kernel: [25235.150751] IP: [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 13 00:53:14 hermes kernel: [25235.150766] PGD 629bf067 PUD 63a37067 PMD 0 
Mar 13 00:53:14 hermes kernel: [25235.150774] Oops: 0000 [#5] SMP 
Mar 13 00:53:14 hermes kernel: [25235.150779] last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/host3/target3:0:0/3:0:0:0/block/sdd/uevent
Mar 13 00:53:14 hermes kernel: [25235.150786] CPU 0 
Mar 13 00:53:14 hermes kernel: [25235.150789] Modules linked in: autofs4 binfmt_misc snd_via82xx gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_mpu401_uart snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi nfsd exportfs snd_seq_midi_event nfs snd_seq lirc_mceusb zfs(P) lockd nfs_acl zcommon(P) lirc_dev auth_rpcgss snd_timer snd_seq_device znvpair(P) zavl(P) zunicode(P) sunrpc spl zlib_deflate fbcon tileblit font bitblit softcursor nvidia(P) ppdev snd vga16fb vgastate parport_pc psmouse serio_raw edac_core shpchp i2c_viapro soundcore edac_mce_amd k8temp lp parport ohci1394 floppy sata_via pata_via ieee1394 sata_sil via_velocity crc_ccitt
Mar 13 00:53:14 hermes kernel: [25235.150872] Pid: 31924, comm: udisks-helper-a Tainted: P      D    2.6.32-39-generic #86-Ubuntu  
Mar 13 00:53:14 hermes kernel: [25235.150877] RIP: 0010:[<ffffffff81137202>]  [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 13 00:53:14 hermes kernel: [25235.150887] RSP: 0018:ffff88007dbd1c68  EFLAGS: 00010002
Mar 13 00:53:14 hermes kernel: [25235.150891] RAX: 0000000000000000 RBX: ffffffff817b7be0 RCX: ffffffff811af82e
Mar 13 00:53:14 hermes kernel: [25235.150896] RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff817b7be0
Mar 13 00:53:14 hermes kernel: [25235.150901] RBP: ffff88007dbd1ca8 R08: ffff880001c120e0 R09: 0000000000000040
Mar 13 00:53:14 hermes kernel: [25235.150906] R10: ffff88007d4a7cc0 R11: 0000000000000000 R12: 0000000000000001
Mar 13 00:53:14 hermes kernel: [25235.150910] R13: 00000000000000d0 R14: 00000000000000d0 R15: 0000000000000246
Mar 13 00:53:14 hermes kernel: [25235.150916] FS:  00007f14771b4700(0000) GS:ffff880001c00000(0000) knlGS:0000000000000000
Mar 13 00:53:14 hermes kernel: [25235.150922] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 13 00:53:14 hermes kernel: [25235.150926] CR2: 0000000000000001 CR3: 00000000267b6000 CR4: 00000000000006f0
Mar 13 00:53:14 hermes kernel: [25235.150931] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 13 00:53:14 hermes kernel: [25235.150936] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 13 00:53:14 hermes kernel: [25235.150941] Process udisks-helper-a (pid: 31924, threadinfo ffff88007dbd0000, task ffff88007b295bc0)
Mar 13 00:53:14 hermes kernel: [25235.150945] Stack:
Mar 13 00:53:14 hermes kernel: [25235.150948]  ffff88007b295bc0 ffffffff811af82e ffff88007dbd1c98 0000000000000000
Mar 13 00:53:14 hermes kernel: [25235.150956] <0> ffff88005f4af660 ffff88005f4af6a8 0000000000000000 0000000000000060
Mar 13 00:53:14 hermes kernel: [25235.150964] <0> ffff88007dbd1cf8 ffffffff811af82e ffff88007b300a00 ffff88007b2afc30
Mar 13 00:53:14 hermes kernel: [25235.150973] Call Trace:
Mar 13 00:53:14 hermes kernel: [25235.150985]  [<ffffffff811af82e>] ? sysfs_get_open_dirent+0x9e/0x1a0
Mar 13 00:53:14 hermes kernel: [25235.150992]  [<ffffffff811af82e>] sysfs_get_open_dirent+0x9e/0x1a0
Mar 13 00:53:14 hermes kernel: [25235.151000]  [<ffffffff811afe5a>] sysfs_open_file+0x16a/0x200
Mar 13 00:53:14 hermes kernel: [25235.151008]  [<ffffffff811603c0>] ? mntput_no_expire+0x30/0x110
Mar 13 00:53:14 hermes kernel: [25235.151015]  [<ffffffff811afcf0>] ? sysfs_open_file+0x0/0x200
Mar 13 00:53:14 hermes kernel: [25235.151023]  [<ffffffff81142623>] __dentry_open+0x113/0x370
Mar 13 00:53:14 hermes kernel: [25235.151031]  [<ffffffff8125566f>] ? security_inode_permission+0x1f/0x30
Mar 13 00:53:14 hermes kernel: [25235.151039]  [<ffffffff8114eadf>] ? inode_permission+0xaf/0xd0
Mar 13 00:53:14 hermes kernel: [25235.151045]  [<ffffffff81142997>] nameidata_to_filp+0x57/0x70
Mar 13 00:53:14 hermes kernel: [25235.151053]  [<ffffffff81152d1a>] do_filp_open+0x2da/0xba0
Mar 13 00:53:14 hermes kernel: [25235.151060]  [<ffffffff8115e8ca>] ? alloc_fd+0x10a/0x150
Mar 13 00:53:14 hermes kernel: [25235.151067]  [<ffffffff81142399>] do_sys_open+0x69/0x170
Mar 13 00:53:14 hermes kernel: [25235.151073]  [<ffffffff811424e0>] sys_open+0x20/0x30
Mar 13 00:53:14 hermes kernel: [25235.151083]  [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
Mar 13 00:53:14 hermes kernel: [25235.151087] Code: 89 c7 fa 66 66 90 66 66 90 65 8b 04 25 18 e4 00 00 48 98 4c 8b 84 c3 f0 02 00 00 4d 8b 20 45 8b 48 18 4d 85 e4 74 64 41 8b 40 14 <49> 8b 04 c4 49 89 00 4c 89 ff 57 9d 66 66 90 66 90 4d 85 e4 0f 
Mar 13 00:53:14 hermes kernel: [25235.151142] RIP  [<ffffffff81137202>] kmem_cache_alloc+0x72/0x150
Mar 13 00:53:14 hermes kernel: [25235.151150]  RSP <ffff88007dbd1c68>
Mar 13 00:53:14 hermes kernel: [25235.151153] CR2: 0000000000000001
Mar 13 00:53:14 hermes kernel: [25235.151159] ---[ end trace 91526fd8d79f0252 ]---
comm commented 12 years ago

OK, so I've done some more testing for the past 6 hours and the rsync is continuing ok. Maybe a reboot was the answer? I'll report if it dies again, otherwise I'll close the issue after a few days.

behlendorf commented 12 years ago

Yes, a reboot would probably be required to pull in the updated kernel modules.

comm commented 12 years ago

OK, so its been running for 24 hours now without hanging (only writing at 5-10MBps, think its a slow NFS/read rate on the other machine, will have to check that later) but I am still occasionally getting the above in the kern.log, but no hanging/panic.

On a side note - may want to make the ubuntu package tell the system it needs a reboot when updated?

behlendorf commented 12 years ago

Your not seeing the following in your kernel log right?

BUG: unable to handle kernel NULL pointer dereference at 0000000000000001

Just perhaps a few warnings about certain threads being slow. As for the reboot it may or may not be required. The scripts do attempt to unload/load the new modules but they may be unsuccessful if those fs's are in use. In that case a reboot is required, otherwise it's not.

comm commented 12 years ago

Hi, No I'm not seeing that BUG since moving to daily. I'll keep an eye out for it.

behlendorf commented 12 years ago

I'm closing this issue since it appears to be resolved.

comm commented 12 years ago

Agreed - no errors anymore, I've rebooted a couple of times and still stable.

Thank you.