dtrace4linux
commented
9 years ago Yes - I found a number of issues.
- support for 32-bit syscalls on a 64-bit host wasnt quite complete
- some issues with clone/execve on 3.16
- issues around the xcall code
Having addressed the above, I have a random fatal-kill-kernel issue with some 32-bit syscalls; difficult to debug - since it hard kills the kernel, which is weird. Despite spending a lot of effort on it, I havent fully figured out what the issue is (and it could even be a compiler bug - its so erratic). I can "patch it up" to hide the issue. It ought to be better than what you are dealing with here.
When you get hard dies like above, check dmesg output and /proc/dtrace/trace. But its not supposed to panic, ever, and by virtue of it paniccing, its really annoying. (I am glad that people run this in VMs - would hate to think people run this on production systems).
On 9 January 2015 at 21:56, Michał Kullng notifications@github.com wrote:
dtrace -A
Killed uname -a
Linux kdev-buildsrv-kmod 3.16.7 #2 https://github.com/dtrace4linux/linux/issues/2 SMP Fri Jan 9 22:44:24 CET 2015 x86_64 GNU/Linux
It's KVM host, on phys machine it's the same.
Are you say something about 3.16 yesterday?
=============== KERNEL STACK ================ [ 17.815087] dtracedrv: module license 'CDDL' taints kernel. [ 17.816949] Disabling lock debugging due to kernel taint [ 26.417831] random: nonblocking pool is initialized [ 29.146782] BUG: unable to handle kernel NULL pointer dereference at (null) [ 29.148036] IP: < (null)> http://null [ 29.148775] PGD d7f95067 PUD 30cb9067 PMD 0 [ 29.149655] Oops: 0010 [#1 https://github.com/dtrace4linux/linux/pull/1] SMP [ 29.150114] Modules linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop ghash_clmulni_intel aesni_intel aes_x86_64 psmouse hid_generic ablk_helper processor cryptd usbhid lrw parport_pc parport hid gf128mul glue_helper thermal_sys serio_raw virtio_balloon i2c_piix4 microcode evdev i2c_core button ext4 crc16 jbd2 mbcache ata_generic virtio_blk uhci_hcd virtio_net ehci_hcd ata_piix libata scsi_mod crc32c_intel usbcore usb_common virtio_pci virtio_ring virtio [ 29.150114] CPU: 0 PID: 2531 Comm: dtrace Tainted: P O 3.16.7 #2 https://github.com/dtrace4linux/linux/issues/2 [ 29.150114] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 [ 29.150114] task: ffff88011aaee8d0 ti: ffff880030bf0000 task.ti: ffff880030bf0000 [ 29.150114] RIP: 0010:[] < (null)> http://null [ 29.150114] RSP: 0018:ffff880030bf3b50 EFLAGS: 00010246 [ 29.150114] RAX: 0000000000000000 RBX: ffff8800c67e8700 RCX: ffffffffa0326dc0 [ 29.150114] RDX: ffffffffa0326dd8 RSI: 0000000000010000 RDI: 0000000000007431 [ 29.150114] RBP: 0000000000000000 R08: 000000000000000a R09: 0000000000000030 [ 29.150114] R10: 0000000000000020 R11: ffffffffa033e9d2 R12: 0000000000000018 [ 29.150114] R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000018 [ 29.150114] FS: 00007f71707ac700(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000 [ 29.150114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 29.150114] CR2: 0000000000000000 CR3: 00000000c66cf000 CR4: 00000000000407f0 [ 29.150114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 29.150114] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 29.150114] Stack: [ 29.150114] ffffffffa0326fe2 0000000000000000 00000001000280da ffff88011fffac18 [ 29.150114] ffff8800c67e8700 0000000000000000 0000000000000001 0000000000000000 [ 29.150114] ffff8800308d3060 ffff8800db5ff010 ffffffffa03270ed ffff8800c67e8700 [ 29.150114] Call Trace: [ 29.150114] [] ? init_syscalls+0x205/0x2fe [dtracedrv] [ 29.150114] [] ? systrace_provide+0x12/0x2ff [dtracedrv] [ 29.150114] [] ? dtrace_probe_provide+0x83/0xf1 [dtracedrv] [ 29.150114] [] ? dtrace_open+0x55/0x10f [dtracedrv] [ 29.150114] [] ? dtracedrv_open+0x18/0x1c [dtracedrv] [ 29.150114] [] ? misc_open+0x117/0x145 [ 29.150114] [] ? chrdev_open+0x12d/0x156 [ 29.150114] [] ? cdev_put+0x19/0x19 [ 29.150114] [] ? do_dentry_open+0x178/0x27d [ 29.150114] [] ? finish_open+0x2a/0x33 [ 29.150114] [] ? do_last+0x85b/0xa0f [ 29.150114] [] ? inode_permission+0x57/0x95 [ 29.150114] [] ? link_path_walk+0x112/0x46f [ 29.150114] [] ? path_openat+0x233/0x4c5 [ 29.150114] [] ? do_filp_open+0x2d/0x75 [ 29.150114] [] ? alloc_fd+0xd9/0xea [ 29.150114] [] ? do_sys_open+0x65/0xe9 [ 29.150114] [] ? system_call_fastpath+0x16/0x1b [ 29.150114] Code: Bad RIP value. [ 29.150114] RIP < (null)> http://null [ 29.150114] RSP [ 29.150114] CR2: 0000000000000000 [ 29.187448] ---[ end trace 6099a1b8fcd07389 ]--- =============== / KERNEL STACK ======================
If you try to run again, dtrace freezing (strace from second run):
================ STRACE =============== (cut) access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\0\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0644, st_size=14768, ...}) = 0 mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f2040ca8000 mprotect(0x7f2040caa000, 2097152, PROT_NONE) = 0 mmap(0x7f2040eaa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f2040eaa000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\357\1\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1603600, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2041912000 mmap(NULL, 3717176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f204091c000 mprotect(0x7f2040a9e000, 2097152, PROT_NONE) = 0 mmap(0x7f2040c9e000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x182000) = 0x7f2040c9e000 mmap(0x7f2040ca3000, 18488, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2040ca3000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2041911000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2041910000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f204190f000 arch_prctl(ARCH_SET_FS, 0x7f2041910700) = 0 mprotect(0x7f2040c9e000, 16384, PROT_READ) = 0 mprotect(0x7f2040eaa000, 4096, PROT_READ) = 0 mprotect(0x7f20410bf000, 4096, PROT_READ) = 0 mprotect(0x7f20412d7000, 4096, PROT_READ) = 0 mprotect(0x7f20414e3000, 4096, PROT_READ) = 0 mprotect(0x7f20416fa000, 4096, PROT_READ) = 0 mprotect(0x7f204191b000, 4096, PROT_READ) = 0 munmap(0x7f2041914000, 19228) = 0 set_tid_address(0x7f20419109d0) = 2816 set_robust_list(0x7f20419109e0, 0x18) = 0 futex(0x7fffc46a15ac, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7f2041910700) = -1 EAGAIN (Resource temporarily unavailable) rt_sigaction(SIGRTMIN, {0x7f20410c6ad0, [], SA_RESTORER|SA_SIGINFO, 0x7f20410d00a0}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x7f20410c6b60, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f20410d00a0}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192
_1024, rlim_max=RLIM_INFINITY}) = 0 brk(0) = 0x1b2c000 brk(0x1b4d000) = 0x1b4d000 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4_1024}) = 0 setrlimit(RLIMIT_NOFILE, {rlim_cur=4_1024, rlim_max=4_1024}) = 0 open("/dev/dtrace/provider", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOTDIR (Not a directory) open("/dev/dtrace", O_RDWR ^C ==================== / STRACE ====================
Ping me if you need more information.
— Reply to this email directly or view it on GitHub https://github.com/dtrace4linux/linux/issues/95.
mikess
dtrace -A
Killed
uname -a
Linux kdev-buildsrv-kmod 3.16.7 #2 SMP Fri Jan 9 22:44:24 CET 2015 x86_64 GNU/Linux
It's KVM host, on phys machine it's the same.
Are you say something about 3.16 yesterday?
=============== KERNEL STACK ================ [ 17.815087] dtracedrv: module license 'CDDL' taints kernel. [ 17.816949] Disabling lock debugging due to kernel taint [ 26.417831] random: nonblocking pool is initialized [ 29.146782] BUG: unable to handle kernel NULL pointer dereference at (null) [ 29.148036] IP: < (null)> [ 29.148775] PGD d7f95067 PUD 30cb9067 PMD 0 [ 29.149655] Oops: 0010 [#1] SMP [ 29.150114] Modules linked in: dtracedrv(PO) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop ghash_clmulni_intel aesni_intel aes_x86_64 psmouse hid_generic ablk_helper processor cryptd usbhid lrw parport_pc parport hid gf128mul glue_helper thermal_sys serio_raw virtio_balloon i2c_piix4 microcode evdev i2c_core button ext4 crc16 jbd2 mbcache ata_generic virtio_blk uhci_hcd virtio_net ehci_hcd ata_piix libata scsi_mod crc32c_intel usbcore usb_common virtio_pci virtio_ring virtio [ 29.150114] CPU: 0 PID: 2531 Comm: dtrace Tainted: P O 3.16.7 #2 [ 29.150114] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 [ 29.150114] task: ffff88011aaee8d0 ti: ffff880030bf0000 task.ti: ffff880030bf0000 [ 29.150114] RIP: 0010:[<0000000000000000>] < (null)> [ 29.150114] RSP: 0018:ffff880030bf3b50 EFLAGS: 00010246 [ 29.150114] RAX: 0000000000000000 RBX: ffff8800c67e8700 RCX: ffffffffa0326dc0 [ 29.150114] RDX: ffffffffa0326dd8 RSI: 0000000000010000 RDI: 0000000000007431 [ 29.150114] RBP: 0000000000000000 R08: 000000000000000a R09: 0000000000000030 [ 29.150114] R10: 0000000000000020 R11: ffffffffa033e9d2 R12: 0000000000000018 [ 29.150114] R13: 0000000000000002 R14: 0000000000000002 R15: 0000000000000018 [ 29.150114] FS: 00007f71707ac700(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000 [ 29.150114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 29.150114] CR2: 0000000000000000 CR3: 00000000c66cf000 CR4: 00000000000407f0 [ 29.150114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 29.150114] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 29.150114] Stack: [ 29.150114] ffffffffa0326fe2 0000000000000000 00000001000280da ffff88011fffac18 [ 29.150114] ffff8800c67e8700 0000000000000000 0000000000000001 0000000000000000 [ 29.150114] ffff8800308d3060 ffff8800db5ff010 ffffffffa03270ed ffff8800c67e8700 [ 29.150114] Call Trace: [ 29.150114] [] ? init_syscalls+0x205/0x2fe [dtracedrv]
[ 29.150114] [] ? systrace_provide+0x12/0x2ff [dtracedrv]
[ 29.150114] [] ? dtrace_probe_provide+0x83/0xf1 [dtracedrv]
[ 29.150114] [] ? dtrace_open+0x55/0x10f [dtracedrv]
[ 29.150114] [] ? dtracedrv_open+0x18/0x1c [dtracedrv]
[ 29.150114] [] ? misc_open+0x117/0x145
[ 29.150114] [] ? chrdev_open+0x12d/0x156
[ 29.150114] [] ? cdev_put+0x19/0x19
[ 29.150114] [] ? do_dentry_open+0x178/0x27d
[ 29.150114] [] ? finish_open+0x2a/0x33
[ 29.150114] [] ? do_last+0x85b/0xa0f
[ 29.150114] [] ? inode_permission+0x57/0x95
[ 29.150114] [] ? link_path_walk+0x112/0x46f
[ 29.150114] [] ? path_openat+0x233/0x4c5
[ 29.150114] [] ? do_filp_open+0x2d/0x75
[ 29.150114] [] ? alloc_fd+0xd9/0xea
[ 29.150114] [] ? do_sys_open+0x65/0xe9
[ 29.150114] [] ? system_call_fastpath+0x16/0x1b
[ 29.150114] Code: Bad RIP value.
[ 29.150114] RIP < (null)>
[ 29.150114] RSP
[ 29.150114] CR2: 0000000000000000
[ 29.187448] ---[ end trace 6099a1b8fcd07389 ]---
=============== / KERNEL STACK ======================
If you try to run again, dtrace freezing (strace from second run):
================ STRACE =============== (cut) access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\0\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0644, st_size=14768, ...}) = 0 mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f2040ca8000 mprotect(0x7f2040caa000, 2097152, PROT_NONE) = 0 mmap(0x7f2040eaa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f2040eaa000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\357\1\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1603600, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2041912000 mmap(NULL, 3717176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f204091c000 mprotect(0x7f2040a9e000, 2097152, PROT_NONE) = 0 mmap(0x7f2040c9e000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x182000) = 0x7f2040c9e000 mmap(0x7f2040ca3000, 18488, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2040ca3000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2041911000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2041910000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f204190f000 arch_prctl(ARCH_SET_FS, 0x7f2041910700) = 0 mprotect(0x7f2040c9e000, 16384, PROT_READ) = 0 mprotect(0x7f2040eaa000, 4096, PROT_READ) = 0 mprotect(0x7f20410bf000, 4096, PROT_READ) = 0 mprotect(0x7f20412d7000, 4096, PROT_READ) = 0 mprotect(0x7f20414e3000, 4096, PROT_READ) = 0 mprotect(0x7f20416fa000, 4096, PROT_READ) = 0 mprotect(0x7f204191b000, 4096, PROT_READ) = 0 munmap(0x7f2041914000, 19228) = 0 set_tid_address(0x7f20419109d0) = 2816 set_robust_list(0x7f20419109e0, 0x18) = 0 futex(0x7fffc46a15ac, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7f2041910700) = -1 EAGAIN (Resource temporarily unavailable) rt_sigaction(SIGRTMIN, {0x7f20410c6ad0, [], SA_RESTORER|SA_SIGINFO, 0x7f20410d00a0}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x7f20410c6b60, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f20410d00a0}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192_1024, rlim_max=RLIM_INFINITY}) = 0 brk(0) = 0x1b2c000 brk(0x1b4d000) = 0x1b4d000 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4_1024}) = 0 setrlimit(RLIMIT_NOFILE, {rlim_cur=4_1024, rlim_max=4_1024}) = 0 open("/dev/dtrace/provider", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOTDIR (Not a directory) open("/dev/dtrace", O_RDWR ^C <unfinished ...> ==================== / STRACE ====================
Ping me if you need more information.