search

dtrace4linux / linux

dtrace for linux - kernel driver and userland tools
http://crtags.blogspot.com
1.17k stars 226 forks source link

Dtrace for system call entries on a process crashes and produces kernel dump #86

Open lbivens opened 10 years ago

lbivens commented 10 years ago

Running a trace for syscall entries on a command crashes dtrace.

As a side note, if I remove the flag -c and run the haproxy command on a separate window, it crashes the computer as soon as I type in the y in haproxy (not even running it)

Command

dtrace -n 'syscall:::entry /execname == "haproxy"/ { @[probefunc] = count() }' -c "haproxy -f haproxy_test.cfg"

Command Output

parent: waiting for child parent: after waitpid pid=4640 status=137f rd_loadobj_iter rd_loadobj_iter: /lib/x86_64-linux-gnu/libc-2.19.so 0x7fba39e41000 rd_loadobj_iter: /lib/x86_64-linux-gnu/libdl-2.19.so 0x7fba3a207000 rd_loadobj_iter: /usr/lib/x86_64-linux-gnu/libelf-0.158.so 0x7fba3a40b000 rd_loadobj_iter: /lib/x86_64-linux-gnu/libpthread-2.19.so 0x7fba3a621000 rd_loadobj_iter: /lib/x86_64-linux-gnu/librt-2.19.so 0x7fba3a83f000 rd_loadobj_iter: /lib/x86_64-linux-gnu/ld-2.19.so 0x7fba3ac60000 proc-stub:rd_event_enable proc-stub:rd_event_addr addr=(nil) proc-stub:rd_event_addr addr=(nil) proc-stub:rd_event_addr addr=(nil) dtrace: description 'syscall:::entry ' matched 661 probes PTRACE_TRACEME: Operation not permitted child 4640 about to exec haproxy dtrace: pid 4640 has exited

Kernel Output

[ 1116.800236] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1116.800240] IP: < (null)> [ 1116.800241] PGD 1ae988067 PUD 1ae98b067 PMD 0 [ 1116.800243] Oops: 0010 [#4] SMP [ 1116.800245] Modules linked in: cfg80211 dtracedrv(POF) ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) snd_hda_codec_hdmi saa7134_alsa tuner_simple tuner_types tea5767 tuner hid_generic snd_usb_audio usbhid snd_usbmidi_lib hid intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp snd_seq_midi snd_seq_midi_event kvm_intel kvm snd_hda_codec_realtek snd_rawmidi crct10dif_pclmul crc32_pclmul nvidia(POF) ghash_clmulni_intel snd_hda_intel bnep rfcomm aesni_intel rc_encore_enltv aes_x86_64 lrw gf128mul glue_helper ablk_helper bluetooth saa7134 snd_hda_codec snd_hwdep cryptd snd_pcm snd_seq tveeprom videobuf_dma_sg rc_core v4l2_common videobuf_core videodev serio_raw snd_seq_device drm snd_page_alloc lpc_ich snd_timer snd mei_me mei mac_hid soundcore video intel_smartconnect binfmt_misc parport_pc ppdev lp parport ahci libahci psmouse atl1c [ 1116.800283] CPU: 3 PID: 4640 Comm: haproxy Tainted: PF D O 3.13.0-30-generic #54-Ubuntu [ 1116.800284] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77-DS3H, BIOS F5 05/11/2012 [ 1116.800285] task: ffff8801e91a17f0 ti: ffff8800bde80000 task.ti: ffff8800bde80000 [ 1116.800286] RIP: 0010:[<0000000000000000>] < (null)> [ 1116.800288] RSP: 0018:ffff8800bde81f48 EFLAGS: 00010286 [ 1116.800289] RAX: 0000000000000038 RBX: 0000000000000000 RCX: 00007fb926391a10 [ 1116.800289] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011 [ 1116.800290] RBP: 00007fffd6076600 R08: 0000000000000000 R09: 0000000000001220 [ 1116.800291] R10: 00007fb926391a10 R11: 0000000000000246 R12: 00007fffd60765c0 [ 1116.800292] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1116.800293] FS: 00007fb926391740(0000) GS:ffff88021ed80000(0000) knlGS:0000000000000000 [ 1116.800294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1116.800295] CR2: 0000000000000000 CR3: 00000001ae97d000 CR4: 00000000001407e0 [ 1116.800295] Stack: [ 1116.800296] ffffffffa0fcd2c4 ffffffff8172ae9c 0000000000000000 0000000000000000 [ 1116.800298] 0000000000000000 00007fffd60765c0 00007fffd6076600 ffffffff8172aeff [ 1116.800300] 0000000000000246 00007fb926391a10 0000000000001220 0000000000000000 [ 1116.800302] Call Trace: [ 1116.800315] [] ? systrace_part1_sys_clone+0xb/0x1c [dtracedrv] [ 1116.800318] [] ? tracesys+0x7e/0xe6 [ 1116.800320] [] ? tracesys+0xe1/0xe6 [ 1116.800321] Code: Bad RIP value. [ 1116.800323] RIP < (null)> [ 1116.800324] RSP [ 1116.800325] CR2: 0000000000000000 [ 1116.800326] ---[ end trace f9799c9ceae785f9 ]---

Kernel

Linux trantor 3.13.0-30-generic #54-Ubuntu SMP Mon Jun 9 22:45:01 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

ryao commented 10 years ago

@lbivens It is non-obvious, but this is a duplicate of #75. The underlying cause is the same.

dtrace4linux commented 10 years ago

I have clone() working. Now I can quickly iterate over the other functions and push out a release.

I am having to specialise for >= 3.7 kernels.

On 30 July 2014 12:10, Richard Yao notifications@github.com wrote:

@lbivens https://github.com/lbivens It is non-obvious, but this is a duplicate of #75 https://github.com/dtrace4linux/linux/issues/75. The underlying cause is the same.

— Reply to this email directly or view it on GitHub https://github.com/dtrace4linux/linux/issues/86#issuecomment-50601436.