search

ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.65k stars 44 forks source link

nprobe 9 crashes with SIP Plugin #412

Closed jgallartm closed 4 years ago

jgallartm commented 4 years ago

Hello

we are trying nprobe 9 in a test machine:

nprobe --version

Welcome to nProbe v.9.0.200316 (r6800) for x86_64-unknown-linux-gnu with native PF_RING acceleration. Copyright 2002-20 ntop.org

Build OS: CentOS Linux release 7.7.1908 (Core) SystemID: A15A40C09208AA8C GIT rev: 9.0-stable:23602d5490e338bf21106069f2e5bebdf510e929:20200316 License: Invalid license (/etc/nprobe.license) [Missing license file]

We are tryng the SIP Plugin, and nprobe stops after a few seconds, leaving this backtrace:

Error in `./nprobe': corrupted size vs. prev_size: 0x00007f140efb00f0
======= Backtrace: =========
/lib64/libc.so.6(+0x7f7c4)[0x7f14285d97c4]
/lib64/libc.so.6(+0x818bb)[0x7f14285db8bb]
/usr/local/lib/nprobe/plugins/libsipPlugin-9.0.200316.so(+0x57e37)[0x7f1420a90e37]
/usr/local/lib/libnprobe-9.0.200316.so(pluginCallback+0x2c1)[0x7f142b700a1a]
/usr/local/lib/libnprobe-9.0.200316.so(processFlowPacket+0x4944)[0x7f142b6f30dc]
./nprobe[0x419804]
./nprobe(decodePacket+0x550)[0x41a132]
./nprobe[0x41366e]
./nprobe[0x41402b]
/lib64/libpthread.so.0(+0x7e65)[0x7f142a231e65]
/lib64/libc.so.6(clone+0x6d)[0x7f142865888d]
======= Memory map: ========
00400000-00442000 r-xp 00000000 08:02 421777 /usr/local/bin/nprobe
00641000-00642000 r--p 00041000 08:02 421777 /usr/local/bin/nprobe
00642000-00643000 rw-p 00042000 08:02 421777 /usr/local/bin/nprobe
00643000-0068d000 rw-p 00000000 00:00 0
012b4000-0133b000 rw-p 00000000 00:00 0 [heap]

Can you take a look?

Thanks

Javi

cardigliano commented 4 years ago

@jgallartm is this still an open issue? Can you reproduce this with latest nprobe?

jgallartm commented 4 years ago

Hi Alfredo

the latest version looks stable.

Regards

cardigliano commented 4 years ago

Ok let's close this thank you.

jgallartm commented 4 years ago

Hello

this is happening again. I'm using version v.9.0.200716. dmesg shows "[6619620.741046] nprobe[223963]: segfault at 7ff1222820d0 ip 00007ff4988cb3ab sp 00007ff4820d5b30 error 4 in libsipPlugin-9.0.200716.so[7ff498875000+e1000"

gdb output: [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/bin/nprobe /etc/nprobe/nprobe-zc1-00.conf'. Program terminated with signal 11, Segmentation fault.

0 0x00007ff4988cb3ab in parse_rtp_codecs () from /usr/local/lib/nprobe/plugins/libsipPlugin-9.0.200716.so

Missing separate debuginfos, use: debuginfo-install nprobe-9.0.200716-6823.x86_64 (gdb) bt

0 0x00007ff4988cb3ab in parse_rtp_codecs () from /usr/local/lib/nprobe/plugins/libsipPlugin-9.0.200716.so

1 0x00007ff4988cc7ba in sipPlugin_packet () from /usr/local/lib/nprobe/plugins/libsipPlugin-9.0.200716.so

2 0x00007ff4a3961d72 in pluginCallback () from /usr/local/lib/libnprobe-9.0.200716.so

3 0x00007ff4a3954428 in processFlowPacket () from /usr/local/lib/libnprobe-9.0.200716.so

4 0x00000000004198a7 in ?? ()

5 0x000000000041a1d5 in decodePacket ()

6 0x00000000004136ce in ?? ()

7 0x00000000004140d8 in ?? ()

8 0x00007ff4a2492e65 in start_thread () from /lib64/libpthread.so.0

9 0x00007ff4a06ae88d in clone () from /lib64/libc.so.6

You can download the core file from here:

https://www.dropbox.com/s/gsovo7gushcug6d/core.223893?dl=0

Best regards

Javi

cardigliano commented 4 years ago

@jgallartm I pushed a safety check for malformed packets, a new build will be available later today

jgallartm commented 4 years ago

Hi Alfredo,

I just installed the latest version, it has not crashed so far.

Javi

jgallartm commented 4 years ago

Hi Alfredo

I see a different type of crash now: Jul 23 10:22:51 pre-mars.sonoc.io nprobe[161441]: 23/Jul/2020 10:22:51 [sipPlugin.c:689] ERROR: Wrong SIP message. No p
Jul 23 10:23:02 pre-mars.sonoc.io nprobe[161441]: 23/Jul/2020 10:23:02 [sipPlugin.c:689] ERROR: Wrong SIP message. No p
Jul 23 10:23:05 pre-mars.sonoc.io nprobe[161441]: 23/Jul/2020 10:23:05 [sipPlugin.c:689] ERROR: Wrong SIP message. No p
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: *** Error in `/usr/local/bin/nprobe': corrupted size vs. prev_size: 0
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: ======= Backtrace: =========
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /lib64/libc.so.6(+0x7f7c4)[0x7f28bfc3a7c4]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /lib64/libc.so.6(+0x818bb)[0x7f28bfc3c8bb]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /usr/local/lib/nprobe/plugins/libsipPlugin-9.0.200721.so(+0x57fe8)[0x
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /usr/local/lib/libnprobe-9.0.200721.so(pluginCallback+0x2c1)[0x7f28c2
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /usr/local/lib/libnprobe-9.0.200721.so(processFlowPacket+0x4957)[0x7f
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /usr/local/bin/nprobe[0x4198a7]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /usr/local/bin/nprobe(decodePacket+0x550)[0x41a1d5]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /usr/local/bin/nprobe[0x4136ce]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /usr/local/bin/nprobe[0x4140d8]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /lib64/libpthread.so.0(+0x7e65)[0x7f28c1a9de65]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: /lib64/libc.so.6(clone+0x6d)[0x7f28bfcb988d]
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: ======= Memory map: ========
Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: 00400000-00442000 r-xp 00000000 08:02 421790 Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: 00641000-00642000 r--p 00041000 08:02 421790 Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: 00642000-00643000 rw-p 00042000 08:02 421790 Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: 00643000-0068d000 rw-p 00000000 00:00 0 Jul 23 10:23:07 pre-mars.sonoc.io nprobe[161441]: 01d46000-01def000 rw-p 00000000 00:00 0

Regards

Javi

jgallartm commented 4 years ago

I tried the nprobes static binary provided by email. Attached is the core file. This is the backtrace: (gdb) bt

0 0x00007fe1cf539337 in raise () from /lib64/libc.so.6

1 0x00007fe1cf53aa28 in abort () from /lib64/libc.so.6

2 0x00007fe1cf57be87 in __libc_message () from /lib64/libc.so.6

3 0x00007fe1cf5827c4 in malloc_printerr () from /lib64/libc.so.6

4 0x00007fe1cf5848bb in _int_free () from /lib64/libc.so.6

5 0x00000000004b5c2e in sipPlugin_packet (callback=packet_callback, packet_if_idx=0, pluginInfo=0x7fe1baffbc88, bkt=0x7fe1a80afe20, flow_direction=dst2src_direction, ip_offset=14, proto=17, isFragment=1 '\001', numPkts=2,

tos=136 '\210', retransmitted_pkt=0 '\000', vlanId=0, ehdr=0x7fe1b40008e0, src=0x7fe1baffc710, sport=5060, dst=0x7fe1baffc6f0, dport=5060, len=1732, flags=0 '\000', tcpSeqNum=0, icmpType=0 '\000', numMplsLabels=0, 
mplsLabels=0x7fe1baffc520 "", h=0x7fe1baffcad0, p=0x7fe1b40008e0 "\f\304z\036wѰ\250n\364\225\002\b", 
payload=0x7fe1b400090a "SIP/2.0 183 Session Progress\r\nCall-ID: c13d096a-4ac9-1239-6498-021923691058\r\nCSeq: 23380014 INVITE\r\nFrom: \"+3908119735722\" <sip:+3908119735722@79.170.64.165;user=phone;noa=international>;tag=60cr8tN18"..., payloadLen=1464, pfinfo=0x7fe1baffc540) at sipPlugin.c:878

6 0x00000000004564fc in pluginCallback (callbackType=packet_callback, packet_if_idx=0, bkt=0x7fe1a80afe20, direction=dst2src_direction, ip_offset=14, proto=17, isFragment=1 '\001', numPkts=2, tos=136 '\210',

retransmitted_pkt=0 '\000', vlanId=0, ehdr=0x7fe1b40008e0, src=0x7fe1baffc710, sport=5060, dst=0x7fe1baffc6f0, dport=5060, len=1732, flags=0 '\000', tcpSeqNum=0, icmpType=0 '\000', numMplsLabels=0, mplsLabels=0x7fe1baffc520 "", 
h=0x7fe1baffcad0, p=0x7fe1b40008e0 "\f\304z\036wѰ\250n\364\225\002\b", 
payload=0x7fe1b400090a "SIP/2.0 183 Session Progress\r\nCall-ID: c13d096a-4ac9-1239-6498-021923691058\r\nCSeq: 23380014 INVITE\r\nFrom: \"+3908119735722\" <sip:+3908119735722@79.170.64.165;user=phone;noa=international>;tag=60cr8tN18"..., payloadLen=1464, pfinfo=0x7fe1baffc540) at plugin.c:507

7 0x000000000041e7e7 in processFlowPacket (pinfo=0x7fe1baffc540) at engine.c:2768

8 0x0000000000440a60 in deepPacketDecode (thread_id=0, packet_if_idx=0, h=0x7fe1baffcad0, p=0x7fe1b40008e0 "\f\304z\036wѰ\250n\364\225\002\b", sampledPacket=0 '\000', rx_direction=1 '\001', numPkts=2, input_index=65535,

output_index=65535, flow_sender_ip=0, packet_hash=0, extraPayloadOffset=0) at nprobe.c:2137

9 0x000000000044138c in decodePacket (thread_id=0, packet_if_idx=0, h=0x7fe1baffcad0, p=0x7fe1b40008e0 "\f\304z\036wѰ\250n\364\225\002\b", sampledPacket=0 '\000', rx_direction=1 '\001', numPkts=1, input_index=65535,

output_index=65535, flow_sender_ip=0, packet_hash=0) at nprobe.c:2353

10 0x000000000043a883 in processPfringPktHdr (hdr=0x7fe1baffcad0, packet=0x7fe1b40008e0 "\f\304z\036wѰ\250n\364\225\002\b", thread_id=0, packet_hash=0, direction=1 '\001') at pro/pf_ring.c:55

11 0x000000000043b28d in fetchPfRingPackets (notUsed=0x0) at pro/pf_ring.c:322

12 0x00007fe1cfddfe65 in start_thread () from /lib64/libpthread.so.0

13 0x00007fe1cf60188d in clone () from /lib64/libc.so.6

Regards core.230619.gz

jgallartm commented 4 years ago

Hello

did you have a chance to take a look at this?

cardigliano commented 4 years ago

@jgallartm it seems I missed this sorry, I will check ti asap