search

CESNET / ipfixcol

IPFIXcol is an implementation of an IPFIX (RFC 7011) collector
Other
64 stars 37 forks source link

tm_template_reference_dec: double free or corruption #13

Closed ghost closed 9 years ago

ghost commented 9 years ago

We have had a rare crash with IPFIXcol recently, which we are still trying to reproduce (so far, without any 'luck'). This is the stack trace:

*** glibc detected *** ./ipfixcol: double free or corruption (!prev): 0x00007fc728eae8c0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x76aa6)[0x7fc73dd4eaa6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fc73dd5384c]
./ipfixcol(tm_template_reference_dec+0x2d)[0x40a02d]
./ipfixcol[0x408f2e]
./ipfixcol[0x4058ae]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fc73e069b50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc73ddb3e6d]
======= Memory map: ========
00400000-0040f000 r-xp 00000000 08:11 284590582                          <path>/ipfixcol/base/src/ipfixcol
0060f000-00610000 rw-p 0000f000 08:11 284590582                          <path>/ipfixcol/base/src/ipfixcol
00b31000-11e96000 rw-p 00000000 00:00 0                                  [heap]
7fc6f8000000-7fc6f9354000 rw-p 00000000 00:00 0
7fc6f9354000-7fc6fc000000 ---p 00000000 00:00 0
7fc700000000-7fc704000000 rw-p 00000000 00:00 0
7fc704000000-7fc70bfff000 rw-p 00000000 00:00 0
7fc70bfff000-7fc70c000000 ---p 00000000 00:00 0
7fc70c000000-7fc70fffe000 rw-p 00000000 00:00 0
7fc70fffe000-7fc710000000 ---p 00000000 00:00 0
7fc710000000-7fc714000000 rw-p 00000000 00:00 0
7fc714000000-7fc717ffd000 rw-p 00000000 00:00 0
7fc717ffd000-7fc718000000 ---p 00000000 00:00 0
7fc718000000-7fc71c000000 rw-p 00000000 00:00 0
7fc720000000-7fc723ff1000 rw-p 00000000 00:00 0
7fc723ff1000-7fc724000000 ---p 00000000 00:00 0
7fc724000000-7fc727ffa000 rw-p 00000000 00:00 0
7fc727ffa000-7fc728000000 ---p 00000000 00:00 0
7fc728000000-7fc72c000000 rw-p 00000000 00:00 0
7fc72c000000-7fc72fffb000 rw-p 00000000 00:00 0
7fc72fffb000-7fc730000000 ---p 00000000 00:00 0
7fc730000000-7fc734000000 rw-p 00000000 00:00 0
7fc734000000-7fc737ffe000 rw-p 00000000 00:00 0
7fc737ffe000-7fc738000000 ---p 00000000 00:00 0
7fc73a5cd000-7fc73a5e2000 r-xp 00000000 08:02 42336260                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc73a5e2000-7fc73a7e2000 ---p 00015000 08:02 42336260                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc73a7e2000-7fc73a7e3000 rw-p 00015000 08:02 42336260                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7fc73a7e3000-7fc73a7e4000 ---p 00000000 00:00 0
7fc73a7e4000-7fc73afe4000 rw-p 00000000 00:00 0
7fc73afe4000-7fc73afe5000 ---p 00000000 00:00 0
7fc73afe5000-7fc73b7e5000 rw-p 00000000 00:00 0
7fc73b7e5000-7fc73b7e6000 ---p 00000000 00:00 0
7fc73b7e6000-7fc73bfe6000 rw-p 00000000 00:00 0
7fc73bfe6000-7fc73bfe7000 ---p 00000000 00:00 0
7fc73bfe7000-7fc73c7e7000 rw-p 00000000 00:00 0
7fc73c7e7000-7fc73c7f7000 r-xp 00000000 08:02 234895476                  /usr/lib/x86_64-linux-gnu/libcares.so.2.0.0
7fc73c7f7000-7fc73c9f6000 ---p 00010000 08:02 234895476                  /usr/lib/x86_64-linux-gnu/libcares.so.2.0.0
7fc73c9f6000-7fc73c9f7000 r--p 0000f000 08:02 234895476                  /usr/lib/x86_64-linux-gnu/libcares.so.2.0.0
7fc73c9f7000-7fc73c9f8000 rw-p 00010000 08:02 234895476                  /usr/lib/x86_64-linux-gnu/libcares.so.2.0.0
7fc73c9f8000-7fc73ca01000 r-xp 00000000 08:11 284623469                  <path>/ipfixcol/plugins/intermediate/proxy/.libs/ipfixcol-proxy-inter.so
7fc73ca01000-7fc73cc01000 ---p 00009000 08:11 284623469                  <path>/ipfixcol/plugins/intermediate/proxy/.libs/ipfixcol-proxy-inter.so
7fc73cc01000-7fc73cc02000 rw-p 00009000 08:11 284623469                  <path>/ipfixcol/plugins/intermediate/proxy/.libs/ipfixcol-proxy-inter.so
7fc73cc02000-7fc73cc07000 r-xp 00000000 08:11 284623430                  <path>/ipfixcol/plugins/intermediate/httpfieldmerge/.libs/ipfixcol-httpfieldmerge-inter.so
7fc73cc07000-7fc73ce06000 ---p 00005000 08:11 284623430                  <path>/ipfixcol/plugins/intermediate/httpfieldmerge/.libs/ipfixcol-httpfieldmerge-inter.so
7fc73ce06000-7fc73ce07000 rw-p 00004000 08:11 284623430                  <path>/ipfixcol/plugins/intermediate/httpfieldmerge/.libs/ipfixcol-httpfieldmerge-inter.so
7fc73ce07000-7fc73ce09000 r-xp 00000000 08:02 234881059                  /usr/lib/libsctp.so.1.0.11
7fc73ce09000-7fc73d008000 ---p 00002000 08:02 234881059                  /usr/lib/libsctp.so.1.0.11
7fc73d008000-7fc73d009000 rw-p 00001000 08:02 234881059                  /usr/lib/libsctp.so.1.0.11
7fc73d009000-7fc73d00d000 r-xp 00000000 08:11 284623394                  <path>/ipfixcol/base/src/storage/forwarding/.libs/ipfixcol-forwarding-output.so
7fc73d00d000-7fc73d20c000 ---p 00004000 08:11 284623394                  <path>/ipfixcol/base/src/storage/forwarding/.libs/ipfixcol-forwarding-output.so
7fc73d20c000-7fc73d20d000 rw-p 00003000 08:11 284623394                  <path>/ipfixcol/base/src/storage/forwarding/.libs/ipfixcol-forwarding-output.so
7fc73d20d000-7fc73d214000 r-xp 00000000 08:02 42336514                   /lib/x86_64-linux-gnu/librt-2.13.so
7fc73d214000-7fc73d413000 ---p 00007000 08:02 42336514                   /lib/x86_64-linux-gnu/librt-2.13.so
7fc73d413000-7fc73d414000 r--p 00006000 08:02 42336514                   /lib/x86_64-linux-gnu/librt-2.13.so
7fc73d414000-7fc73d415000 rw-p 00007000 08:02 42336514                   /lib/x86_64-linux-gnu/librt-2.13.so
7fc73d415000-7fc73d41b000 r-xp 00000000 08:11 284623280                  <path>/ipfixcol/base/src/input/udp/.libs/ipfixcol-udp-input.so
7fc73d41b000-7fc73d61b000 ---p 00006000 08:11 284623280                  <path>/ipfixcol/base/src/input/udp/.libs/ipfixcol-udp-input.so
7fc73d61b000-7fc73d61c000 rw-p 00006000 08:11 284623280                  <path>/ipfixcol/base/src/input/udp/.libs/ipfixcol-udp-input.so
7fc73d61c000-7fc73d69d000 r-xp 00000000 08:02 42336297                   /lib/x86_64-linux-gnu/libm-2.13.so
7fc73d69d000-7fc73d89c000 ---p 00081000 08:02 42336297                   /lib/x86_64-linux-gnu/libm-2.13.so
7fc73d89c000-7fc73d89d000 r--p 00080000 08:02 42336297                   /lib/x86_64-linux-gnu/libm-2.13.so
7fc73d89d000-7fc73d89e000 rw-p 00081000 08:02 42336297                   /lib/x86_64-linux-gnu/libm-2.13.so
7fc73d89e000-7fc73d8c0000 r-xp 00000000 08:02 42336288                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7fc73d8c0000-7fc73dabf000 ---p 00022000 08:02 42336288                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7fc73dabf000-7fc73dac0000 r--p 00021000 08:02 42336288                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7fc73dac0000-7fc73dac1000 rw-p 00022000 08:02 42336288                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7fc73dac1000-7fc73dad7000 r-xp 00000000 08:02 42336346                   /lib/x86_64-linux-gnu/libz.so.1.2.7
7fc73dad7000-7fc73dcd6000 ---p 00016000 08:02 42336346                   /lib/x86_64-linux-gnu/libz.so.1.2.7
7fc73dcd6000-7fc73dcd7000 r--p 00015000 08:02 42336346                   /lib/x86_64-linux-gnu/libz.so.1.2.7
7fc73dcd7000-7fc73dcd8000 rw-p 00016000 08:02 42336346                   /lib/x86_64-linux-gnu/libz.so.1.2.7
7fc73dcd8000-7fc73de59000 r-xp 00000000 08:02 42336289                   /lib/x86_64-linux-gnu/libc-2.13.so
7fc73de59000-7fc73e059000 ---p 00181000 08:02 42336289                   /lib/x86_64-linux-gnu/libc-2.13.so
7fc73e059000-7fc73e05d000 r--p 00181000 08:02 42336289                   /lib/x86_64-linux-gnu/libc-2.13.so
7fc73e05d000-7fc73e05e000 rw-p 00185000 08:02 42336289                   /lib/x86_64-linux-gnu/libc-2.13.so
7fc73e05e000-7fc73e063000 rw-p 00000000 00:00 0
7fc73e063000-7fc73e07a000 r-xp 00000000 08:02 42336506                   /lib/x86_64-linux-gnu/libpthread-2.13.so
7fc73e07a000-7fc73e279000 ---p 00017000 08:02 42336506                   /lib/x86_64-linux-gnu/libpthread-2.13.so
7fc73e279000-7fc73e27a000 r--p 00016000 08:02 42336506                   /lib/x86_64-linux-gnu/libpthread-2.13.so
7fc73e27a000-7fc73e27b000 rw-p 00017000 08:02 42336506                   /lib/x86_64-linux-gnu/libpthread-2.13.so
7fc73e27b000-7fc73e27f000 rw-p 00000000 00:00 0
7fc73e27f000-7fc73e3d4000 r-xp 00000000 08:02 234885316                  /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
7fc73e3d4000-7fc73e5d4000 ---p 00155000 08:02 234885316                  /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
7fc73e5d4000-7fc73e5dc000 r--p 00155000 08:02 234885316                  /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
7fc73e5dc000-7fc73e5de000 rw-p 0015d000 08:02 234885316                  /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
7fc73e5de000-7fc73e5df000 rw-p 00000000 00:00 0
7fc73e5df000-7fc73e5e1000 r-xp 00000000 08:02 42336295                   /lib/x86_64-linux-gnu/libdl-2.13.so
7fc73e5e1000-7fc73e7e1000 ---p 00002000 08:02 42336295                   /lib/x86_64-linux-gnu/libdl-2.13.so
7fc73e7e1000-7fc73e7e2000 r--p 00002000 08:02 42336295                   /lib/x86_64-linux-gnu/libdl-2.13.so
7fc73e7e2000-7fc73e7e3000 rw-p 00003000 08:02 42336295                   /lib/x86_64-linux-gnu/libdl-2.13.so
7fc73e7e3000-7fc73e803000 r-xp 00000000 08:02 42336283                   /lib/x86_64-linux-gnu/ld-2.13.so
7fc73e9f0000-7fc73e9f5000 rw-p 00000000 00:00 0
7fc73e9ff000-7fc73ea02000 rw-p 00000000 00:00 0
7fc73ea02000-7fc73ea03000 r--p 0001f000 08:02 42336283                   /lib/x86_64-linux-gnu/ld-2.13.so
7fc73ea03000-7fc73ea04000 rw-p 00020000 08:02 42336283                   /lib/x86_64-linux-gnu/ld-2.13.so
7fc73ea04000-7fc73ea05000 rw-p 00000000 00:00 0
7fffb022e000-7fffb024f000 rw-p 00000000 00:00 0                          [stack]
7fffb02cc000-7fffb02cd000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

Although we were running several plugins at the time of the crash, it really seems to be related to IPFIXcol's base code. tm_template_reference_dec is called only once in the codebase, namely from base/src/queues.c:253, and the double free or corruption appears to have occurred in base/src/template_manager.c:739. Can a double free perhaps be caused when the last templ->next in tm_template_reference_dec is freed both in tm_template_reference_dec:736 and tm_template_reference_dec:739? Or can you think of a scenario that could cause the double free?

mikeek commented 9 years ago

Hi, your scenario is right, that is the cause of the double free problem. I discovered and fixed this issue 2 weeks ago (58a47be6eea978871a5b17683006795ced3a7526) but only in our working repository (I forgot to merge changes with github). Everything should be OK now.

Thanks for the report