search

Cacti / spine

Spine C Based Poller for Cacti
GNU Lesser General Public License v2.1
81 stars 45 forks source link

Spine 1.2.28 encountered segfault #356

Open kiwildante opened 1 month ago

kiwildante commented 1 month ago

Describe the bug i have a few cacti server that collect data for my device, i have server that only have about a hundred devices, and i have server that have a thousand devices, both produce segfault, but the hundred one only produce a few, but in the thousand device produce a lot with no backtrace, a few have a backtrace, i already set nofiles to a larger one, and have mysql max connection larger, but it still produce the same result, i run with 20/10 p/t for the large one, and 3/3 for the hundred. it affect the polling time to be maximum, cause maybe the poller is waiting the spine to complete but the spine produce a segfault

To Reproduce Steps to reproduce the behavior:

  1. Go to 'logs'
  2. Change to cacti_stderr.log
  3. Some Spine Error segfault
  4. See error

Expected behavior poller runs normally without segfault

Screenshots here for the hundred device log image here for the thousand device image here's with backtrace image here's poller information for the thousand one image

Server (please complete the following information):

Compiling (please complete the following information):

Note: you can find glibc version by running ldd --version

Note: if source is github, please include last commit reference

Additional context

full backtrace log

2024-10-16 12:23:44 - FATAL: Spine Encountered a Segmentation Fault

Generating backtrace...0 line(s)... Error in `/usr/local/spine/bin/spine': double free or corruption (!prev): 0x000000000255aa40 ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f041833d7f5] /lib/x86_64-linux-gnu/libc.so.6(+0x8038a)[0x7f041834638a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f041834a58c] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x56846)[0x7f0418c0c846] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x281ba)[0x7f0418bde1ba] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x2ec11)[0x7f0418be4c11] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_send_query+0x57)[0x7f0418be1187] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_real_query+0x11)[0x7f0418be1201] /usr/local/spine/bin/spine[0x4057ab] /usr/local/spine/bin/spine[0x40e479] /usr/local/spine/bin/spine[0x412ab2] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f04186976ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f04183cd51d] ======= Memory map: ======== 00400000-00424000 r-xp 00000000 08:01 922807 /usr/local/spine/bin/spine 00623000-00624000 r--p 00023000 08:01 922807 /usr/local/spine/bin/spine 00624000-00625000 rw-p 00024000 08:01 922807 /usr/local/spine/bin/spine 00625000-0069a000 rw-p 00000000 00:00 0 024d4000-0258a000 rw-p 00000000 00:00 0 [heap] 7f03e4000000-7f03e4039000 rw-p 00000000 00:00 0 7f03e4039000-7f03e8000000 ---p 00000000 00:00 0 7f03e8000000-7f03e8023000 rw-p 00000000 00:00 0 7f03e8023000-7f03ec000000 ---p 00000000 00:00 0 7f03ec000000-7f03ec022000 rw-p 00000000 00:00 0 7f03ec022000-7f03f0000000 ---p 00000000 00:00 0 7f03f0000000-7f03f003f000 rw-p 00000000 00:00 0 7f03f003f000-7f03f4000000 ---p 00000000 00:00 0 7f03f4000000-7f03f4022000 rw-p 00000000 00:00 0 7f03f4022000-7f03f8000000 ---p 00000000 00:00 0 7f03f8000000-7f03f8022000 rw-p 00000000 00:00 0 7f03f8022000-7f03fc000000 ---p 00000000 00:00 0 7f03fdffc000-7f03fdffd000 ---p 00000000 00:00 0 7f03fdffd000-7f03fe7fd000 rw-p 00000000 00:00 0 7f03fe7fd000-7f03fe7fe000 ---p 00000000 00:00 0 7f03fe7fe000-7f03feffe000 rw-p 00000000 00:00 0 7f03feffe000-7f03fefff000 ---p 00000000 00:00 0 7f03fefff000-7f03ff7ff000 rw-p 00000000 00:00 0 7f03ff7ff000-7f03ff800000 ---p 00000000 00:00 0 7f03ff800000-7f0400000000 rw-p 00000000 00:00 0 7f0400000000-7f0400022000 rw-p 00000000 00:00 0 7f0400022000-7f0404000000 ---p 00000000 00:00 0 7f0404000000-7f0404039000 rw-p 00000000 00:00 0 7f0404039000-7f0408000000 ---p 00000000 00:00 0 7f0408000000-7f0408022000 rw-p 00000000 00:00 0 7f0408022000-7f040c000000 ---p 00000000 00:00 0 7f040c000000-7f040c022000 rw-p 00000000 00:00 0 7f040c022000-7f0410000000 ---p 00000000 00:00 0 7f0410000000-7f0410022000 rw-p 00000000 00:00 0 7f0410022000-7f0414000000 ---p 00000000 00:00 0 7f041404b000-7f041404c000 ---p 00000000 00:00 0 7f041404c000-7f041484c000 rw-p 00000000 00:00 0 7f041484c000-7f041484d000 ---p 00000000 00:00 0 7f041484d000-7f041504d000 rw-p 00000000 00:00 0 7f041504d000-7f041504e000 ---p 00000000 00:00 0 7f041504e000-7f041584e000 rw-p 00000000 00:00 0 7f041584e000-7f041584f000 ---p 00000000 00:00 0 7f041584f000-7f041604f000 rw-p 00000000 00:00 0 7f041604f000-7f0416050000 ---p 00000000 00:00 0 7f0416050000-7f0416850000 rw-p 00000000 00:00 0 7f0416850000-7f0416851000 ---p 00000000 00:00 0 7f0416851000-7f0417051000 rw-p 00000000 00:00 0 7f0417051000-7f041705c000 r-xp 00000000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f041705c000-7f041725b000 ---p 0000b000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f041725b000-7f041725c000 r--p 0000a000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f041725c000-7f041725d000 rw-p 0000b000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f041725d000-7f0417263000 rw-p 00000000 00:00 0 7f0417263000-7f0417279000 r-xp 00000000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f0417279000-7f0417478000 ---p 00016000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f0417478000-7f0417479000 rw-p 00015000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f0417479000-7f04175eb000 r-xp 00000000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f04175eb000-7f04177eb000 ---p 00172000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f04177eb000-7f04177f5000 r--p 00172000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f04177f5000-7f04177f7000 rw-p 0017c000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f04177f7000-7f04177fb000 rw-p 00000000 00:00 0 7f04177fb000-7f0417859000 r-xp 00000000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f0417859000-7f0417a59000 ---p 0005e000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f0417a59000-7f0417a5d000 r--p 0005e000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f0417a5d000-7f0417a63000 rw-p 00062000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f0417a63000-7f0417a7c000 r-xp 00000000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f0417a7c000-7f0417c7b000 ---p 00019000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f0417c7b000-7f0417c7c000 r--p 00018000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f0417c7c000-7f0417c7d000 rw-p 00019000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f0417c7d000-7f0417c80000 r-xp 00000000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so 7f0417c80000-7f0417e7f000 ---p 00003000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so 7f0417e7f000-7f0417e80000 r--p 00002000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so 7f0417e80000-7f0417e81000 rw-p 00003000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so 7f0417e81000-7f041809c000 r-xp 00000000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f041809c000-7f041829b000 ---p 0021b000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f041829b000-7f04182b7000 r--p 0021a000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f04182b7000-7f04182c3000 rw-p 00236000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f04182c3000-7f04182c6000 rw-p 00000000 00:00 0 7f04182c6000-7f0418486000 r-xp 00000000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so 7f0418486000-7f0418686000 ---p 001c0000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so 7f0418686000-7f041868a000 r--p 001c0000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so 7f041868a000-7f041868c000 rw-p 001c4000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so 7f041868c000-7f0418690000 rw-p 00000000 00:00 0 7f0418690000-7f04186a8000 r-xp 00000000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f04186a8000-7f04188a7000 ---p 00018000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f04188a7000-7f04188a8000 r--p 00017000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f04188a8000-7f04188a9000 rw-p 00018000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f04188a9000-7f04188ad000 rw-p 00000000 00:00 0 7f04188ad000-7f04189b5000 r-xp 00000000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so 7f04189b5000-7f0418bb4000 ---p 00108000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so 7f0418bb4000-7f0418bb5000 r--p 00107000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so 7f0418bb5000-7f0418bb6000 rw-p 00108000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so 7f0418bb6000-7f0418ef2000 r-xp 00000000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f0418ef2000-7f04190f1000 ---p 0033c000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f04190f1000-7f04190f5000 r--p 0033b000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f04190f5000-7f0419167000 rw-p 0033f000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f0419167000-7f041916d000 rw-p 00000000 00:00 0 7f041916d000-7f0419210000 r-xp 00000000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f0419210000-7f0419410000 ---p 000a3000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f0419410000-7f0419411000 r--p 000a3000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f0419411000-7f0419413000 rw-p 000a4000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f0419413000-7f0419449000 rw-p 00000000 00:00 0 7f0419449000-7f041946f000 r-xp 00000000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so 7f04194c2000-7f041965a000 r--p 00000000 08:01 788398 /usr/lib/locale/locale-archive 7f041965a000-7f0419663000 rw-p 00000000 00:00 0 7f041966d000-7f041966e000 rw-p 00000000 00:00 0 7f041966e000-7f041966f000 r--p 00025000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so 7f041966f000-7f0419670000 rw-p 00026000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so 7f0419670000-7f0419671000 rw-p 00000000 00:00 0 7ffc62213000-7ffc62314000 rw-p 00000000 00:00 0 [stack] 7ffc623cd000-7ffc623cf000 r--p 00000000 00:00 0 [vvar] 7ffc623cf000-7ffc623d1000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] 2024-10-16 12:30:27 - FATAL: Spine Interrupted by Abort Signal

TheWitness commented 4 weeks ago

You likely ran out of connections. Look at the connections logic on the technical support page.

TheWitness commented 4 weeks ago

Also, for 1000 devices, I would normally use 2 processes and maybe 20 threads.

kiwildante commented 4 weeks ago

i already sent the screenshot for the connection logic, i set the max connection to 2000 and the min req is 400, so i guess it's not the connection problem, allright i'll try to change it to 2 process and 20 threads and see if it still produce the segfault

kiwildante commented 4 weeks ago

i already set the process to 2 and threads to 20, but it makes the segfault occur more often from before image image

TheWitness commented 3 weeks ago

What's you log level?

TheWitness commented 3 weeks ago

What operating system and version?

kiwildante commented 3 weeks ago

i'm setting it to low right now, i'm using Ubuntu 16.04.7 LTS

kiwildante commented 3 weeks ago

in the poller setting i set the invalid data logging to detailed

TheWitness commented 3 weeks ago

Did you get a core file? Can you do a simple backtrace of it one that provides line numbers and files?

kiwildante commented 2 weeks ago

i'm sorry, i already tried to get the core files, but it didn't record anything, i'll try it again later

kiwildante commented 2 weeks ago

i can only provide you with this cacti log backtrace that happens in one of my 100 devices server, it uses 3 process and 3 thread

2024-10-28 11:40:20 - FATAL: Spine Interrupted by Abort Signal -- *** Error in `/usr/local/spine/bin/spine': double free or corruption (!prev): 0x0000000000917260 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f66c85007f5] /lib/x86_64-linux-gnu/libc.so.6(+0x8038a)[0x7f66c850938a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f66c850d58c] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x56846)[0x7f66c8dcf846] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x281ba)[0x7f66c8da11ba] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x2ec11)[0x7f66c8da7c11] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_send_query+0x57)[0x7f66c8da4187] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_real_query+0x11)[0x7f66c8da4201] /usr/local/spine/bin/spine[0x4055fb] /usr/local/spine/bin/spine[0x40e565] /usr/local/spine/bin/spine[0x412422] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f66c885a6ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f66c85904dd] ======= Memory map: ======== 00400000-00422000 r-xp 00000000 09:00 25692760 /usr/local/spine/bin/spine 00622000-00623000 r--p 00022000 09:00 25692760 /usr/local/spine/bin/spine 00623000-00624000 rw-p 00023000 09:00 25692760 /usr/local/spine/bin/spine 00624000-00699000 rw-p 00000000 00:00 0 008c5000-0093c000 rw-p 00000000 00:00 0 [heap] 7f66b0000000-7f66b0023000 rw-p 00000000 00:00 0 7f66b0023000-7f66b4000000 ---p 00000000 00:00 0 7f66b8000000-7f66b803a000 rw-p 00000000 00:00 0 7f66b803a000-7f66bc000000 ---p 00000000 00:00 0 7f66bc000000-7f66bc040000 rw-p 00000000 00:00 0 7f66bc040000-7f66c0000000 ---p 00000000 00:00 0 7f66c0000000-7f66c003a000 rw-p 00000000 00:00 0 7f66c003a000-7f66c4000000 ---p 00000000 00:00 0 7f66c5210000-7f66c5211000 ---p 00000000 00:00 0 7f66c5211000-7f66c5a11000 rw-p 00000000 00:00 0 7f66c5a11000-7f66c5a12000 ---p 00000000 00:00 0 7f66c5a12000-7f66c6212000 rw-p 00000000 00:00 0 7f66c6212000-7f66c6213000 ---p 00000000 00:00 0 7f66c6213000-7f66c6a13000 rw-p 00000000 00:00 0 7f66c6a13000-7f66c6a14000 ---p 00000000 00:00 0 7f66c6a14000-7f66c7214000 rw-p 00000000 00:00 0 7f66c7214000-7f66c721f000 r-xp 00000000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c721f000-7f66c741e000 ---p 0000b000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c741e000-7f66c741f000 r--p 0000a000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c741f000-7f66c7420000 rw-p 0000b000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c7420000-7f66c7426000 rw-p 00000000 00:00 0 7f66c7426000-7f66c743c000 r-xp 00000000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c743c000-7f66c763b000 ---p 00016000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c763b000-7f66c763c000 rw-p 00015000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c763c000-7f66c77ae000 r-xp 00000000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c77ae000-7f66c79ae000 ---p 00172000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79ae000-7f66c79b8000 r--p 00172000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79b8000-7f66c79ba000 rw-p 0017c000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79ba000-7f66c79be000 rw-p 00000000 00:00 0 7f66c79be000-7f66c7a1c000 r-xp 00000000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7a1c000-7f66c7c1c000 ---p 0005e000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c1c000-7f66c7c20000 r--p 0005e000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c20000-7f66c7c27000 rw-p 00062000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c27000-7f66c7c40000 r-xp 00000000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7c40000-7f66c7e3f000 ---p 00019000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e3f000-7f66c7e40000 r--p 00018000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e40000-7f66c7e41000 rw-p 00019000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e41000-7f66c7e44000 r-xp 00000000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c7e44000-7f66c8043000 ---p 00003000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8043000-7f66c8044000 r--p 00002000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8044000-7f66c8045000 rw-p 00003000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8045000-7f66c825f000 r-xp 00000000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c825f000-7f66c845e000 ---p 0021a000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c845e000-7f66c847a000 r--p 00219000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c847a000-7f66c8486000 rw-p 00235000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c8486000-7f66c8489000 rw-p 00000000 00:00 0 7f66c8489000-7f66c8649000 r-xp 00000000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c8649000-7f66c8849000 ---p 001c0000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c8849000-7f66c884d000 r--p 001c0000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c884d000-7f66c884f000 rw-p 001c4000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c884f000-7f66c8853000 rw-p 00000000 00:00 0 7f66c8853000-7f66c886b000 r-xp 00000000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c886b000-7f66c8a6a000 ---p 00018000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6a000-7f66c8a6b000 r--p 00017000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6b000-7f66c8a6c000 rw-p 00018000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6c000-7f66c8a70000 rw-p 00000000 00:00 0 7f66c8a70000-7f66c8b78000 r-xp 00000000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8b78000-7f66c8d77000 ---p 00108000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d77000-7f66c8d78000 r--p 00107000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d78000-7f66c8d79000 rw-p 00108000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d79000-7f66c90b5000 r-xp 00000000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c90b5000-7f66c92b4000 ---p 0033c000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c92b4000-7f66c92b8000 r--p 0033b000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c92b8000-7f66c932a000 rw-p 0033f000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c932a000-7f66c9330000 rw-p 00000000 00:00 0 7f66c9330000-7f66c93d3000 r-xp 00000000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c93d3000-7f66c95d3000 ---p 000a3000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d3000-7f66c95d4000 r--p 000a3000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d4000-7f66c95d6000 rw-p 000a4000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d6000-7f66c960c000 rw-p 00000000 00:00 0 7f66c960c000-7f66c9632000 r-xp 00000000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9686000-7f66c981e000 r--p 00000000 09:00 24908536 /usr/lib/locale/locale-archive 7f66c981e000-7f66c9827000 rw-p 00000000 00:00 0 7f66c9830000-7f66c9831000 rw-p 00000000 00:00 0 7f66c9831000-7f66c9832000 r--p 00025000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9832000-7f66c9833000 rw-p 00026000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9833000-7f66c9834000 rw-p 00000000 00:00 0 7ffe15453000-7ffe15552000 rw-p 00000000 00:00 0 [stack] 7ffe155f7000-7ffe155fa000 r--p 00000000 00:00 0 [vvar] 7ffe155fa000-7ffe155fc000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
TheWitness commented 2 weeks ago

Where are you saving core files? Can you enable the capture of the core file and then backtrace the core file? If now. Go to your build directory, copy the spine.conf file into that build location. And then run spine this way:

cd /spine/build/directory
gdb ./spine
run -V 3 -S
# wait for crash
bt
# capture output
quit
TheWitness commented 2 weeks ago

@kiwildante, any update?

kiwildante commented 1 week ago

i'm sorry for the late update, i'm still not get the core files, i set it in the tmp folder, but it still not showing i'll try with the gdb

kiwildante commented 1 week ago

here's the bt (gdb) bt

0 0x00007fac1dbc13b0 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

1 0x00007fac1db89e31 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

2 0x00007fac1db8f162 in mysql_send_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

3 0x00007fac1db8f201 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

4 0x00000000004057ab in db_query (mysql=mysql@entry=0x7fabf8fd3910, type=type@entry=0,

query=query@entry=0x7fabf8fd8c90 "SELECT SQL_NO_CACHE id, hostname, snmp_community, snmp_version, snmp_username, snmp_password, snmp_auth_protocol, snmp_priv_passphrase, snmp_priv_protocol, snmp_context, snmp_engine_id, snmp_port, snm"...) at sql.c:159

5 0x000000000040e479 in poll_host (device_counter=device_counter@entry=72, host_id=host_id@entry=1495, host_thread=host_thread@entry=1,

host_threads=host_threads@entry=1, host_data_ids=host_data_ids@entry=8, host_time=host_time@entry=0x7fabf8ff8de0 "1730715374",
host_errors=0x7fabf8ff8d0c, host_time_double=host_time_double@entry=1730715374.817122) at poller.c:565

6 0x0000000000412ab2 in child (arg=0x17a3f50) at poller.c:109

7 0x00007fac1d6456ba in start_thread (arg=0x7fabf8ff9700) at pthread_create.c:333

8 0x00007fac1d37b51d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

TheWitness commented 1 week ago

Yikes, that looks like a mysqlclient issue. Are you using MariaDB or MySQL? What version? You might want to see if there is a client update somewhere. It's an old server os though right Ubuntu 16.04?

TheWitness commented 1 week ago

Also, if you were using a previous spine version, which one was it?

kiwildante commented 1 week ago

i'm using mysql Ver 14.14 Distrib 5.7.33, for Linux (x86_64) using EditLine wrapper, it already the latest update, yes i'm using Distributor ID: Ubuntu Description: Ubuntu 16.04.7 LTS Release: 16.04 Codename: xenial

kiwildante commented 1 week ago

i don't remember exactly, cause i have a few server, and each one i installed it with the new version, but lately i've been upgrading all of them to the latest, i think it's around 1.2.3, 1.2.14. and 1.2.18 i guess

TheWitness commented 1 week ago

I might recommend you to stay on the older spine until you get your OS updated to something more recent. I don't think there's any harm to be honest.

kiwildante commented 1 week ago

for now i set it to 50/20, and total connection is max 2048 min req 1250

kiwildante commented 1 week ago

what version u can recommend using for ubuntu 16.04 and 18.04, and if the version of the spine and cacti are different is there any issue ? or do i have to downgrade the cacti version too ?

TheWitness commented 1 week ago

First, make sure you compile the spine on the Linux you are running on, or at least on an older release. Second, as long as you are at Cacti 1.2.x or above, whatever version you are using will even work with 1.3.

I don't see spine architecture changing drastically in my lifetime any way. It's design is set in stone. There are a few enhancements around SNMP that will happen over time, other than that and maybe some optimization on startup, and ipv6 ping, but not much more going on. Regardless of those changes, it will remain compatible with future releases.

TheWitness commented 1 week ago

In more recent versions of spine, I went to connection pooling that creates all the MySQL/MariaDB connections up front. My guess is that is causing issues with your mysqlclient. Likely a memory leak of some kind in the client API with so many connections sharing the same memory space.

I debated giving people the ability to switch from pooling to on demand connections again, but thus far, too many other things going on in my life.

kiwildante commented 1 week ago

can u tell me which version of spine that more stable for ubuntu 16.04 or 18.04 or u can tell me which version of the spine dan use on demand connection

TheWitness commented 6 days ago

Review the change log for the term 'connection pool'. Use a version before that. Make sure you compile it using the machine it runs on.

kiwildante commented 4 days ago

allright, thanks for the help