WARN ( default_mysql/mysql ): Abnormal exit status detected for child PID

[DocMAX]

Description I compiled a newer version on Debian 10 (old was 1.7.2). But i get this error. (No issue with Debian verison) How can i debug this? With debug enabled there is not more info.

Feb 27 04:40:01 server nfacctd[9850]: WARN ( default_mysql/mysql ): Abnormal exit status detected for child PID 10242
Feb 27 04:43:01 server nfacctd[9850]: WARN ( default_mysql/mysql ): Abnormal exit status detected for child PID 10875
Feb 27 04:44:01 server nfacctd[9850]: WARN ( default_mysql/mysql ): Abnormal exit status detected for child PID 11197
Feb 27 04:45:01 server nfacctd[9850]: WARN ( default_mysql/mysql ): Abnormal exit status detected for child PID 11456

Version

NetFlow Accounting Daemon, nfacctd 1.7.7-git [20210226-2 (8fb94f03)]

Arguments:
 '--enable-mysql' '--enable-geoipv2' '--enable-debug' '--prefix=/usr/local/' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'                                                                  

Libs:
cdada 0.3.4
libpcap version 1.9.1 (with TPACKET_V3)
MariaDB 10.3.27
MaxmindDB 1.3.2

Plugins:
memory
print
nfprobe
sfprobe
tee
mysql

System:
Linux 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64

Compiler:
gcc 8.3.0

[DocMAX]

Just compiled a few versions. The last working version is 1.7.5! Here is the gdb bt of 1.7.7:

(gdb) bt
#0  __strstr_sse2_unaligned () at ../sysdeps/x86_64/multiarch/strstr-sse2-unaligned.S:40
#1  0x000056114c841503 in MY_DB_Connect (db=0x56114ca8e080 <p>, host=0x0) at mysql_plugin.c:698
#2  0x000056114c845c03 in sql_exit_gracefully (signum=2) at sql_common.c:1113
#3  <signal handler called>
#4  0x00007f07cd05d7e4 in __GI___poll (fds=0x7ffff5ce83b8, nfds=1, timeout=52000) at ../sysdeps/unix/sysv/linux/poll.c:29
#5  0x000056114c83e785 in mysql_plugin (pipe_fd=4, cfgptr=0x561155b2fd18, ptr=0x56114c9f1100 <channels_list>) at mysql_plugin.c:144
#6  0x000056114c7d851b in load_plugins (req=0x7ffff5d1e240) at plugin_hooks.c:287
#7  0x000056114c7c244d in main (argc=6, argv=0x7ffff5d1e408, envp=0x7ffff5d1e440) at nfacctd.c:1113
(gdb) info line
Line 40 of "../sysdeps/x86_64/multiarch/strstr-sse2-unaligned.S" starts at address 0x7f07cd0109bd <__strstr_sse2_unaligned+77> and ends at 0x7f07cd0109c1 <__strstr_sse2_unaligned+81>.
(gdb) disassemble
Dump of assembler code for function __strstr_sse2_unaligned:
   0x00007f07cd010970 <+0>:     movzbl (%rsi),%eax
   0x00007f07cd010973 <+3>:     test   %al,%al
   0x00007f07cd010975 <+5>:     je     0x7f07cd010b17 <__strstr_sse2_unaligned+423>
   0x00007f07cd01097b <+11>:    movzbl 0x1(%rsi),%edx
   0x00007f07cd01097f <+15>:    test   %dl,%dl
   0x00007f07cd010981 <+17>:    je     0x7f07cd010a40 <__strstr_sse2_unaligned+208>
   0x00007f07cd010987 <+23>:    movd   %eax,%xmm1
   0x00007f07cd01098b <+27>:    movd   %edx,%xmm2
   0x00007f07cd01098f <+31>:    mov    %rdi,%rax
   0x00007f07cd010992 <+34>:    and    $0xfff,%eax
   0x00007f07cd010997 <+39>:    punpcklbw %xmm1,%xmm1
   0x00007f07cd01099b <+43>:    cmp    $0xfbf,%rax
   0x00007f07cd0109a1 <+49>:    punpcklbw %xmm2,%xmm2
   0x00007f07cd0109a5 <+53>:    punpcklwd %xmm1,%xmm1
   0x00007f07cd0109a9 <+57>:    punpcklwd %xmm2,%xmm2
   0x00007f07cd0109ad <+61>:    pshufd $0x0,%xmm1,%xmm1
   0x00007f07cd0109b2 <+66>:    pshufd $0x0,%xmm2,%xmm2
   0x00007f07cd0109b7 <+71>:    ja     0x7f07cd010cc0 <__strstr_sse2_unaligned+848>
=> 0x00007f07cd0109bd <+77>:    movdqu (%rdi),%xmm3
   0x00007f07cd0109c1 <+81>:    pxor   %xmm5,%xmm5
   0x00007f07cd0109c5 <+85>:    movdqu 0x1(%rdi),%xmm4
   0x00007f07cd0109ca <+90>:    movdqa %xmm3,%xmm6
   0x00007f07cd0109ce <+94>:    pcmpeqb %xmm1,%xmm3
   0x00007f07cd0109d2 <+98>:    pcmpeqb %xmm2,%xmm4
   0x00007f07cd0109d6 <+102>:   movdqu 0x10(%rdi),%xmm0
   0x00007f07cd0109db <+107>:   pcmpeqb %xmm5,%xmm6
   0x00007f07cd0109df <+111>:   pminub %xmm4,%xmm3
   0x00007f07cd0109e3 <+115>:   movdqa %xmm3,%xmm4
   0x00007f07cd0109e7 <+119>:   movdqu 0x11(%rdi),%xmm3

[DocMAX]

This piece of software is full of bugs! With 1.7.5:

Feb 27 14:45:01 server nfacctd[17768]: DEBUG ( default_mysql/mysql ): FAILED query follows:
                                       UPDATE `acct` SET packets=packets+1, bytes=bytes+54, stamp_updated=NOW() WHERE FROM_UNIXTIME(1614433440) = stamp_inserted AND ip_src='192.168.1.2' AND ip_dst='192.168.1.1' AND src_port=48193 AND dst_port=53 AND ip_proto='udp' AND country_ip_src='O1' AND lat_ip_src='0.000000' AND  AND lon_ip_src='0.000000'
Feb 27 14:45:01 server nfacctd[17768]: ERROR ( default_mysql/mysql ): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'AND lon_ip_src='0.000000'' at line 1

please fix https://github.com/pmacct/pmacct/blob/eb6039ba73a2ede734b6ef266a7230705b7cd1e5/src/sql_common.c#L2276

[paololucente]

Dear @DocMAX ,

I am sorry to find you in such a bad mood today and if This piece of software is full of bugs! was true then, given the extended software user base, yours would not be the only single one bug issue open at this one point in time. This said, let's get to business:

1) With regards to the Abnormal exit status issue: that message indicates a crash in the child (the database writer) so it is normal to not find more info in the logs - even at debug level. You could provide me useful information could you please provide me the output of gdb (preferrably) or valgrind? https://github.com/pmacct/pmacct/blob/1.7.6/QUICKSTART#L2864-#L2884 . If this does not apply to more recent versions of code, we can forget about it and leave history behind us.

2) I see that 'AND AND' issue. The latest commit 2ec83acd338b61c8726c5f155295fcf71ae7faba should address it. Please let me know if you get a chance to give it a try.

Paolo

[DocMAX]

i allready gave you the gdb output of latest master... what do you mean?

[paololucente]

Cool, can you confirm you do not include a sql_host directive in the config? If so, situation would be pretty clear and i'd have a fix for this right away.

[DocMAX]

My current config working with 1.7.5 is this:

daemonize: false
#debug_internal_msg: true
#debug: true
pidfile: /var/run/nfacctd.pid
nfacctd_ip: 0.0.0.0
syslog: daemon
#geoip_ipv4_file: /usr/share/GeoIP/GeoIP.dat
#geoipv2_file: /var/lib/GeoIP/GeoLite2-ASN.mmdb
geoipv2_file: /var/lib/GeoIP/GeoLite2-City.mmdb
#geoipv2_file: /var/lib/GeoIP/GeoLite2-Country.mmdb
aggregate: src_mac, dst_mac, src_host, dst_host, proto, src_port, dst_port, src_host_country, src_host_pocode, src_host_coords, flows
plugins: mysql
sql_table: acct
sql_refresh_time: 60
sql_optimize_clauses: true
sql_history: 1m
sql_history_roundoff: m

[paololucente]

Cool, so most probably 1922062277d87da024938cc3057df9a4c3725e76 would solve the Abnormal exit status issue you were experiencing. So, in short, please give a try to master code and let me know what works and what does not work still.

[DocMAX]

everything is working now. thank you!