netdata / netdata

Architected for speed. Automated for easy. Monitoring and troubleshooting, transformed!
https://www.netdata.cloud
GNU General Public License v3.0
72k stars 5.93k forks source link

[Bug]: ebpf.plugin core dumped #13390

Closed tickez closed 2 years ago

tickez commented 2 years ago

Bug description

After starting the netdata agent, systemd reports a core dump of the ebpf.plugin.

Starting the plugin directly from the CLI results in the same behavior. It fails with both, the static build and the compiled version.

Jul 16 20:05:15 www systemd-coredump[4020]: [🡕] Process 3772 (ebpf.plugin) of user 491 dumped core.

                                            Found module /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin without build-id.
                                            Found module linux-vdso.so.1 with build-id: 1d8950f257479087bcdda44a121002411cba55af
                                            Stack trace of thread 3846:
                                            #0  0x0000000000730a29 n/a (/opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin + 0x330a29)
❯ ./plugins.d/ebpf.plugin
2022-07-16 20:12:39:  INFO  : MAIN : Does not have a configuration file inside `/opt/netdata/etc/netdata/ebpf.d.conf. It will try to load stock file.
2022-07-16 20:12:39:  INFO  : MAIN : Name resolution is disabled, collector will not parser "hostnames" list.
2022-07-16 20:12:39:  INFO  : MAIN : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
2022-07-16 20:12:39:  INFO  : MAIN : Cannot read process groups configuration file '/opt/netdata/etc/netdata/apps_groups.conf'. Will try '/opt/netdata/usr/lib/netdata/conf.d/apps_groups.conf'
2022-07-16 20:12:39:  ERROR : MAIN : PROCFILE: Cannot open file '/proc/3772/status' (errno 2, No such file or directory)
2022-07-16 20:12:39:  ERROR : MAIN : Cannot open /proc/3772/status
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF PROCESS : thread created with task id 4869
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF PROCESS : set name of thread 4869 to EBPF PROCESS
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF VFS : thread created with task id 4875
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF VFS : set name of thread 4875 to EBPF VFS
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF FILESYSTEM : thread created with task id 4876
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF FILESYSTEM : set name of thread 4876 to EBPF FILESYSTEM
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF MOUNT : thread created with task id 4878
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF MOUNT : set name of thread 4878 to EBPF MOUNT
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF FD : thread created with task id 4879
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF FD : set name of thread 4879 to EBPF FD
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOFTIRQ : thread created with task id 4881
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOFTIRQ : set name of thread 4881 to EBPF SOFTIRQ
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SHM : thread created with task id 4883
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SHM : set name of thread 4883 to EBPF SHM
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF HARDIRQ : thread created with task id 4880
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF HARDIRQ : set name of thread 4880 to EBPF HARDIRQ
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF MDFLUSH : thread created with task id 4884
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF MDFLUSH : set name of thread 4884 to EBPF MDFLUSH
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF OOMKILL : thread created with task id 4882
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF OOMKILL : set name of thread 4882 to EBPF OOMKILL
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF DISK : thread created with task id 4877
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF DISK : set name of thread 4877 to EBPF DISK
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SWAP : thread created with task id 4874
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SWAP : set name of thread 4874 to EBPF SWAP
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF DCSTAT : thread created with task id 4873
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF DCSTAT : set name of thread 4873 to EBPF DCSTAT
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SYNC : thread created with task id 4872
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SYNC : set name of thread 4872 to EBPF SYNC
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF CACHESTAT : thread created with task id 4871
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF CACHESTAT : set name of thread 4871 to EBPF CACHESTAT
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : thread created with task id 4870
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : set name of thread 4870 to EBPF SOCKET
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : Name resolution is disabled, collector will not parser "hostnames" list.
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : The network value of CIDR 127.0.0.1/8 was updated for 127.0.0.0 .
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : The range/value (127.0.0.1/8) is inside the range/value (127.0.0.1/8) already inserted, it will be ignored.
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : The range/value (10.0.0.0/8) is inside the range/value (10.0.0.0/8) already inserted, it will be ignored.
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : The range/value (172.16.0.0/12) is inside the range/value (172.16.0.0/12) already inserted, it will be ignored.
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : The range/value (192.168.0.0/16) is inside the range/value (192.168.0.0/16) already inserted, it will be ignored.
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : The range/value (fc00::/7) is inside the range/value (fc00::/7) already inserted, it will be ignored.
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SOCKET : The range/value (::1/128) is inside the range/value (::1/128) already inserted, it will be ignored.
2022-07-16 20:12:39: ebpf.plugin INFO  : SOFTIRQ KERNEL : thread created with task id 4886
2022-07-16 20:12:39: ebpf.plugin INFO  : SOFTIRQ KERNEL : set name of thread 4886 to SOFTIRQ KERNEL
CHART system.process_thread '' 'Start process' 'calls/s' 'processes' '' 'line' 21002 5 '' 'ebpf.plugin' 'process'
DIMENSION _do_fork process absolute 1 1
DIMENSION sys_clone thread absolute 1 1
CHART system.exit '' 'Exit process' 'calls/s' 'processes' '' 'line' 21003 5 '' 'ebpf.plugin' 'process'
DIMENSION do_exit process absolute 1 1
DIMENSION release_task task absolute 1 1
CHART system.process_status '' 'Process not closed' 'difference' 'processes' '' line 21004 5 '' 'ebpf.plugin' 'process'
DIMENSION process '' absolute 1 1
DIMENSION zombie '' absolute 1 1
CHART netdata.ebpf_threads '' 'Threads info.' 'threads' 'ebpf' '' 'line' 140000 5 '' 'ebpf.plugin' 'process'
DIMENSION total total absolute 1 1
DIMENSION running running absolute 1 1
CHART netdata.ebpf_load_methods '' 'Load info.' 'methods' 'ebpf' '' 'line' 140001 5 '' 'ebpf.plugin' 'process'
DIMENSION legacy legacy absolute 1 1
DIMENSION co-re co-re absolute 1 1
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF FILESYSTEM : thread with task id 4876 finished
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF DISK : thread with task id 4877 finished
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF SWAP : thread with task id 4874 finished
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF CACHESTAT : thread with task id 4871 finished
2022-07-16 20:12:39: ebpf.plugin INFO  : HARDIRQ KERNEL : thread created with task id 4887
2022-07-16 20:12:39: ebpf.plugin INFO  : HARDIRQ KERNEL : set name of thread 4887 to HARDIRQ KERNEL
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF CGROUP : thread created with task id 4888
2022-07-16 20:12:39: ebpf.plugin INFO  : EBPF CGROUP : set name of thread 4888 to EBPF CGROUP
CHART apps.process_create '' 'Process started' 'calls/s' 'processes' '' 'stacked' 20065 5 '' 'ebpf.plugin' 'process'
DIMENSION netdata '' absolute 1 1
DIMENSION apps.plugin '' absolute 1 1
DIMENSION python.d.plugin '' absolute 1 1
DIMENSION tc-qos-helper '' absolute 1 1
DIMENSION go.d.plugin '' absolute 1 1
DIMENSION ebpf.plugin '' absolute 1 1
DIMENSION fail2ban '' absolute 1 1
DIMENSION sql '' absolute 1 1
DIMENSION email '' absolute 1 1
DIMENSION logs '' absolute 1 1
DIMENSION nms '' absolute 1 1
DIMENSION ssh '' absolute 1 1
DIMENSION time '' absolute 1 1
DIMENSION dhcp '' absolute 1 1
DIMENSION dns '' absolute 1 1
DIMENSION antivirus '' absolute 1 1
DIMENSION cron '' absolute 1 1
DIMENSION X '' absolute 1 1
DIMENSION ksmd '' absolute 1 1
DIMENSION system '' absolute 1 1
DIMENSION kernel '' absolute 1 1
DIMENSION node '' absolute 1 1
DIMENSION other '' absolute 1 1
CHART apps.thread_create '' 'Threads started' 'calls/s' 'processes' '' 'stacked' 20066 5 '' 'ebpf.plugin' 'process'
DIMENSION netdata '' absolute 1 1
DIMENSION apps.plugin '' absolute 1 1
DIMENSION python.d.plugin '' absolute 1 1
DIMENSION tc-qos-helper '' absolute 1 1
DIMENSION go.d.plugin '' absolute 1 1
DIMENSION ebpf.plugin '' absolute 1 1
DIMENSION fail2ban '' absolute 1 1
DIMENSION sql '' absolute 1 1
DIMENSION email '' absolute 1 1
DIMENSION logs '' absolute 1 1
DIMENSION nms '' absolute 1 1
DIMENSION ssh '' absolute 1 1
DIMENSION time '' absolute 1 1
DIMENSION dhcp '' absolute 1 1
DIMENSION dns '' absolute 1 1
DIMENSION antivirus '' absolute 1 1
DIMENSION cron '' absolute 1 1
DIMENSION X '' absolute 1 1
DIMENSION ksmd '' absolute 1 1
DIMENSION system '' absolute 1 1
DIMENSION kernel '' absolute 1 1
DIMENSION node '' absolute 1 1
DIMENSION other '' absolute 1 1
CHART apps.task_exit '' 'Tasks starts exit process.' 'calls/s' 'processes' '' 'stacked' 20067 5 '' 'ebpf.plugin' 'process'
DIMENSION netdata '' absolute 1 1
DIMENSION apps.plugin '' absolute 1 1
DIMENSION python.d.plugin '' absolute 1 1
DIMENSION tc-qos-helper '' absolute 1 1
DIMENSION go.d.plugin '' absolute 1 1
DIMENSION ebpf.plugin '' absolute 1 1
DIMENSION fail2ban '' absolute 1 1
DIMENSION sql '' absolute 1 1
DIMENSION email '' absolute 1 1
DIMENSION logs '' absolute 1 1
DIMENSION nms '' absolute 1 1
DIMENSION ssh '' absolute 1 1
DIMENSION time '' absolute 1 1
DIMENSION dhcp '' absolute 1 1
DIMENSION dns '' absolute 1 1
DIMENSION antivirus '' absolute 1 1
DIMENSION cron '' absolute 1 1
DIMENSION X '' absolute 1 1
DIMENSION ksmd '' absolute 1 1
DIMENSION system '' absolute 1 1
DIMENSION kernel '' absolute 1 1
DIMENSION node '' absolute 1 1
DIMENSION other '' absolute 1 1
CHART apps.task_close '' 'Tasks closed' 'calls/s' 'processes' '' 'stacked' 20068 5 '' 'ebpf.plugin' 'process'
DIMENSION netdata '' absolute 1 1
DIMENSION apps.plugin '' absolute 1 1
DIMENSION python.d.plugin '' absolute 1 1
DIMENSION tc-qos-helper '' absolute 1 1
DIMENSION go.d.plugin '' absolute 1 1
DIMENSION ebpf.plugin '' absolute 1 1
DIMENSION fail2ban '' absolute 1 1
DIMENSION sql '' absolute 1 1
DIMENSION email '' absolute 1 1
DIMENSION logs '' absolute 1 1
DIMENSION nms '' absolute 1 1
DIMENSION ssh '' absolute 1 1
DIMENSION time '' absolute 1 1
DIMENSION dhcp '' absolute 1 1
DIMENSION dns '' absolute 1 1
DIMENSION antivirus '' absolute 1 1
DIMENSION cron '' absolute 1 1
DIMENSION X '' absolute 1 1
DIMENSION ksmd '' absolute 1 1
DIMENSION system '' absolute 1 1
DIMENSION kernel '' absolute 1 1
DIMENSION node '' absolute 1 1
DIMENSION other '' absolute 1 1
CHART apps.outbound_conn_v4 '' 'Calls to tcp_v4_connection' 'connections/s' 'net' '' 'stacked' 20080 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.outbound_conn_v6 '' 'Calls to tcp_v6_connection' 'connections/s' 'net' '' 'stacked' 20081 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.total_bandwidth_sent '' 'Bytes sent' 'kilobits/s' 'net' '' 'stacked' 20082 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.total_bandwidth_recv '' 'bytes received' 'kilobits/s' 'net' '' 'stacked' 20083 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.bandwidth_tcp_send '' 'Calls for tcp_sendmsg' 'calls/s' 'net' '' 'stacked' 20084 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.bandwidth_tcp_recv '' 'Calls for tcp_cleanup_rbuf' 'calls/s' 'net' '' 'stacked' 20085 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.bandwidth_tcp_retransmit '' 'Calls for tcp_retransmit' 'calls/s' 'net' '' 'stacked' 20086 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.bandwidth_udp_send '' 'Calls for udp_sendmsg' 'calls/s' 'net' '' 'stacked' 20087 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.bandwidth_udp_recv '' 'Calls for udp_recvmsg' 'calls/s' 'net' '' 'stacked' 20088 5 '' 'ebpf.plugin' 'socket'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.file_deleted '' 'Files deleted' 'calls/s' 'vfs' '' 'stacked' 20065 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.vfs_write_call '' 'Write to disk' 'calls/s' 'vfs' '' 'stacked' 20066 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.vfs_read_call '' 'Read from disk' 'calls/s' 'vfs' '' 'stacked' 20068 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.vfs_write_bytes '' 'Bytes written on disk' 'bytes/s' 'vfs' '' 'stacked' 20070 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.vfs_read_bytes '' 'Bytes read from disk' 'bytes/s' 'vfs' '' 'stacked' 20071 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.vfs_fsync '' 'Calls for <code>vfs_fsync</code>' 'calls/s' 'vfs' '' 'stacked' 20072 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.vfs_open '' 'Calls for <code>vfs_open</code>' 'calls/s' 'vfs' '' 'stacked' 20074 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.vfs_create '' 'Calls for <code>vfs_create</code>' 'calls/s' 'vfs' '' 'stacked' 20076 5 '' 'ebpf.plugin' 'vfs'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.file_open '' 'Number of open files' 'calls/s' 'file_access' '' 'stacked' 20061 5 '' 'ebpf.plugin' 'process'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.file_closed '' 'Files closed' 'calls/s' 'file_access' '' 'stacked' 20063 5 '' 'ebpf.plugin' 'process'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.oomkills '' 'OOM kills' 'kills' 'mem' '' 'stacked' 20020 5 '' 'ebpf.plugin' 'oomkill'
DIMENSION netdata '' absolute 1 1
DIMENSION apps.plugin '' absolute 1 1
DIMENSION python.d.plugin '' absolute 1 1
DIMENSION tc-qos-helper '' absolute 1 1
DIMENSION go.d.plugin '' absolute 1 1
DIMENSION ebpf.plugin '' absolute 1 1
DIMENSION fail2ban '' absolute 1 1
DIMENSION sql '' absolute 1 1
DIMENSION email '' absolute 1 1
DIMENSION logs '' absolute 1 1
DIMENSION nms '' absolute 1 1
DIMENSION ssh '' absolute 1 1
DIMENSION time '' absolute 1 1
DIMENSION dhcp '' absolute 1 1
DIMENSION dns '' absolute 1 1
DIMENSION antivirus '' absolute 1 1
DIMENSION cron '' absolute 1 1
DIMENSION X '' absolute 1 1
DIMENSION ksmd '' absolute 1 1
DIMENSION system '' absolute 1 1
DIMENSION kernel '' absolute 1 1
DIMENSION node '' absolute 1 1
DIMENSION other '' absolute 1 1
CHART apps.shmget_call '' 'Calls to syscall <code>shmget(2)</code>.' 'calls/s' 'ipc shm (eBPF)' '' 'stacked' 20191 5 '' 'ebpf.plugin' 'shm'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.shmat_call '' 'Calls to syscall <code>shmat(2)</code>.' 'calls/s' 'ipc shm (eBPF)' '' 'stacked' 20192 5 '' 'ebpf.plugin' 'shm'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.shmdt_call '' 'Calls to syscall <code>shmdt(2)</code>.' 'calls/s' 'ipc shm (eBPF)' '' 'stacked' 20193 5 '' 'ebpf.plugin' 'shm'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
CHART apps.shmctl_call '' 'Calls to syscall <code>shmctl(2)</code>.' 'calls/s' 'ipc shm (eBPF)' '' 'stacked' 20194 5 '' 'ebpf.plugin' 'shm'
DIMENSION netdata '' incremental 1 1
DIMENSION apps.plugin '' incremental 1 1
DIMENSION python.d.plugin '' incremental 1 1
DIMENSION tc-qos-helper '' incremental 1 1
DIMENSION go.d.plugin '' incremental 1 1
DIMENSION ebpf.plugin '' incremental 1 1
DIMENSION fail2ban '' incremental 1 1
DIMENSION sql '' incremental 1 1
DIMENSION email '' incremental 1 1
DIMENSION logs '' incremental 1 1
DIMENSION nms '' incremental 1 1
DIMENSION ssh '' incremental 1 1
DIMENSION time '' incremental 1 1
DIMENSION dhcp '' incremental 1 1
DIMENSION dns '' incremental 1 1
DIMENSION antivirus '' incremental 1 1
DIMENSION cron '' incremental 1 1
DIMENSION X '' incremental 1 1
DIMENSION ksmd '' incremental 1 1
DIMENSION system '' incremental 1 1
DIMENSION kernel '' incremental 1 1
DIMENSION node '' incremental 1 1
DIMENSION other '' incremental 1 1
2022-07-16 20:12:40: ebpf.plugin ERROR : EBPF MDFLUSH : Cannot monitor MD devices, because md is not loaded. (errno 22, Invalid argument)
libbpf: prog 'netdata_tcp_v4_connect_fexit': BPF program load failed: Invalid argument
libbpf: prog 'netdata_tcp_v4_connect_fexit': -- BEGIN PROG LOAD LOG --
last insn is not an exit or jmp
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: failed to load program 'netdata_tcp_v4_connect_fexit'
libbpf: failed to load object 'socket_bpf'
libbpf: failed to load BPF skeleton 'socket_bpf': -22
failed to load BPF object: -22
2022-07-16 20:12:42: ebpf.plugin ERROR : EBPF SOCKET : Cannot open or load BPF file for thread socket (errno 22, Invalid argument)
CHART system.softirq_latency '' 'Software IRQ latency' 'milliseconds' 'softirqs' '' 'stacked' 951 5 '' 'ebpf.plugin' 'softirq'
DIMENSION HI HI incremental 1 1
DIMENSION TIMER TIMER incremental 1 1
DIMENSION NET_TX NET_TX incremental 1 1
DIMENSION NET_RX NET_RX incremental 1 1
DIMENSION BLOCK BLOCK incremental 1 1
DIMENSION IRQ_POLL IRQ_POLL incremental 1 1
DIMENSION TASKLET TASKLET incremental 1 1
DIMENSION SCHED SCHED incremental 1 1
DIMENSION HRTIMER HRTIMER incremental 1 1
DIMENSION RCU RCU incremental 1 1
CHART system.hardirq_latency '' 'Hardware IRQ latency' 'milliseconds' 'interrupts' '' 'stacked' 2900 5 '' 'ebpf.plugin' 'hardirq'
DIMENSION apic_thermal apic_thermal incremental 1 1
DIMENSION apic_threshold apic_threshold incremental 1 1
DIMENSION apic_error apic_error incremental 1 1
DIMENSION apic_deferred_error apic_deferred_error incremental 1 1
DIMENSION apic_spurious apic_spurious incremental 1 1
DIMENSION func_call func_call incremental 1 1
DIMENSION func_call_single func_call_single incremental 1 1
DIMENSION reschedule reschedule incremental 1 1
DIMENSION local_timer local_timer incremental 1 1
DIMENSION irq_work irq_work incremental 1 1
DIMENSION x86_platform_ipi x86_platform_ipi incremental 1 1
CHART filesystem.file_descriptor '' 'Open and close calls' 'calls/s' 'file_access' '' 'line' 2195 5 '' 'ebpf.plugin' 'filedescriptor'
DIMENSION do_sys_open open incremental 1 1
DIMENSION __close_fd close incremental 1 1
CHART filesystem.vfs_deleted_objects '' 'Remove files' 'calls/s' 'vfs' '' 'line' 2150 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_unlink delete incremental 1 1
CHART filesystem.vfs_io '' 'Calls to IO' 'calls/s' 'vfs' '' 'line' 2151 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_read read incremental 1 1
DIMENSION vfs_write write incremental 1 1
CHART filesystem.vfs_io_bytes '' 'Bytes written and read' 'bytes/s' 'vfs' '' line 2152 5 '' 'ebpf.plugin' 'filesystem'
DIMENSION vfs_read read incremental 1 1
DIMENSION vfs_write write incremental -1 1
CHART filesystem.vfs_fsync '' 'Calls for <code>vfs_fsync</code>' 'calls/s' 'vfs' '' 'line' 2154 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_fsync fsync incremental 1 1
CHART filesystem.vfs_open '' 'Calls for <code>vfs_open</code>' 'calls/s' 'vfs' '' 'line' 2156 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_open open incremental 1 1
CHART filesystem.vfs_create '' 'Calls for <code>vfs_create</code>' 'calls/s' 'vfs' '' 'line' 2158 5 '' 'ebpf.plugin' 'vfs'
DIMENSION vfs_create create incremental 1 1
BEGIN apps.oomkills
SET netdata = 0
SET apps.plugin = 0
SET python.d.plugin = 0
SET tc-qos-helper = 0
SET go.d.plugin = 0
SET ebpf.plugin = 0
SET fail2ban = 0
SET sql = 0
SET email = 0
SET logs = 0
SET nms = 0
SET ssh = 0
SET time = 0
SET dhcp = 0
SET dns = 0
SET antivirus = 0
SET cron = 0
SET X = 0
SET ksmd = 0
SET system = 0
SET kernel = 0
SET node = 0
SET other = 0
END
BEGIN netdata.ebpf_threads
SET total = 10
SET running = 6
END
BEGIN netdata.ebpf_load_methods
SET legacy = 6
SET co-re = 0
END
BEGIN system.exit
SET do_exit = 4
SET release_task = -4
END
BEGIN system.process_thread
SET _do_fork = 3
SET sys_clone = 2
END
BEGIN system.process_status
SET process = 1
SET zombie = 0
END
BEGIN apps.process_create
SET netdata = 0
SET apps.plugin = 0
SET python.d.plugin = 0
SET tc-qos-helper = 0
SET go.d.plugin = 0
SET ebpf.plugin = 1
SET fail2ban = 0
SET sql = 0
SET email = 2
SET logs = 0
SET nms = 0
SET ssh = 0
SET time = 0
SET dhcp = 0
SET dns = 0
SET antivirus = 0
SET cron = 0
SET X = 0
SET ksmd = 0
SET system = 0
SET kernel = 0
SET node = 0
SET other = 0
END
BEGIN apps.thread_create
SET netdata = 0
SET apps.plugin = 0
SET python.d.plugin = 0
SET tc-qos-helper = 0
SET go.d.plugin = 0
SET ebpf.plugin = 1
SET fail2ban = 0
SET sql = 0
SET email = 1
SET logs = 0
SET nms = 0
SET ssh = 0
SET time = 0
SET dhcp = 0
SET dns = 0
SET antivirus = 0
SET cron = 0
SET X = 0
SET ksmd = 0
SET system = 0
SET kernel = 0
SET node = 0
SET other = 0
END
BEGIN apps.task_exit
SET netdata = 0
SET apps.plugin = 0
SET python.d.plugin = 0
SET tc-qos-helper = 0
SET go.d.plugin = 0
SET ebpf.plugin = 0
SET fail2ban = 0
SET sql = 0
SET email = 0
SET logs = 0
SET nms = 0
SET ssh = 0
SET time = 0
SET dhcp = 0
SET dns = 0
SET antivirus = 0
SET cron = 0
SET X = 0
SET ksmd = 0
SET system = 0
SET kernel = 0
SET node = 0
SET other = 0
END
BEGIN apps.task_close
SET netdata = 0
SET apps.plugin = 0
SET python.d.plugin = 0
SET tc-qos-helper = 0
SET go.d.plugin = 0
SET ebpf.plugin = 0
SET fail2ban = 0
SET sql = 0
SET email = 0
SET logs = 0
SET nms = 0
SET ssh = 0
SET time = 0
SET dhcp = 0
SET dns = 0
SET antivirus = 0
SET cron = 0
SET X = 0
SET ksmd = 0
SET system = 0
SET kernel = 0
SET node = 0
SET other = 0
END
CHART mem.file_sync '' 'Monitor calls for <code>fsync(2)</code> and <code>fdatasync(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21300 5 '' 'ebpf.plugin' 'sync'
DIMENSION fsync fsync incremental 1 1
DIMENSION fdatasync fdatasync incremental 1 1
CHART mem.memory_map '' 'Monitor calls for <code>msync(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21301 5 '' 'ebpf.plugin' 'sync'
DIMENSION msync msync incremental 1 1
CHART mem.sync '' 'Monitor calls for <code>sync(2)</code> and <code>syncfs(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21302 5 '' 'ebpf.plugin' 'sync'
DIMENSION sync sync incremental 1 1
DIMENSION syncfs syncfs incremental 1 1
CHART mem.file_segment '' 'Monitor calls for <code>sync_file_range(2)</code>.' 'calls/s' 'synchronization (eBPF)' '' 'line' 21303 5 '' 'ebpf.plugin' 'sync'
DIMENSION sync_file_range sync_file_range incremental 1 1
2022-07-16 20:12:42: ebpf.plugin INFO  : EBPF MDFLUSH : thread with task id 4884 finished
[1]    4868 segmentation fault (core dumped)  ./plugins.d/ebpf.plugin

Expected behavior

The plugin should not produce a core dump on this system.

Steps to reproduce

  1. Start netdata agent ...

Installation method

kickstart.sh

System info

❯ uname -a; grep -HvE "^#|URL" /etc/*release
Linux www 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux
/etc/os-release:NAME="openSUSE Leap"
/etc/os-release:VERSION="15.4"
/etc/os-release:ID="opensuse-leap"
/etc/os-release:ID_LIKE="suse opensuse"
/etc/os-release:VERSION_ID="15.4"
/etc/os-release:PRETTY_NAME="openSUSE Leap 15.4"
/etc/os-release:ANSI_COLOR="0;32"
/etc/os-release:CPE_NAME="cpe:/o:opensuse:leap:15.4"
/etc/os-release:LOGO="distributor-logo-Leap"

Netdata build info

❯ /opt/netdata/sbin/netdata -W buildinfo
Version: netdata v1.35.0-191-g5b1c80ebc
Configure options:  '--prefix=/opt/netdata/usr' '--sysconfdir=/opt/netdata/etc' '--localstatedir=/opt/netdata/var' '--libexecdir=/opt/netdata/usr/libexec' '--libdir=/opt/netdata/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--enable-cloud' '--without-bundled-protobuf' '--disable-dependency-tracking' '--with-bundled-libJudy' 'CFLAGS=-static -O2 -I/openssl-static/include -pipe' 'LDFLAGS=-static -L/openssl-static/lib' 'PKG_CONFIG_PATH=/openssl-static/lib/pkgconfig'
Install type: kickstart-static
    Binary architecture: x86_64
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES 
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    NO
    EBPF:                    YES
    IPMI:                    NO
    NFACCT:                  NO
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES

Additional info

No response

thiagoftsm commented 2 years ago

Hello @tickez ,

Thank you for your report, I will investigate this tomorrow :handshake: .

Best regards!

thiagoftsm commented 2 years ago

Hello @tickez ,

I built an environment with Vagrant OpenSuse 15.4 image. This image has the following kernel:

localhost:/usr/libexec/netdata # uname -a
Linux localhost 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux

Considering the logs you gave us, I suppose you installed netdata with following command (or using wget):

curl https://my-netdata.io/kickstart.sh > /tmp/netdata-kickstart.sh && sh /tmp/netdata-kickstart.sh

firstly can you confirm my conclusion? Now, after to run this command I started netdata with default configuration during 30 minutes and I did not observe any issues with ebpf.plugin on my environment. After this I decided to compile direct on my VM netdata, and for this I ran the following steps:

$ git clone https://github.com/thiagoftsm/netdata.git
$ cd netdata
$ sudo su
# ./packaging/installer/install-required-packages.sh --yes
# zypper install gcc-c++
# ./netdata-installer.sh --dont-wait --dont-start-it 2> err >out

Before we proceed (@Ferroin it looks like we are missing a packages to install netdata on OpenSuse 15.4`).

After to run these commands netdata was installed with success, and I ran it another 30 minutes without any issue. I stopped netdata and after this I enabled all ebpf.plugin threads, after to start I again did not have errors.

Of course I believe in you, and the fact that I cannot recreate the issue force me to require you run few commands on your environment to address the issue properly, so when you have a time, please, run the following commands and give us the output:

# coredumpctl list
# coredumpctl dump --output compare PID
# gdb /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin compare
# bt

Where PID was the PID that crashed. I am needing the backtrace (bt) to understand what is happening in your environment.

Best regards!

tickez commented 2 years ago

Hi @thiagoftsm

Thanks for looking into this. I used the wget kickstarter with this exact command line wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --no-updates --disable-telemetry

The machine runs virtualized on kvm and I was able to reproduce the exact same behavior on another openSUSE 15.4 running on VMWare Fusion. As you can see from the below, I had netdata installed initially through compilation and then re-installed using the static binaries. The plugin dumps core in both cases but with a slightly different reason but both seem to be related to a malloc().

❯ coredumpctl list
TIME                           PID UID GID SIG     COREFILE EXE                                                      SIZE
Mon 2022-07-11 10:46:11 CEST  5553 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:46:34 CEST  7403 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:47:17 CEST  9684 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:47:47 CEST  9840 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:48:16 CEST  9879 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:48:47 CEST 10026 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:49:16 CEST 10061 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:49:46 CEST 10231 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:50:16 CEST 10269 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:50:46 CEST 10426 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Mon 2022-07-11 10:51:16 CEST 10482 491 483 SIGABRT missing  /usr/libexec/netdata/plugins.d/ebpf.plugin                n/a
Fri 2022-07-15 10:54:51 CEST 10609 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin             582.5K
Fri 2022-07-15 10:55:04 CEST 34775 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin             293.7K
Fri 2022-07-15 11:00:31 CEST  7070 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin             305.7K
Fri 2022-07-15 17:24:08 CEST  2153 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin               8.8M
Sat 2022-07-16 19:21:02 CEST  2243 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin              11.6M
Sat 2022-07-16 19:21:21 CEST  2669 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin              11.8M
Sat 2022-07-16 19:21:38 CEST  2705 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin              12.0M
Sat 2022-07-16 19:23:49 CEST  2738 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin              11.8M
Sat 2022-07-16 19:24:02 CEST  3087 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin              11.6M
Sat 2022-07-16 19:37:02 CEST 28288 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin              13.3M
Sat 2022-07-16 19:41:59 CEST 32795 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin              11.3M
Sat 2022-07-16 19:42:17 CEST   493 491 483 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin               9.5M
Sat 2022-07-16 19:46:36 CEST  1240   0   0 SIGABRT present  /usr/libexec/netdata/plugins.d/ebpf.plugin               7.0M
Sat 2022-07-16 20:05:15 CEST  3772 491 483 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   3.2M
Sat 2022-07-16 20:12:42 CEST  4868   0   0 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   1.8M
Sun 2022-07-17 15:21:29 CEST 11672 491 483 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   1.8M
Sun 2022-07-17 15:21:43 CEST 11939 491 483 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   4.6M
Sun 2022-07-17 15:21:59 CEST 11969 491 483 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   1.8M
Sun 2022-07-17 15:22:16 CEST 12059 491 483 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   3.2M
Sun 2022-07-17 15:22:36 CEST 12096 491 483 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   1.8M
Sun 2022-07-17 15:22:54 CEST 12150 491 483 SIGSEGV present  /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin   1.8M

I provide only the most recent core dump

❯ coredumpctl dump --output compare 12150
           PID: 12150 (ebpf.plugin)
           UID: 491 (netdata)
           GID: 483 (netdata)
        Signal: 11 (SEGV)
     Timestamp: Sun 2022-07-17 15:22:53 CEST (3h 22min ago)
  Command Line: /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1
    Executable: /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin
 Control Group: /system.slice/netdata.service
          Unit: netdata.service
         Slice: system.slice
       Boot ID: 235a54d7c42647119a62fd358faef149
    Machine ID: cc07d9b2683f40c2f4b40b4e00000c9c
      Hostname: www
       Storage: /var/lib/systemd/coredump/core.ebpf\x2eplugin.491.235a54d7c42647119a62fd358faef149.12150.1658064173000000.zst (present)
     Disk Size: 1.8M
       Message: Process 12150 (ebpf.plugin) of user 491 dumped core.

                Found module /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin without build-id.
                Found module linux-vdso.so.1 with build-id: 1d8950f257479087bcdda44a121002411cba55af
                Stack trace of thread 12159:
                #0  0x000000000071de5d n/a (/opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin + 0x31de5d)
                #1  0x00007f300c19a9d0 n/a (n/a + 0x0)

From the debugger

❯ gdb /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin compare
GNU gdb (GDB; SUSE Linux Enterprise 15) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin...
[New LWP 12159]
[New LWP 12163]
[New LWP 12150]
[New LWP 12151]
[New LWP 12154]
[New LWP 12155]
[New LWP 12157]
[New LWP 12160]
[New LWP 12161]
[New LWP 12162]
[New LWP 12164]
[New LWP 12165]
[New LWP 12166]
[New LWP 12167]
[New LWP 12168]
[New LWP 12169]
[New LWP 12152]
[New LWP 12170]
Core was generated by `/opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  get_meta (p=p@entry=0x7f300c19a9d0 "") at src/malloc/mallocng/meta.h:141
141 src/malloc/mallocng/meta.h: Datei oder Verzeichnis nicht gefunden.
[Current thread is 1 (LWP 12159)]
(gdb) bt
#0  get_meta (p=p@entry=0x7f300c19a9d0 "") at src/malloc/mallocng/meta.h:141
#1  0x000000000071e1d5 in __libc_free (p=0x7f300c19a9d0) at src/malloc/mallocng/free.c:105
#2  0x0000000000441b65 in thread_cleanup ()
#3  0x0000000000441cda in thread_start ()
#4  0x000000000072bc8f in start (p=0x7f300bfd1ac8) at src/thread/pthread_create.c:203
#5  0x000000000072d876 in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC
(gdb) 

Let me know if you need anything more, I can create a clone from the VMWare VM in case required.

thiagoftsm commented 2 years ago

Hello @tickez ,

Thank you for your additional information. After you publish these message, I tried to recreate the issue with Ubuntu 22.04 and OpenSuse without success, but I could create the same issues with Arch Linux. I already started working with this issue, and I expect to have the final solution in next hours.

Best regards!

mailinglists35 commented 1 year ago

this is occuring on OL9.2

$ cat /var/log/netdata/error.log
2023-05-28 04:35:47: netdata INFO  : MAIN : IEEE754: system is using IEEE754 DOUBLE PRECISION values
2023-05-28 04:35:47: netdata INFO  : MAIN : TIMEZONE: using the link of /etc/localtime: 'Europe/Bucharest'
2023-05-28 04:35:47: netdata INFO  : MAIN : TIMEZONE: fixed as 'Europe/Bucharest'
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: next: initialize ML
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in      18 ms, initialize ML - next: initialize signals
2023-05-28 04:35:47: netdata INFO  : MAIN : SIGNAL: Not enabling reaper
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, initialize signals - next: initialize static threads
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, initialize static threads - next: initialize web server
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in      48 ms, initialize web server - next: initialize httpd server
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, initialize httpd server - next: set resource limits
2023-05-28 04:35:47: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1024, max = 524288
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, set resource limits - next: become daemon
2023-05-28 04:35:47: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 0
2023-05-28 04:35:47: netdata INFO  : MAIN : Adjusted netdata scheduling policy to batch (3), with priority 0.
2023-05-28 04:35:47: netdata INFO  : MAIN : Running with process scheduling policy 'batch', nice level 19
2023-05-28 04:35:47: netdata INFO  : MAIN : netdata started on pid 887.
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       3 ms, become daemon - next: initialize threads after fork
2023-05-28 04:35:47: netdata INFO  : MAIN : Set threads stack size to 8388608 bytes
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, initialize threads after fork - next: initialize registry
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, initialize registry - next: fork the spawn server
2023-05-28 04:35:47: netdata INFO  : MAIN : Initializing spawn client.
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       1 ms, fork the spawn server - next: collecting system info
2023-05-28 04:35:47: netdata INFO  : MAIN : Executing /usr/libexec/netdata/plugins.d/system-info.sh
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_NAME=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_CONTAINER_OS_DETECTION=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_CONTAINER_IS_OFFICIAL_IMAGE=false
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_HOST_OS_NAME=Oracle Linux Server
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_HOST_OS_ID=ol
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_HOST_OS_ID_LIKE=fedora
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION=9.2
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_HOST_OS_VERSION_ID=9.2
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_HOST_OS_DETECTION=/etc/os-release
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_HOST_IS_K8S_NODE=false
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=5.15.0-101.103.2.1.el9uek.x86_64
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=systemd-detect-virt
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER=none
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=systemd-detect-virt
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=8
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_FREQ=3500000000
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_CPU_DETECTION=lscpu
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_RAM=16698769408
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=250327785472
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_INSTANCE_CLOUD_TYPE=unknown
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_INSTANCE_CLOUD_INSTANCE_TYPE=unknown
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA_INSTANCE_CLOUD_INSTANCE_REGION=unknown
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in     141 ms, collecting system info - next: initialize RRD structures
2023-05-28 04:35:47: netdata INFO  : MAIN : SQLite database /var/cache/netdata/netdata-meta.db initialization
2023-05-28 04:35:47: netdata INFO  : MAIN : Database version is 0, current version is 7. Running migration for metadata ...
2023-05-28 04:35:47: netdata INFO  : MAIN : Running database migration v0 to v1
2023-05-28 04:35:47: netdata INFO  : MAIN : Running "v1 to v2" database migration
2023-05-28 04:35:47: netdata INFO  : MAIN : Running "v2 to v3" database migration
2023-05-28 04:35:47: netdata INFO  : MAIN : Running database migration v3 to v4
2023-05-28 04:35:47: netdata INFO  : MAIN : Running "v4 to v5" database migration
2023-05-28 04:35:47: netdata INFO  : MAIN : Running "v5 to v6" database migration
2023-05-28 04:35:47: netdata INFO  : MAIN : Running "v6 to v7" database migration
2023-05-28 04:35:47: netdata INFO  : MAIN : SQLite database initialization completed
2023-05-28 04:35:47: netdata INFO  : MAIN : SQLite database /var/cache/netdata/context-meta.db initialization
2023-05-28 04:35:47: netdata INFO  : MAIN : Database version is 0, current version is 1. Running migration for context ...
2023-05-28 04:35:47: netdata INFO  : MAIN : Running database migration v0 to v1
2023-05-28 04:35:47: netdata INFO  : MAIN : Cannot open the file /var/lib/netdata/health.silencers.json, so Netdata will work with the default health configuration.
2023-05-28 04:35:47: netdata INFO  : MAIN : CONFIG: cannot load user config '/etc/netdata/stream.conf'. Will try stock config.
2023-05-28 04:35:47: netdata INFO  : MAIN : DBENGINE: Initializing ...
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : thread created with task id 1076
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : set name of thread 1076 to DBENGINE_INIT
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : thread created with task id 1077
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : set name of thread 1077 to DBENGINE_INIT
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : thread created with task id 1078
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : set name of thread 1078 to DBENGINE_INIT
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: found 0 files in path /var/cache/netdata/dbengine
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: data files not found, creating in path "/var/cache/netdata/dbengine".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: found 0 files in path /var/cache/netdata/dbengine-tier2
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: data files not found, creating in path "/var/cache/netdata/dbengine-tier2".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: creating new data and journal files in path /var/cache/netdata/dbengine-tier2
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: creating new data and journal files in path /var/cache/netdata/dbengine
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: found 0 files in path /var/cache/netdata/dbengine-tier1
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: data files not found, creating in path "/var/cache/netdata/dbengine-tier1".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: creating new data and journal files in path /var/cache/netdata/dbengine-tier1
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: created data file "/var/cache/netdata/dbengine-tier2/datafile-1-0000000001.ndf".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: created journal file "/var/cache/netdata/dbengine-tier2/journalfile-1-0000000001.njf".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: populating retention to MRG from 1 journal files of tier 2, using 1 threads...
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : thread with task id 1078 finished
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: created data file "/var/cache/netdata/dbengine/datafile-1-0000000001.ndf".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: created data file "/var/cache/netdata/dbengine-tier1/datafile-1-0000000001.ndf".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: created journal file "/var/cache/netdata/dbengine/journalfile-1-0000000001.njf".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: populating retention to MRG from 1 journal files of tier 0, using 1 threads...
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : thread with task id 1076 finished
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: created journal file "/var/cache/netdata/dbengine-tier1/journalfile-1-0000000001.njf".
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : DBENGINE: populating retention to MRG from 1 journal files of tier 1, using 1 threads...
2023-05-28 04:35:47: netdata INFO  : DBENGINE_INIT : thread with task id 1077 finished
2023-05-28 04:35:47: netdata INFO  : MAIN : DBENGINE: tier 0 is ready for data collection and queries
2023-05-28 04:35:47: netdata INFO  : MAIN : DBENGINE: tier 1 is ready for data collection and queries
2023-05-28 04:35:47: netdata INFO  : MAIN : DBENGINE: tier 2 is ready for data collection and queries
2023-05-28 04:35:47: netdata INFO  : METASYNC : Starting metadata sync thread with 1024 entries command queue
2023-05-28 04:35:47: netdata INFO  : MAIN : SQLite metadata sync initialization complete
2023-05-28 04:35:47: netdata INFO  : MAIN : Host 'localhost.localdomain' (at registry as 'localhost.localdomain') with guid 'f8889f32-fcf7-11ed-bbba-51084a8ba1a2' initialized, os 'linux', timezone 'Europe/Bucharest', tags '', program_name 'netdata', program_version 'v1.39.0-69-nightly', update every 1, memory mode dbengine, history entries 0, streaming disabled (to '' with api key ''), health enabled, cache_dir '/var/cache/netdata', alarms default handler '', alarms default recipient ''
2023-05-28 04:35:47: netdata INFO  : MAIN : Creating archived hosts
2023-05-28 04:35:47: netdata INFO  : MAIN : Created 0 archived hosts
2023-05-28 04:35:47: netdata INFO  : MAIN : ACLK sync initialization completed
2023-05-28 04:35:47: netdata INFO  : ACLKSYNC : Starting ACLK synchronization thread
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in      62 ms, initialize RRD structures - next: check for incomplete shutdown
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, check for incomplete shutdown - next: collect claiming info
2023-05-28 04:35:47: netdata INFO  : MAIN : Unable to load '/var/lib/netdata/cloud.d/claimed_id', setting state to AGENT_UNCLAIMED
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       0 ms, collect claiming info - next: collect host labels
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       5 ms, collect host labels - next: start the static threads
2023-05-28 04:35:47: netdata INFO  : P[timex] : thread created with task id 1133
2023-05-28 04:35:47: netdata INFO  : P[timex] : set name of thread 1133 to P[timex]
2023-05-28 04:35:47: netdata INFO  : HEALTH : thread created with task id 1135
2023-05-28 04:35:47: netdata INFO  : P[idlejitter] : thread created with task id 1134
2023-05-28 04:35:47: netdata INFO  : STATS_SQLITE3 : thread created with task id 1138
2023-05-28 04:35:47: netdata INFO  : STATS_SQLITE3 : set name of thread 1138 to STATS_SQLITE3
2023-05-28 04:35:47: netdata INFO  : HEALTH : set name of thread 1135 to HEALTH
2023-05-28 04:35:47: netdata INFO  : PLUGINSD : thread created with task id 1139
2023-05-28 04:35:47: netdata INFO  : PLUGINSD : set name of thread 1139 to PLUGINSD
2023-05-28 04:35:47: netdata INFO  : STATS_GLOBAL : thread created with task id 1136
2023-05-28 04:35:47: netdata INFO  : STATSD_FLUSH : thread created with task id 1141
2023-05-28 04:35:47: netdata INFO  : STATSD_FLUSH : set name of thread 1141 to STATSD_FLUSH
2023-05-28 04:35:47: netdata INFO  : STATS_WORKERS : thread created with task id 1137
2023-05-28 04:35:47: netdata INFO  : EXPORTING : thread created with task id 1142
2023-05-28 04:35:47: netdata INFO  : P[idlejitter] : set name of thread 1134 to P[idlejitter]
2023-05-28 04:35:47: netdata INFO  : SERVICE : thread created with task id 1140
2023-05-28 04:35:47: netdata INFO  : STATS_WORKERS : set name of thread 1137 to STATS_WORKERS
2023-05-28 04:35:47: netdata INFO  : STATS_GLOBAL : set name of thread 1136 to STATS_GLOBAL
2023-05-28 04:35:47: netdata INFO  : EXPORTING : set name of thread 1142 to EXPORTING
2023-05-28 04:35:47: netdata INFO  : WEB[1] : thread created with task id 1143
2023-05-28 04:35:47: netdata INFO  : WEB[1] : set name of thread 1143 to WEB[1]
2023-05-28 04:35:47: netdata INFO  : PD[apps] : thread created with task id 1144
2023-05-28 04:35:47: netdata INFO  : EXPORTING : CONFIG: cannot load user exporting config '/etc/netdata/exporting.conf'. Will try the stock version.
2023-05-28 04:35:47: netdata INFO  : ACLK_MAIN : thread created with task id 1145
2023-05-28 04:35:47: netdata INFO  : ACLK_MAIN : set name of thread 1145 to ACLK_MAIN
2023-05-28 04:35:47: netdata INFO  : PD[go.d] : thread created with task id 1146
2023-05-28 04:35:47: netdata INFO  : ACLK_MAIN : Waiting for Cloud to be enabled
2023-05-28 04:35:47: netdata INFO  : SERVICE : set name of thread 1140 to SERVICE
2023-05-28 04:35:47: netdata INFO  : PD[go.d] : set name of thread 1146 to PD[go.d]
2023-05-28 04:35:47: netdata INFO  : PD[apps] : set name of thread 1144 to PD[apps]
2023-05-28 04:35:47: netdata INFO  : RRDCONTEXT : thread created with task id 1147
2023-05-28 04:35:47: netdata INFO  : RRDCONTEXT : set name of thread 1147 to RRDCONTEXT
2023-05-28 04:35:47: netdata INFO  : PD[ebpf] : thread created with task id 1148
2023-05-28 04:35:47: netdata INFO  : PD[ebpf] : set name of thread 1148 to PD[ebpf]
2023-05-28 04:35:47: netdata INFO  : REPLAY[1] : thread created with task id 1149
2023-05-28 04:35:47: netdata INFO  : REPLAY[1] : set name of thread 1149 to REPLAY[1]
2023-05-28 04:35:47: netdata INFO  : PD[python.d] : thread created with task id 1151
2023-05-28 04:35:47: netdata INFO  : P[tc] : thread created with task id 1150
2023-05-28 04:35:47: netdata INFO  : P[tc] : set name of thread 1150 to P[tc]
2023-05-28 04:35:47: netdata INFO  : PD[python.d] : set name of thread 1151 to PD[python.d]
2023-05-28 04:35:47: netdata INFO  : P[diskspace] : thread created with task id 1153
2023-05-28 04:35:47: netdata INFO  : P[diskspace] : set name of thread 1153 to P[diskspace]
2023-05-28 04:35:47: netdata INFO  : PD[charts.d] : thread created with task id 1156
2023-05-28 04:35:47: netdata INFO  : P[cgroups] : thread created with task id 1158
2023-05-28 04:35:47: netdata INFO  : PD[charts.d] : set name of thread 1156 to PD[charts.d]
2023-05-28 04:35:47: netdata INFO  : P[cgroups] : set name of thread 1158 to P[cgroups]
2023-05-28 04:35:47: netdata INFO  : PD[go.d] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/go.d.plugin' running on pid 1152
2023-05-28 04:35:47: netdata INFO  : P[proc] : thread created with task id 1155
2023-05-28 04:35:47: netdata INFO  : PREDICT : thread created with task id 1160
2023-05-28 04:35:47: netdata INFO  : PREDICT : set name of thread 1160 to PREDICT
2023-05-28 04:35:47: netdata INFO  : P[proc] : set name of thread 1155 to P[proc]
2023-05-28 04:35:47: netdata INFO  : WEB[1] : To use encryption it is necessary to set "ssl certificate" and "ssl key" in [web] !

2023-05-28 04:35:47: netdata INFO  : PD[apps] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/apps.plugin' running on pid 1157
2023-05-28 04:35:47: netdata INFO  : PD[perf] : thread created with task id 1161
2023-05-28 04:35:47: netdata INFO  : PD[perf] : set name of thread 1161 to PD[perf]
2023-05-28 04:35:47: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1154
2023-05-28 04:35:47: netdata INFO  : WEB[1] : starting worker 2
2023-05-28 04:35:47: netdata INFO  : PD[python.d] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 1159
2023-05-28 04:35:47: netdata INFO  : WEB[1] : starting worker 3
2023-05-28 04:35:47: netdata INFO  : WEB[2] : thread created with task id 1162
2023-05-28 04:35:47: netdata INFO  : WEB[2] : set name of thread 1162 to WEB[2]
2023-05-28 04:35:47: netdata INFO  : WEB[1] : starting worker 4
2023-05-28 04:35:47: netdata INFO  : WEB[3] : thread created with task id 1163
2023-05-28 04:35:47: netdata INFO  : WEB[3] : set name of thread 1163 to WEB[3]
2023-05-28 04:35:47: netdata INFO  : WEB[1] : starting worker 5
2023-05-28 04:35:47: netdata INFO  : WEB[4] : thread created with task id 1164
2023-05-28 04:35:47: netdata INFO  : WEB[1] : starting worker 6
2023-05-28 04:35:47: netdata INFO  : WEB[4] : set name of thread 1164 to WEB[4]
2023-05-28 04:35:47: netdata INFO  : WEB[4] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO  : PD[debugfs] : thread created with task id 1168
2023-05-28 04:35:47: netdata INFO  : PD[debugfs] : set name of thread 1168 to PD[debugfs]
2023-05-28 04:35:47: netdata INFO  : TRAIN[0] : thread created with task id 1166
2023-05-28 04:35:47: netdata INFO  : TRAIN[0] : set name of thread 1166 to TRAIN[0]
2023-05-28 04:35:47: netdata INFO  : WEB[6] : thread created with task id 1170
2023-05-28 04:35:47: netdata INFO  : WEB[6] : set name of thread 1170 to WEB[6]
2023-05-28 04:35:47: netdata INFO  : PD[ioping] : thread created with task id 1171
2023-05-28 04:35:47: netdata INFO  : PD[ioping] : set name of thread 1171 to PD[ioping]
2023-05-28 04:35:47: netdata INFO  : WEB[4] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO  : EXPORTING : No connector instances to activate
2023-05-28 04:35:47: netdata INFO  : EXPORTING : EXPORTING: no exporting connectors configured
2023-05-28 04:35:47: netdata INFO  : EXPORTING : cleaning up...
2023-05-28 04:35:47: netdata INFO  : EXPORTING : thread with task id 1142 finished
2023-05-28 04:35:47: netdata INFO  : TRAIN[3] : thread created with task id 1174
2023-05-28 04:35:47: netdata INFO  : TRAIN[3] : set name of thread 1174 to TRAIN[3]
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       1 ms, start the static threads - next: initialize commands API
2023-05-28 04:35:47: netdata INFO  : MAIN : Initializing command server.
2023-05-28 04:35:47: netdata INFO  : WEB[6] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO  : WEB[6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO  : WEB[1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO  : WEB[1] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO  : WEB[3] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO  : WEB[3] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO  : P[diskspace slo : thread created with task id 1173
2023-05-28 04:35:47: netdata INFO  : P[diskspace slo : set name of thread 1173 to P[diskspace slo
2023-05-28 04:35:47: netdata INFO  : PD[debugfs] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/debugfs.plugin' running on pid 1178
2023-05-28 04:35:47: netdata INFO  : P[proc netdev] : thread created with task id 1176
2023-05-28 04:35:47: netdata INFO  : P[proc netdev] : set name of thread 1176 to P[proc netdev]
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: in       1 ms, initialize commands API - next: ready
2023-05-28 04:35:47: netdata INFO  : MAIN : NETDATA STARTUP: completed in 296 ms. Enjoy real-time performance monitoring!
2023-05-28 04:35:47: netdata INFO  : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'START' '-' '-'
2023-05-28 04:35:47: netdata INFO  : HEALTH : HEALTH [localhost.localdomain]: Table health_log_f8889f32_fcf7_11ed_bbba_51084a8ba1a2, contains 0 entries.
2023-05-28 04:35:47: netdata INFO  : PD[ioping] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ioping.plugin' running on pid 1177
2023-05-28 04:35:47: netdata INFO  : WEB[2] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO  : WEB[2] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO  : TRAIN[2] : thread created with task id 1169
2023-05-28 04:35:47: netdata INFO  : TRAIN[2] : set name of thread 1169 to TRAIN[2]
2023-05-28 04:35:47: netdata INFO  : WEB[5] : thread created with task id 1165
2023-05-28 04:35:47: netdata INFO  : WEB[5] : set name of thread 1165 to WEB[5]
2023-05-28 04:35:47: netdata INFO  : WEB[5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO  : WEB[5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO  : TRAIN[1] : thread created with task id 1167
2023-05-28 04:35:47: netdata INFO  : TRAIN[1] : set name of thread 1167 to TRAIN[1]
2023-05-28 04:35:47: netdata INFO  : PD[perf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/perf.plugin' running on pid 1172
2023-05-28 04:35:47: netdata INFO  : PD[charts.d] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/charts.d.plugin' running on pid 1180
2023-05-28 04:35:47: netdata INFO  : STATSD_IN[1] : thread created with task id 1187
2023-05-28 04:35:47: netdata INFO  : STATSD_IN[1] : set name of thread 1187 to STATSD_IN[1]
2023-05-28 04:35:47: netdata INFO  : STATSD_IN[1] : POLLFD: LISTENER: listening on 'udp:[::1]:8125'
2023-05-28 04:35:47: netdata INFO  : STATSD_IN[1] : POLLFD: LISTENER: listening on 'udp:127.0.0.1:8125'
2023-05-28 04:35:47: netdata INFO  : STATSD_IN[1] : POLLFD: LISTENER: listening on 'tcp:[::1]:8125'
2023-05-28 04:35:47: netdata INFO  : STATSD_IN[1] : POLLFD: LISTENER: listening on 'tcp:127.0.0.1:8125'
2023-05-28 04:35:47: netdata INFO  : PD[perf] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:47: netdata INFO  : PD[perf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/perf.plugin' (pid 1172) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:47: netdata ERROR : PD[perf] : child pid 1172 exited with code 1.
2023-05-28 04:35:47: netdata ERROR : PD[perf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/perf.plugin' (pid 1172) exited with error code 1 and haven't collected any data. Disabling it.
2023-05-28 04:35:47: netdata INFO  : PD[perf] : thread with task id 1161 finished
2023-05-28 04:35:47: netdata INFO  : PD[ioping] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:47: netdata INFO  : PD[ioping] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ioping.plugin' (pid 1177) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:47: netdata ERROR : PD[ioping] : child pid 1177 exited with code 1.
2023-05-28 04:35:47: netdata ERROR : PD[ioping] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ioping.plugin' (pid 1177) exited with error code 1 and haven't collected any data. Disabling it.
2023-05-28 04:35:47: netdata INFO  : PD[ioping] : thread with task id 1171 finished
2023-05-28 04:35:47: netdata ERROR : MAIN : child pid 1182 exited with code 6.
2023-05-28 04:35:47: netdata ERROR : MAIN : Execution of anonymous statistics script returned 6.
2023-05-28 04:35:47: netdata ERROR : MAIN : Execution of anonymous statistics script returned http code 000.
2023-05-28 04:35:47: netdata INFO  : ANALYTICS : thread created with task id 1275
2023-05-28 04:35:47: netdata INFO  : ANALYTICS : set name of thread 1275 to ANALYTICS
2023-05-28 04:35:47: netdata INFO  : PD[charts.d] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:47: netdata INFO  : PD[charts.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 1180) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:47: netdata INFO  : PD[charts.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 1180) does not generate useful output but it reports success (exits with 0). Will not start it again - it is now disabled..
2023-05-28 04:35:48: netdata INFO  : P[proc] : Using now_boottime_usec() for uptime (dt is 3 ms)
2023-05-28 04:35:48: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:35:48: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1154) disconnected after 13 successful data collections (ENDs).
2023-05-28 04:35:48: netdata ERROR : PD[ebpf] : child pid 1154 core dumped by signal 11.
2023-05-28 04:35:48: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1154) exited with error code -2, but has given useful output in the past (13 times). Waiting a bit before starting it again.
2023-05-28 04:35:49: netdata INFO  : PD[python.d] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:49: netdata INFO  : PD[python.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 1159) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:49: netdata INFO  : PD[python.d] : child pid 1159 killed by signal 15.
2023-05-28 04:35:49: netdata INFO  : PD[python.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 1159) does not generate useful output but it reports success (exits with 0). Will not start it again - it is now disabled..
2023-05-28 04:35:57: netdata INFO  : PD[charts.d] : thread with task id 1156 finished
2023-05-28 04:35:58: netdata ERROR : PLUGIN[cgroups] : child pid 1452 exited with code 1.
2023-05-28 04:35:58: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1499
2023-05-28 04:35:59: netdata INFO  : PD[python.d] : thread with task id 1151 finished
2023-05-28 04:36:00: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:36:00: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1499) disconnected after 14 successful data collections (ENDs).
2023-05-28 04:36:00: netdata ERROR : PD[ebpf] : child pid 1499 core dumped by signal 11.
2023-05-28 04:36:00: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1499) exited with error code -2, but has given useful output in the past (27 times). Waiting a bit before starting it again.
2023-05-28 04:36:10: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1540
2023-05-28 04:36:12: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:36:12: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1540) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:36:12: netdata ERROR : PD[ebpf] : child pid 1540 core dumped by signal 11.
2023-05-28 04:36:12: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1540) exited with error code -2, but has given useful output in the past (27 times). Waiting a bit before starting it again.
2023-05-28 04:36:22: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1581
2023-05-28 04:36:23: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:36:23: netdata INFO  : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1581) disconnected after 12 successful data collections (ENDs).
2023-05-28 04:36:23: netdata ERROR : PD[ebpf] : child pid 1581 core dumped by signal 11.
2023-05-28 04:36:23: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1581) exited with error code -2, but has given useful output in the past (39 times). Waiting a bit before starting it again.
2023-05-28 04:36:33: netdata LOG FLOOD PROTECTION too many logs (201 logs in 46 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1154 seconds.
mailinglists35 commented 1 year ago
$ journalctl -u netdata --no-pager
May 28 04:35:46 localhost systemd[1]: Starting Real time performance monitoring...
May 28 04:35:47 localhost systemd[1]: Started Real time performance monitoring.
May 28 04:35:47 localhost.localdomain netdata[887]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
May 28 04:35:47 localhost.localdomain netdata[887]: 2023-05-28 04:35:47: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
May 28 04:35:47 localhost.localdomain netdata[887]: Found 0 legacy dbengines, setting multidb diskspace to 256MB
May 28 04:35:47 localhost.localdomain netdata[887]: 2023-05-28 04:35:47: netdata INFO  : MAIN : Found 0 legacy dbengines, setting multidb diskspace to 256MB
May 28 04:35:47 localhost.localdomain netdata[887]: 2023-05-28 04:35:47: netdata INFO  : MAIN : Created file '/var/lib/netdata/dbengine_multihost_size' to store the computed value
May 28 04:35:47 localhost.localdomain netdata[887]: Created file '/var/lib/netdata/dbengine_multihost_size' to store the computed value
May 28 04:35:47 localhost.localdomain perf.plugin[1172]: no charts enabled - nothing to do.
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: PROCFILE: Cannot open file '/etc/netdata/apps_groups.conf'
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: Loaded config file '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: started on pid 1157
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: set name of thread 1219 to APPS_READER
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1304
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1304 to EBPF CGROUP INT
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1307
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1307 to EBPF CACHESTAT
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1315
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1315 to EBPF SOFTIRQ
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1312
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1312 to EBPF MOUNT
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1306
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1306 to EBPF PROCESS
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1310
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1310 to EBPF SYNC
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1311
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1316
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1316 to EBPF OOMKILL
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1317
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1317 to EBPF SHM
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1314
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1314 to EBPF FD
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1311 to EBPF SWAP
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread with task id 1316 finished
May 28 04:35:48 localhost.localdomain debugfs.plugin[1178]: Zswap is disabled
May 28 04:35:48 localhost.localdomain apps.plugin[1157]: Using now_boottime_usec() for uptime (dt is 1 ms)
May 28 04:35:48 localhost.localdomain systemd-coredump[1396]: [🡕]  Process 1154 (ebpf.plugin) of user 990 dumped core.
May 28 04:35:58 localhost.localdomain cgroup-network[1452]: running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/sys/fs/cgroup/system.slice/systemd-udevd.service/udev'
May 28 04:35:58 localhost.localdomain cgroup-network[1452]: child pid 1453 exited with code 1.
May 28 04:35:58 localhost.localdomain cgroup-network[1452]: there are no double-linked host interfaces available.
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1500
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1500 to EBPF CGROUP INT
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1501
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1507
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1507 to EBPF SOFTIRQ
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1504
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1505
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1505 to EBPF MOUNT
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1506
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1506 to EBPF FD
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1502
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1502 to EBPF CACHESTAT
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1503
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1503 to EBPF SYNC
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1504 to EBPF SWAP
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1501 to EBPF PROCESS
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1508
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1508 to EBPF OOMKILL
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1509
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1509 to EBPF SHM
May 28 04:35:59 localhost.localdomain ebpf.plugin[1499]: thread with task id 1508 finished

then it loops forever between the occurences of "Process 1581 (ebpf.plugin) of user 990 dumped core."

thiagoftsm commented 1 year ago

Hello @mailinglists35 ,

The issue you are reporting appear to have relationship with this https://github.com/netdata/netdata/issues/15103. I will continue investigating the issue tomorow, because I am already in contact with user that reported the issue.

Best regards!