Closed tickez closed 2 years ago
Hello @tickez ,
Thank you for your report, I will investigate this tomorrow :handshake: .
Best regards!
Hello @tickez ,
I built an environment with Vagrant OpenSuse 15.4 image. This image has the following kernel:
localhost:/usr/libexec/netdata # uname -a
Linux localhost 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux
Considering the logs you gave us, I suppose you installed netdata with following command (or using wget):
curl https://my-netdata.io/kickstart.sh > /tmp/netdata-kickstart.sh && sh /tmp/netdata-kickstart.sh
firstly can you confirm my conclusion? Now, after to run this command I started netdata
with default configuration during 30 minutes and I did not observe any issues with ebpf.plugin
on my environment. After this I decided to compile direct on my VM netdata, and for this I ran the following steps:
$ git clone https://github.com/thiagoftsm/netdata.git
$ cd netdata
$ sudo su
# ./packaging/installer/install-required-packages.sh --yes
# zypper install gcc-c++
# ./netdata-installer.sh --dont-wait --dont-start-it 2> err >out
Before we proceed (@Ferroin it looks like we are missing a packages to install netdata on OpenSuse 15.4`).
After to run these commands netdata was installed with success, and I ran it another 30 minutes without any issue. I stopped netdata
and after this I enabled all ebpf.plugin
threads, after to start I again did not have errors.
Of course I believe in you, and the fact that I cannot recreate the issue force me to require you run few commands on your environment to address the issue properly, so when you have a time, please, run the following commands and give us the output:
# coredumpctl list
# coredumpctl dump --output compare PID
# gdb /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin compare
# bt
Where PID
was the PID that crashed.
I am needing the backtrace (bt
) to understand what is happening in your environment.
Best regards!
Hi @thiagoftsm
Thanks for looking into this. I used the wget kickstarter with this exact command line
wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --no-updates --disable-telemetry
The machine runs virtualized on kvm and I was able to reproduce the exact same behavior on another openSUSE 15.4 running on VMWare Fusion.
As you can see from the below, I had netdata installed initially through compilation and then re-installed using the static binaries. The plugin dumps core in both cases but with a slightly different reason but both seem to be related to a malloc()
.
❯ coredumpctl list
TIME PID UID GID SIG COREFILE EXE SIZE
Mon 2022-07-11 10:46:11 CEST 5553 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:46:34 CEST 7403 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:47:17 CEST 9684 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:47:47 CEST 9840 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:48:16 CEST 9879 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:48:47 CEST 10026 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:49:16 CEST 10061 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:49:46 CEST 10231 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:50:16 CEST 10269 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:50:46 CEST 10426 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Mon 2022-07-11 10:51:16 CEST 10482 491 483 SIGABRT missing /usr/libexec/netdata/plugins.d/ebpf.plugin n/a
Fri 2022-07-15 10:54:51 CEST 10609 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 582.5K
Fri 2022-07-15 10:55:04 CEST 34775 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 293.7K
Fri 2022-07-15 11:00:31 CEST 7070 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 305.7K
Fri 2022-07-15 17:24:08 CEST 2153 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 8.8M
Sat 2022-07-16 19:21:02 CEST 2243 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 11.6M
Sat 2022-07-16 19:21:21 CEST 2669 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 11.8M
Sat 2022-07-16 19:21:38 CEST 2705 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 12.0M
Sat 2022-07-16 19:23:49 CEST 2738 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 11.8M
Sat 2022-07-16 19:24:02 CEST 3087 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 11.6M
Sat 2022-07-16 19:37:02 CEST 28288 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 13.3M
Sat 2022-07-16 19:41:59 CEST 32795 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 11.3M
Sat 2022-07-16 19:42:17 CEST 493 491 483 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 9.5M
Sat 2022-07-16 19:46:36 CEST 1240 0 0 SIGABRT present /usr/libexec/netdata/plugins.d/ebpf.plugin 7.0M
Sat 2022-07-16 20:05:15 CEST 3772 491 483 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 3.2M
Sat 2022-07-16 20:12:42 CEST 4868 0 0 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1.8M
Sun 2022-07-17 15:21:29 CEST 11672 491 483 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1.8M
Sun 2022-07-17 15:21:43 CEST 11939 491 483 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 4.6M
Sun 2022-07-17 15:21:59 CEST 11969 491 483 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1.8M
Sun 2022-07-17 15:22:16 CEST 12059 491 483 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 3.2M
Sun 2022-07-17 15:22:36 CEST 12096 491 483 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1.8M
Sun 2022-07-17 15:22:54 CEST 12150 491 483 SIGSEGV present /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1.8M
I provide only the most recent core dump
❯ coredumpctl dump --output compare 12150
PID: 12150 (ebpf.plugin)
UID: 491 (netdata)
GID: 483 (netdata)
Signal: 11 (SEGV)
Timestamp: Sun 2022-07-17 15:22:53 CEST (3h 22min ago)
Command Line: /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1
Executable: /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin
Control Group: /system.slice/netdata.service
Unit: netdata.service
Slice: system.slice
Boot ID: 235a54d7c42647119a62fd358faef149
Machine ID: cc07d9b2683f40c2f4b40b4e00000c9c
Hostname: www
Storage: /var/lib/systemd/coredump/core.ebpf\x2eplugin.491.235a54d7c42647119a62fd358faef149.12150.1658064173000000.zst (present)
Disk Size: 1.8M
Message: Process 12150 (ebpf.plugin) of user 491 dumped core.
Found module /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin without build-id.
Found module linux-vdso.so.1 with build-id: 1d8950f257479087bcdda44a121002411cba55af
Stack trace of thread 12159:
#0 0x000000000071de5d n/a (/opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin + 0x31de5d)
#1 0x00007f300c19a9d0 n/a (n/a + 0x0)
From the debugger
❯ gdb /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin compare
GNU gdb (GDB; SUSE Linux Enterprise 15) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin...
[New LWP 12159]
[New LWP 12163]
[New LWP 12150]
[New LWP 12151]
[New LWP 12154]
[New LWP 12155]
[New LWP 12157]
[New LWP 12160]
[New LWP 12161]
[New LWP 12162]
[New LWP 12164]
[New LWP 12165]
[New LWP 12166]
[New LWP 12167]
[New LWP 12168]
[New LWP 12169]
[New LWP 12152]
[New LWP 12170]
Core was generated by `/opt/netdata/usr/libexec/netdata/plugins.d/ebpf.plugin 1'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 get_meta (p=p@entry=0x7f300c19a9d0 "") at src/malloc/mallocng/meta.h:141
141 src/malloc/mallocng/meta.h: Datei oder Verzeichnis nicht gefunden.
[Current thread is 1 (LWP 12159)]
(gdb) bt
#0 get_meta (p=p@entry=0x7f300c19a9d0 "") at src/malloc/mallocng/meta.h:141
#1 0x000000000071e1d5 in __libc_free (p=0x7f300c19a9d0) at src/malloc/mallocng/free.c:105
#2 0x0000000000441b65 in thread_cleanup ()
#3 0x0000000000441cda in thread_start ()
#4 0x000000000072bc8f in start (p=0x7f300bfd1ac8) at src/thread/pthread_create.c:203
#5 0x000000000072d876 in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC
(gdb)
Let me know if you need anything more, I can create a clone from the VMWare VM in case required.
Hello @tickez ,
Thank you for your additional information. After you publish these message, I tried to recreate the issue with Ubuntu 22.04
and OpenSuse
without success, but I could create the same issues with Arch Linux
. I already started working with this issue, and I expect to have the final solution in next hours.
Best regards!
this is occuring on OL9.2
$ cat /var/log/netdata/error.log
2023-05-28 04:35:47: netdata INFO : MAIN : IEEE754: system is using IEEE754 DOUBLE PRECISION values
2023-05-28 04:35:47: netdata INFO : MAIN : TIMEZONE: using the link of /etc/localtime: 'Europe/Bucharest'
2023-05-28 04:35:47: netdata INFO : MAIN : TIMEZONE: fixed as 'Europe/Bucharest'
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: next: initialize ML
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 18 ms, initialize ML - next: initialize signals
2023-05-28 04:35:47: netdata INFO : MAIN : SIGNAL: Not enabling reaper
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, initialize signals - next: initialize static threads
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, initialize static threads - next: initialize web server
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 48 ms, initialize web server - next: initialize httpd server
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, initialize httpd server - next: set resource limits
2023-05-28 04:35:47: netdata INFO : MAIN : resources control: allowed file descriptors: soft = 1024, max = 524288
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, set resource limits - next: become daemon
2023-05-28 04:35:47: netdata INFO : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 0
2023-05-28 04:35:47: netdata INFO : MAIN : Adjusted netdata scheduling policy to batch (3), with priority 0.
2023-05-28 04:35:47: netdata INFO : MAIN : Running with process scheduling policy 'batch', nice level 19
2023-05-28 04:35:47: netdata INFO : MAIN : netdata started on pid 887.
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 3 ms, become daemon - next: initialize threads after fork
2023-05-28 04:35:47: netdata INFO : MAIN : Set threads stack size to 8388608 bytes
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, initialize threads after fork - next: initialize registry
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, initialize registry - next: fork the spawn server
2023-05-28 04:35:47: netdata INFO : MAIN : Initializing spawn client.
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 1 ms, fork the spawn server - next: collecting system info
2023-05-28 04:35:47: netdata INFO : MAIN : Executing /usr/libexec/netdata/plugins.d/system-info.sh
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_CONTAINER_OS_NAME=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_CONTAINER_OS_ID_LIKE=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_CONTAINER_OS_VERSION_ID=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_CONTAINER_OS_DETECTION=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_CONTAINER_IS_OFFICIAL_IMAGE=false
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_HOST_OS_NAME=Oracle Linux Server
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_HOST_OS_ID=ol
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_HOST_OS_ID_LIKE=fedora
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION=9.2
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_HOST_OS_VERSION_ID=9.2
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_HOST_OS_DETECTION=/etc/os-release
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_HOST_IS_K8S_NODE=false
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_NAME=Linux
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_KERNEL_VERSION=5.15.0-101.103.2.1.el9uek.x86_64
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_ARCHITECTURE=x86_64
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_VIRTUALIZATION=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_VIRT_DETECTION=systemd-detect-virt
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER=none
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_CONTAINER_DETECTION=systemd-detect-virt
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_LOGICAL_CPU_COUNT=8
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_VENDOR=GenuineIntel
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_MODEL=Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_FREQ=3500000000
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_CPU_DETECTION=lscpu
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_RAM=16698769408
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_RAM_DETECTION=procfs
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_TOTAL_DISK_SIZE=250327785472
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_SYSTEM_DISK_DETECTION=sysfs
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_INSTANCE_CLOUD_TYPE=unknown
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_INSTANCE_CLOUD_INSTANCE_TYPE=unknown
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA_INSTANCE_CLOUD_INSTANCE_REGION=unknown
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 141 ms, collecting system info - next: initialize RRD structures
2023-05-28 04:35:47: netdata INFO : MAIN : SQLite database /var/cache/netdata/netdata-meta.db initialization
2023-05-28 04:35:47: netdata INFO : MAIN : Database version is 0, current version is 7. Running migration for metadata ...
2023-05-28 04:35:47: netdata INFO : MAIN : Running database migration v0 to v1
2023-05-28 04:35:47: netdata INFO : MAIN : Running "v1 to v2" database migration
2023-05-28 04:35:47: netdata INFO : MAIN : Running "v2 to v3" database migration
2023-05-28 04:35:47: netdata INFO : MAIN : Running database migration v3 to v4
2023-05-28 04:35:47: netdata INFO : MAIN : Running "v4 to v5" database migration
2023-05-28 04:35:47: netdata INFO : MAIN : Running "v5 to v6" database migration
2023-05-28 04:35:47: netdata INFO : MAIN : Running "v6 to v7" database migration
2023-05-28 04:35:47: netdata INFO : MAIN : SQLite database initialization completed
2023-05-28 04:35:47: netdata INFO : MAIN : SQLite database /var/cache/netdata/context-meta.db initialization
2023-05-28 04:35:47: netdata INFO : MAIN : Database version is 0, current version is 1. Running migration for context ...
2023-05-28 04:35:47: netdata INFO : MAIN : Running database migration v0 to v1
2023-05-28 04:35:47: netdata INFO : MAIN : Cannot open the file /var/lib/netdata/health.silencers.json, so Netdata will work with the default health configuration.
2023-05-28 04:35:47: netdata INFO : MAIN : CONFIG: cannot load user config '/etc/netdata/stream.conf'. Will try stock config.
2023-05-28 04:35:47: netdata INFO : MAIN : DBENGINE: Initializing ...
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : thread created with task id 1076
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : set name of thread 1076 to DBENGINE_INIT
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : thread created with task id 1077
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : set name of thread 1077 to DBENGINE_INIT
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : thread created with task id 1078
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : set name of thread 1078 to DBENGINE_INIT
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: found 0 files in path /var/cache/netdata/dbengine
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: data files not found, creating in path "/var/cache/netdata/dbengine".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: found 0 files in path /var/cache/netdata/dbengine-tier2
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: data files not found, creating in path "/var/cache/netdata/dbengine-tier2".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: creating new data and journal files in path /var/cache/netdata/dbengine-tier2
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: creating new data and journal files in path /var/cache/netdata/dbengine
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: found 0 files in path /var/cache/netdata/dbengine-tier1
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: data files not found, creating in path "/var/cache/netdata/dbengine-tier1".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: creating new data and journal files in path /var/cache/netdata/dbengine-tier1
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: created data file "/var/cache/netdata/dbengine-tier2/datafile-1-0000000001.ndf".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: created journal file "/var/cache/netdata/dbengine-tier2/journalfile-1-0000000001.njf".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: populating retention to MRG from 1 journal files of tier 2, using 1 threads...
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : thread with task id 1078 finished
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: created data file "/var/cache/netdata/dbengine/datafile-1-0000000001.ndf".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: created data file "/var/cache/netdata/dbengine-tier1/datafile-1-0000000001.ndf".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: created journal file "/var/cache/netdata/dbengine/journalfile-1-0000000001.njf".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: populating retention to MRG from 1 journal files of tier 0, using 1 threads...
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : thread with task id 1076 finished
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: created journal file "/var/cache/netdata/dbengine-tier1/journalfile-1-0000000001.njf".
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : DBENGINE: populating retention to MRG from 1 journal files of tier 1, using 1 threads...
2023-05-28 04:35:47: netdata INFO : DBENGINE_INIT : thread with task id 1077 finished
2023-05-28 04:35:47: netdata INFO : MAIN : DBENGINE: tier 0 is ready for data collection and queries
2023-05-28 04:35:47: netdata INFO : MAIN : DBENGINE: tier 1 is ready for data collection and queries
2023-05-28 04:35:47: netdata INFO : MAIN : DBENGINE: tier 2 is ready for data collection and queries
2023-05-28 04:35:47: netdata INFO : METASYNC : Starting metadata sync thread with 1024 entries command queue
2023-05-28 04:35:47: netdata INFO : MAIN : SQLite metadata sync initialization complete
2023-05-28 04:35:47: netdata INFO : MAIN : Host 'localhost.localdomain' (at registry as 'localhost.localdomain') with guid 'f8889f32-fcf7-11ed-bbba-51084a8ba1a2' initialized, os 'linux', timezone 'Europe/Bucharest', tags '', program_name 'netdata', program_version 'v1.39.0-69-nightly', update every 1, memory mode dbengine, history entries 0, streaming disabled (to '' with api key ''), health enabled, cache_dir '/var/cache/netdata', alarms default handler '', alarms default recipient ''
2023-05-28 04:35:47: netdata INFO : MAIN : Creating archived hosts
2023-05-28 04:35:47: netdata INFO : MAIN : Created 0 archived hosts
2023-05-28 04:35:47: netdata INFO : MAIN : ACLK sync initialization completed
2023-05-28 04:35:47: netdata INFO : ACLKSYNC : Starting ACLK synchronization thread
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 62 ms, initialize RRD structures - next: check for incomplete shutdown
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, check for incomplete shutdown - next: collect claiming info
2023-05-28 04:35:47: netdata INFO : MAIN : Unable to load '/var/lib/netdata/cloud.d/claimed_id', setting state to AGENT_UNCLAIMED
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 0 ms, collect claiming info - next: collect host labels
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 5 ms, collect host labels - next: start the static threads
2023-05-28 04:35:47: netdata INFO : P[timex] : thread created with task id 1133
2023-05-28 04:35:47: netdata INFO : P[timex] : set name of thread 1133 to P[timex]
2023-05-28 04:35:47: netdata INFO : HEALTH : thread created with task id 1135
2023-05-28 04:35:47: netdata INFO : P[idlejitter] : thread created with task id 1134
2023-05-28 04:35:47: netdata INFO : STATS_SQLITE3 : thread created with task id 1138
2023-05-28 04:35:47: netdata INFO : STATS_SQLITE3 : set name of thread 1138 to STATS_SQLITE3
2023-05-28 04:35:47: netdata INFO : HEALTH : set name of thread 1135 to HEALTH
2023-05-28 04:35:47: netdata INFO : PLUGINSD : thread created with task id 1139
2023-05-28 04:35:47: netdata INFO : PLUGINSD : set name of thread 1139 to PLUGINSD
2023-05-28 04:35:47: netdata INFO : STATS_GLOBAL : thread created with task id 1136
2023-05-28 04:35:47: netdata INFO : STATSD_FLUSH : thread created with task id 1141
2023-05-28 04:35:47: netdata INFO : STATSD_FLUSH : set name of thread 1141 to STATSD_FLUSH
2023-05-28 04:35:47: netdata INFO : STATS_WORKERS : thread created with task id 1137
2023-05-28 04:35:47: netdata INFO : EXPORTING : thread created with task id 1142
2023-05-28 04:35:47: netdata INFO : P[idlejitter] : set name of thread 1134 to P[idlejitter]
2023-05-28 04:35:47: netdata INFO : SERVICE : thread created with task id 1140
2023-05-28 04:35:47: netdata INFO : STATS_WORKERS : set name of thread 1137 to STATS_WORKERS
2023-05-28 04:35:47: netdata INFO : STATS_GLOBAL : set name of thread 1136 to STATS_GLOBAL
2023-05-28 04:35:47: netdata INFO : EXPORTING : set name of thread 1142 to EXPORTING
2023-05-28 04:35:47: netdata INFO : WEB[1] : thread created with task id 1143
2023-05-28 04:35:47: netdata INFO : WEB[1] : set name of thread 1143 to WEB[1]
2023-05-28 04:35:47: netdata INFO : PD[apps] : thread created with task id 1144
2023-05-28 04:35:47: netdata INFO : EXPORTING : CONFIG: cannot load user exporting config '/etc/netdata/exporting.conf'. Will try the stock version.
2023-05-28 04:35:47: netdata INFO : ACLK_MAIN : thread created with task id 1145
2023-05-28 04:35:47: netdata INFO : ACLK_MAIN : set name of thread 1145 to ACLK_MAIN
2023-05-28 04:35:47: netdata INFO : PD[go.d] : thread created with task id 1146
2023-05-28 04:35:47: netdata INFO : ACLK_MAIN : Waiting for Cloud to be enabled
2023-05-28 04:35:47: netdata INFO : SERVICE : set name of thread 1140 to SERVICE
2023-05-28 04:35:47: netdata INFO : PD[go.d] : set name of thread 1146 to PD[go.d]
2023-05-28 04:35:47: netdata INFO : PD[apps] : set name of thread 1144 to PD[apps]
2023-05-28 04:35:47: netdata INFO : RRDCONTEXT : thread created with task id 1147
2023-05-28 04:35:47: netdata INFO : RRDCONTEXT : set name of thread 1147 to RRDCONTEXT
2023-05-28 04:35:47: netdata INFO : PD[ebpf] : thread created with task id 1148
2023-05-28 04:35:47: netdata INFO : PD[ebpf] : set name of thread 1148 to PD[ebpf]
2023-05-28 04:35:47: netdata INFO : REPLAY[1] : thread created with task id 1149
2023-05-28 04:35:47: netdata INFO : REPLAY[1] : set name of thread 1149 to REPLAY[1]
2023-05-28 04:35:47: netdata INFO : PD[python.d] : thread created with task id 1151
2023-05-28 04:35:47: netdata INFO : P[tc] : thread created with task id 1150
2023-05-28 04:35:47: netdata INFO : P[tc] : set name of thread 1150 to P[tc]
2023-05-28 04:35:47: netdata INFO : PD[python.d] : set name of thread 1151 to PD[python.d]
2023-05-28 04:35:47: netdata INFO : P[diskspace] : thread created with task id 1153
2023-05-28 04:35:47: netdata INFO : P[diskspace] : set name of thread 1153 to P[diskspace]
2023-05-28 04:35:47: netdata INFO : PD[charts.d] : thread created with task id 1156
2023-05-28 04:35:47: netdata INFO : P[cgroups] : thread created with task id 1158
2023-05-28 04:35:47: netdata INFO : PD[charts.d] : set name of thread 1156 to PD[charts.d]
2023-05-28 04:35:47: netdata INFO : P[cgroups] : set name of thread 1158 to P[cgroups]
2023-05-28 04:35:47: netdata INFO : PD[go.d] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/go.d.plugin' running on pid 1152
2023-05-28 04:35:47: netdata INFO : P[proc] : thread created with task id 1155
2023-05-28 04:35:47: netdata INFO : PREDICT : thread created with task id 1160
2023-05-28 04:35:47: netdata INFO : PREDICT : set name of thread 1160 to PREDICT
2023-05-28 04:35:47: netdata INFO : P[proc] : set name of thread 1155 to P[proc]
2023-05-28 04:35:47: netdata INFO : WEB[1] : To use encryption it is necessary to set "ssl certificate" and "ssl key" in [web] !
2023-05-28 04:35:47: netdata INFO : PD[apps] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/apps.plugin' running on pid 1157
2023-05-28 04:35:47: netdata INFO : PD[perf] : thread created with task id 1161
2023-05-28 04:35:47: netdata INFO : PD[perf] : set name of thread 1161 to PD[perf]
2023-05-28 04:35:47: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1154
2023-05-28 04:35:47: netdata INFO : WEB[1] : starting worker 2
2023-05-28 04:35:47: netdata INFO : PD[python.d] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 1159
2023-05-28 04:35:47: netdata INFO : WEB[1] : starting worker 3
2023-05-28 04:35:47: netdata INFO : WEB[2] : thread created with task id 1162
2023-05-28 04:35:47: netdata INFO : WEB[2] : set name of thread 1162 to WEB[2]
2023-05-28 04:35:47: netdata INFO : WEB[1] : starting worker 4
2023-05-28 04:35:47: netdata INFO : WEB[3] : thread created with task id 1163
2023-05-28 04:35:47: netdata INFO : WEB[3] : set name of thread 1163 to WEB[3]
2023-05-28 04:35:47: netdata INFO : WEB[1] : starting worker 5
2023-05-28 04:35:47: netdata INFO : WEB[4] : thread created with task id 1164
2023-05-28 04:35:47: netdata INFO : WEB[1] : starting worker 6
2023-05-28 04:35:47: netdata INFO : WEB[4] : set name of thread 1164 to WEB[4]
2023-05-28 04:35:47: netdata INFO : WEB[4] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO : PD[debugfs] : thread created with task id 1168
2023-05-28 04:35:47: netdata INFO : PD[debugfs] : set name of thread 1168 to PD[debugfs]
2023-05-28 04:35:47: netdata INFO : TRAIN[0] : thread created with task id 1166
2023-05-28 04:35:47: netdata INFO : TRAIN[0] : set name of thread 1166 to TRAIN[0]
2023-05-28 04:35:47: netdata INFO : WEB[6] : thread created with task id 1170
2023-05-28 04:35:47: netdata INFO : WEB[6] : set name of thread 1170 to WEB[6]
2023-05-28 04:35:47: netdata INFO : PD[ioping] : thread created with task id 1171
2023-05-28 04:35:47: netdata INFO : PD[ioping] : set name of thread 1171 to PD[ioping]
2023-05-28 04:35:47: netdata INFO : WEB[4] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO : EXPORTING : No connector instances to activate
2023-05-28 04:35:47: netdata INFO : EXPORTING : EXPORTING: no exporting connectors configured
2023-05-28 04:35:47: netdata INFO : EXPORTING : cleaning up...
2023-05-28 04:35:47: netdata INFO : EXPORTING : thread with task id 1142 finished
2023-05-28 04:35:47: netdata INFO : TRAIN[3] : thread created with task id 1174
2023-05-28 04:35:47: netdata INFO : TRAIN[3] : set name of thread 1174 to TRAIN[3]
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 1 ms, start the static threads - next: initialize commands API
2023-05-28 04:35:47: netdata INFO : MAIN : Initializing command server.
2023-05-28 04:35:47: netdata INFO : WEB[6] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO : WEB[6] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO : WEB[1] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO : WEB[1] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO : WEB[3] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO : WEB[3] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO : P[diskspace slo : thread created with task id 1173
2023-05-28 04:35:47: netdata INFO : P[diskspace slo : set name of thread 1173 to P[diskspace slo
2023-05-28 04:35:47: netdata INFO : PD[debugfs] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/debugfs.plugin' running on pid 1178
2023-05-28 04:35:47: netdata INFO : P[proc netdev] : thread created with task id 1176
2023-05-28 04:35:47: netdata INFO : P[proc netdev] : set name of thread 1176 to P[proc netdev]
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: in 1 ms, initialize commands API - next: ready
2023-05-28 04:35:47: netdata INFO : MAIN : NETDATA STARTUP: completed in 296 ms. Enjoy real-time performance monitoring!
2023-05-28 04:35:47: netdata INFO : MAIN : /usr/libexec/netdata/plugins.d/anonymous-statistics.sh 'START' '-' '-'
2023-05-28 04:35:47: netdata INFO : HEALTH : HEALTH [localhost.localdomain]: Table health_log_f8889f32_fcf7_11ed_bbba_51084a8ba1a2, contains 0 entries.
2023-05-28 04:35:47: netdata INFO : PD[ioping] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ioping.plugin' running on pid 1177
2023-05-28 04:35:47: netdata INFO : WEB[2] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO : WEB[2] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO : TRAIN[2] : thread created with task id 1169
2023-05-28 04:35:47: netdata INFO : TRAIN[2] : set name of thread 1169 to TRAIN[2]
2023-05-28 04:35:47: netdata INFO : WEB[5] : thread created with task id 1165
2023-05-28 04:35:47: netdata INFO : WEB[5] : set name of thread 1165 to WEB[5]
2023-05-28 04:35:47: netdata INFO : WEB[5] : POLLFD: LISTENER: listening on 'tcp:0.0.0.0:19999'
2023-05-28 04:35:47: netdata INFO : WEB[5] : POLLFD: LISTENER: listening on 'tcp:[::]:19999'
2023-05-28 04:35:47: netdata INFO : TRAIN[1] : thread created with task id 1167
2023-05-28 04:35:47: netdata INFO : TRAIN[1] : set name of thread 1167 to TRAIN[1]
2023-05-28 04:35:47: netdata INFO : PD[perf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/perf.plugin' running on pid 1172
2023-05-28 04:35:47: netdata INFO : PD[charts.d] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/charts.d.plugin' running on pid 1180
2023-05-28 04:35:47: netdata INFO : STATSD_IN[1] : thread created with task id 1187
2023-05-28 04:35:47: netdata INFO : STATSD_IN[1] : set name of thread 1187 to STATSD_IN[1]
2023-05-28 04:35:47: netdata INFO : STATSD_IN[1] : POLLFD: LISTENER: listening on 'udp:[::1]:8125'
2023-05-28 04:35:47: netdata INFO : STATSD_IN[1] : POLLFD: LISTENER: listening on 'udp:127.0.0.1:8125'
2023-05-28 04:35:47: netdata INFO : STATSD_IN[1] : POLLFD: LISTENER: listening on 'tcp:[::1]:8125'
2023-05-28 04:35:47: netdata INFO : STATSD_IN[1] : POLLFD: LISTENER: listening on 'tcp:127.0.0.1:8125'
2023-05-28 04:35:47: netdata INFO : PD[perf] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:47: netdata INFO : PD[perf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/perf.plugin' (pid 1172) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:47: netdata ERROR : PD[perf] : child pid 1172 exited with code 1.
2023-05-28 04:35:47: netdata ERROR : PD[perf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/perf.plugin' (pid 1172) exited with error code 1 and haven't collected any data. Disabling it.
2023-05-28 04:35:47: netdata INFO : PD[perf] : thread with task id 1161 finished
2023-05-28 04:35:47: netdata INFO : PD[ioping] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:47: netdata INFO : PD[ioping] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ioping.plugin' (pid 1177) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:47: netdata ERROR : PD[ioping] : child pid 1177 exited with code 1.
2023-05-28 04:35:47: netdata ERROR : PD[ioping] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ioping.plugin' (pid 1177) exited with error code 1 and haven't collected any data. Disabling it.
2023-05-28 04:35:47: netdata INFO : PD[ioping] : thread with task id 1171 finished
2023-05-28 04:35:47: netdata ERROR : MAIN : child pid 1182 exited with code 6.
2023-05-28 04:35:47: netdata ERROR : MAIN : Execution of anonymous statistics script returned 6.
2023-05-28 04:35:47: netdata ERROR : MAIN : Execution of anonymous statistics script returned http code 000.
2023-05-28 04:35:47: netdata INFO : ANALYTICS : thread created with task id 1275
2023-05-28 04:35:47: netdata INFO : ANALYTICS : set name of thread 1275 to ANALYTICS
2023-05-28 04:35:47: netdata INFO : PD[charts.d] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:47: netdata INFO : PD[charts.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 1180) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:47: netdata INFO : PD[charts.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 1180) does not generate useful output but it reports success (exits with 0). Will not start it again - it is now disabled..
2023-05-28 04:35:48: netdata INFO : P[proc] : Using now_boottime_usec() for uptime (dt is 3 ms)
2023-05-28 04:35:48: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:35:48: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1154) disconnected after 13 successful data collections (ENDs).
2023-05-28 04:35:48: netdata ERROR : PD[ebpf] : child pid 1154 core dumped by signal 11.
2023-05-28 04:35:48: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1154) exited with error code -2, but has given useful output in the past (13 times). Waiting a bit before starting it again.
2023-05-28 04:35:49: netdata INFO : PD[python.d] : PLUGINSD: plugin called DISABLE. Disabling it.
2023-05-28 04:35:49: netdata INFO : PD[python.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 1159) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:35:49: netdata INFO : PD[python.d] : child pid 1159 killed by signal 15.
2023-05-28 04:35:49: netdata INFO : PD[python.d] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/python.d.plugin' (pid 1159) does not generate useful output but it reports success (exits with 0). Will not start it again - it is now disabled..
2023-05-28 04:35:57: netdata INFO : PD[charts.d] : thread with task id 1156 finished
2023-05-28 04:35:58: netdata ERROR : PLUGIN[cgroups] : child pid 1452 exited with code 1.
2023-05-28 04:35:58: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1499
2023-05-28 04:35:59: netdata INFO : PD[python.d] : thread with task id 1151 finished
2023-05-28 04:36:00: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:36:00: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1499) disconnected after 14 successful data collections (ENDs).
2023-05-28 04:36:00: netdata ERROR : PD[ebpf] : child pid 1499 core dumped by signal 11.
2023-05-28 04:36:00: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1499) exited with error code -2, but has given useful output in the past (27 times). Waiting a bit before starting it again.
2023-05-28 04:36:10: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1540
2023-05-28 04:36:12: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:36:12: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1540) disconnected after 0 successful data collections (ENDs).
2023-05-28 04:36:12: netdata ERROR : PD[ebpf] : child pid 1540 core dumped by signal 11.
2023-05-28 04:36:12: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1540) exited with error code -2, but has given useful output in the past (27 times). Waiting a bit before starting it again.
2023-05-28 04:36:22: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain' connected to '/usr/libexec/netdata/plugins.d/ebpf.plugin' running on pid 1581
2023-05-28 04:36:23: netdata ERROR : PD[ebpf] : PARSER: read failed: end of file
2023-05-28 04:36:23: netdata INFO : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1581) disconnected after 12 successful data collections (ENDs).
2023-05-28 04:36:23: netdata ERROR : PD[ebpf] : child pid 1581 core dumped by signal 11.
2023-05-28 04:36:23: netdata ERROR : PD[ebpf] : PLUGINSD: 'host:localhost.localdomain', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 1581) exited with error code -2, but has given useful output in the past (39 times). Waiting a bit before starting it again.
2023-05-28 04:36:33: netdata LOG FLOOD PROTECTION too many logs (201 logs in 46 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1154 seconds.
$ journalctl -u netdata --no-pager
May 28 04:35:46 localhost systemd[1]: Starting Real time performance monitoring...
May 28 04:35:47 localhost systemd[1]: Started Real time performance monitoring.
May 28 04:35:47 localhost.localdomain netdata[887]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
May 28 04:35:47 localhost.localdomain netdata[887]: 2023-05-28 04:35:47: netdata INFO : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
May 28 04:35:47 localhost.localdomain netdata[887]: Found 0 legacy dbengines, setting multidb diskspace to 256MB
May 28 04:35:47 localhost.localdomain netdata[887]: 2023-05-28 04:35:47: netdata INFO : MAIN : Found 0 legacy dbengines, setting multidb diskspace to 256MB
May 28 04:35:47 localhost.localdomain netdata[887]: 2023-05-28 04:35:47: netdata INFO : MAIN : Created file '/var/lib/netdata/dbengine_multihost_size' to store the computed value
May 28 04:35:47 localhost.localdomain netdata[887]: Created file '/var/lib/netdata/dbengine_multihost_size' to store the computed value
May 28 04:35:47 localhost.localdomain perf.plugin[1172]: no charts enabled - nothing to do.
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: PROCFILE: Cannot open file '/etc/netdata/apps_groups.conf'
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: Loaded config file '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: started on pid 1157
May 28 04:35:47 localhost.localdomain apps.plugin[1157]: set name of thread 1219 to APPS_READER
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1304
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1304 to EBPF CGROUP INT
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1307
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1307 to EBPF CACHESTAT
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1315
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1315 to EBPF SOFTIRQ
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1312
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1312 to EBPF MOUNT
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1306
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1306 to EBPF PROCESS
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1310
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1310 to EBPF SYNC
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1311
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1316
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1316 to EBPF OOMKILL
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1317
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1317 to EBPF SHM
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread created with task id 1314
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1314 to EBPF FD
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: set name of thread 1311 to EBPF SWAP
May 28 04:35:47 localhost.localdomain ebpf.plugin[1154]: thread with task id 1316 finished
May 28 04:35:48 localhost.localdomain debugfs.plugin[1178]: Zswap is disabled
May 28 04:35:48 localhost.localdomain apps.plugin[1157]: Using now_boottime_usec() for uptime (dt is 1 ms)
May 28 04:35:48 localhost.localdomain systemd-coredump[1396]: [🡕] Process 1154 (ebpf.plugin) of user 990 dumped core.
May 28 04:35:58 localhost.localdomain cgroup-network[1452]: running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/sys/fs/cgroup/system.slice/systemd-udevd.service/udev'
May 28 04:35:58 localhost.localdomain cgroup-network[1452]: child pid 1453 exited with code 1.
May 28 04:35:58 localhost.localdomain cgroup-network[1452]: there are no double-linked host interfaces available.
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: Does not have a configuration file inside `/etc/netdata/ebpf.d.conf. It will try to load stock file.
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: Cannot read process groups configuration file '/etc/netdata/apps_groups.conf'. Will try '/usr/lib/netdata/conf.d/apps_groups.conf'
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1500
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1500 to EBPF CGROUP INT
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1501
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1507
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1507 to EBPF SOFTIRQ
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1504
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1505
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1505 to EBPF MOUNT
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1506
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1506 to EBPF FD
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1502
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1502 to EBPF CACHESTAT
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1503
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1503 to EBPF SYNC
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1504 to EBPF SWAP
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1501 to EBPF PROCESS
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1508
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1508 to EBPF OOMKILL
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: thread created with task id 1509
May 28 04:35:58 localhost.localdomain ebpf.plugin[1499]: set name of thread 1509 to EBPF SHM
May 28 04:35:59 localhost.localdomain ebpf.plugin[1499]: thread with task id 1508 finished
then it loops forever between the occurences of "Process 1581 (ebpf.plugin) of user 990 dumped core."
Hello @mailinglists35 ,
The issue you are reporting appear to have relationship with this https://github.com/netdata/netdata/issues/15103. I will continue investigating the issue tomorow, because I am already in contact with user that reported the issue.
Best regards!
Bug description
After starting the netdata agent, systemd reports a core dump of the ebpf.plugin.
Starting the plugin directly from the CLI results in the same behavior. It fails with both, the static build and the compiled version.
Expected behavior
The plugin should not produce a core dump on this system.
Steps to reproduce
Installation method
kickstart.sh
System info
Netdata build info
Additional info
No response