search

netdata / netdata

Architected for speed. Automated for easy. Monitoring and troubleshooting, transformed!
https://www.netdata.cloud
GNU General Public License v3.0
72.06k stars 5.93k forks source link

[Bug]: Netdata uninstall doesn't clean up properly #16147

Closed IanTrudel closed 1 year ago

IanTrudel commented 1 year ago

Bug description

The uninstaller now works (see #16119) but the service is not stopped and disabled before removing the files causing the service still being active. Furthermore, /usr/lib/systemd/system-preset/50-netdata.preset and some kickstarter temp files are not deleted.

○ netdata.service - Real time performance monitoring
     Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; preset: enabled)
     Active: inactive (dead) since Fri 2023-10-06 11:27:28 EDT; 7min ago
   Duration: 58.203s
    Process: 18885 ExecStartPre=/bin/mkdir -p /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 18887 ExecStartPre=/bin/chown -R netdata /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 18889 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
    Process: 18891 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
    Process: 18893 ExecStart=/opt/netdata/usr/sbin/netdata -P /run/netdata/netdata.pid -D (code=exited, status=0/SUCCESS)
   Main PID: 18893 (code=exited, status=0/SUCCESS)
        CPU: 5.804s

oct 06 11:27:15 linux [19114]: thread with task id 19138 finished
oct 06 11:27:15 linux [19114]: thread with task id 19139 finished
oct 06 11:27:15 linux [19114]: thread with task id 19134 finished
oct 06 11:27:15 linux [19114]: thread with task id 19135 finished
oct 06 11:27:15 linux [19114]: thread with task id 19141 finished
oct 06 11:27:16 linux [19114]: eBPF cannot unload all threads on time, but it will go away
oct 06 11:27:17 linux [19114]: eBPF cannot unload all threads on time, but it will go away
oct 06 11:27:28 linux systemd[1]: netdata.service: Deactivated successfully.
oct 06 11:27:28 linux systemd[1]: Stopped Real time performance monitoring.
oct 06 11:27:28 linux systemd[1]: netdata.service: Consumed 5.804s CPU time.

Expected behavior

A shiny kitchen sink.

Steps to reproduce

  1. curl https://my-netdata.io/kickstart.sh > /tmp/netdata-kickstart.sh && sh /tmp/netdata-kickstart.sh --uninstall
  2. systemctl status netdata
  3. find /usr/lib/systemd -iname "*netdata*"
  4. find /tmp -iname "*kickstart*"

Installation method

kickstart.sh

System info

Arch Linux

Netdata build info

Packaging:
    Netdata Version ____________________________________________ : v1.42.4
    Installation Type __________________________________________ : kickstart-static
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ : unknown
    Configure Options __________________________________________ :  '--prefix=/opt/netdata/usr' '--sysconfdir=/opt/netdata/etc' '--localstatedir=/opt/netdata/var' '--libexecdir=/opt/netdata/usr/libexec' '--libdir=/opt/netdata/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--enable-cloud' '--without-bundled-protobuf' '--disable-dependency-tracking' '--enable-lto' 'CFLAGS=-ffunction-sections -fdata-sections -static -O2 -funroll-loops -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/usr/include/libmnl -pipe' 'LDFLAGS=-Wl,--gc-sections -Wl,-z,stack-size=8388608 -static -L/openssl-static/lib64 -L/libnetfilter-acct-static/lib -lnetfilter_acct -L/usr/lib -lmnl' 'PKG_CONFIG=pkg-config --static' 'PKG_CONFIG_PATH=/openssl-static/lib64/pkgconfig:/libnetfilter-acct-static/lib/pkgconfig:/usr/lib/pkgconfig'
Default Directories:
    User Configurations ________________________________________ : /opt/netdata/etc/netdata
    Stock Configurations _______________________________________ : /opt/netdata/usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /opt/netdata/var/cache/netdata
    Permanent Databases ________________________________________ : /opt/netdata/var/lib/netdata
    Plugins ____________________________________________________ : /opt/netdata/usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /opt/netdata/usr/share/netdata/web
    Log Files __________________________________________________ : /opt/netdata/var/log/netdata
    Lock Files _________________________________________________ : /opt/netdata/var/lib/netdata/lock
    Home _______________________________________________________ : /opt/netdata/var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 5.18.6
    Operating System ___________________________________________ : Arch Linux
    Operating System ID ________________________________________ : arch
    Operating System ID Like ___________________________________ : unknown
    Operating System Version ___________________________________ : rolling
    Operating System Version ID ________________________________ : none
    Detection __________________________________________________ : Mixed
Hardware:
    CPU Cores __________________________________________________ : 
    CPU Frequency ______________________________________________ : 3400000000
    CPU Architecture ___________________________________________ : 33622183936
    RAM Bytes __________________________________________________ : 500118700032
    Disk Capacity ______________________________________________ : x86_64
    Virtualization Technology __________________________________ : none
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : none
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : none
    Container Operating System ID ______________________________ : none
    Container Operating System ID Like _________________________ : none
    Container Operating System Version _________________________ : none
    Container Operating System Version ID ______________________ : none
    Container Operating System Detection _______________________ : none
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (lz4)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    map ________________________________________________________ : YES
    save _______________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Judy (high-performance dynamic arrays and hashtables) ______ : YES (bundled)
    dlib (robust machine learning toolkit) _____________________ : YES (bundled)
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libm (mathematical functions) ______________________________ : YES
    jemalloc ___________________________________________________ : NO
    TCMalloc ___________________________________________________ : NO
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : NO
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : NO
    nfacct (gather netfilter accounting) _______________________ : YES
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : NO
    Xen VBD Error Tracking _____________________________________ : NO
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : NO
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

Additional info

No response

ilyam8 commented 1 year ago

Since I was fixing #16119 I did install/uninstall several times - only partially confirm the OP issue.

the service is not stopped and disabled

Can't reproduce. There is stop_all_netdata() in the script.

/usr/lib/systemd/system-preset/50-netdata.preset

Confirm.

some kickstarter temp files

Confirm. There is netdata-kickstart-XXX dir with install-type file

This is my uninstaller logs (removed rm -rf /opt/netdata verbose output):

./kickstart --uninstall ```bash $ sudo ./kickstart.sh --uninstall --- Using /tmp/netdata-kickstart-epqP7itTRW as a temporary directory. --- --- Checking for existing installations of Netdata... --- [/tmp/netdata-kickstart-epqP7itTRW]# test -x /opt/netdata/usr/libexec/netdata/netdata-uninstaller.sh OK --- Found existing netdata-uninstaller. Running it.. --- [/tmp/netdata-kickstart-epqP7itTRW]# /opt/netdata/usr/libexec/netdata/netdata-uninstaller.sh --yes Fri 06 Oct 2023 07:00:22 PM EEST : INFO: netdata-uninstaller.sh: Stopping a possibly running netdata... Do you want to delete this file '/etc/logrotate.d/netdata' ? [y/n] y [/tmp/netdata-kickstart-epqP7itTRW]# rm -v /etc/logrotate.d/netdata removed '/etc/logrotate.d/netdata' OK Do you want to delete this file '/lib/systemd/system/netdata.service' ? [y/n] y [/tmp/netdata-kickstart-epqP7itTRW]# rm -v /lib/systemd/system/netdata.service removed '/lib/systemd/system/netdata.service' OK Do you want to delete this file '/lib/systemd/system/netdata-updater.service' ? [y/n] y [/tmp/netdata-kickstart-epqP7itTRW]# rm -v /lib/systemd/system/netdata-updater.service removed '/lib/systemd/system/netdata-updater.service' OK Do you want to delete this file '/lib/systemd/system/netdata-updater.timer' ? [y/n] y [/tmp/netdata-kickstart-epqP7itTRW]# rm -v /lib/systemd/system/netdata-updater.timer removed '/lib/systemd/system/netdata-updater.timer' OK Do you want to delete this file '/etc/cron.daily/netdata-updater' ? [y/n] y [/tmp/netdata-kickstart-epqP7itTRW]# rm -v /etc/cron.daily/netdata-updater removed '/etc/cron.daily/netdata-updater' OK Do you want to delete this directory '/opt/netdata' ? [y/n] y [/tmp/netdata-kickstart-epqP7itTRW]# rm -v -f -R /opt/netdata ... ... ... removed directory '/opt/netdata' OK Do you want to delete 'netdata' from following groups: ' docker nginx varnish haproxy adm nsd proxy squid ceph nobody' ? [y/n] y Fri 06 Oct 2023 07:00:49 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from docker group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata docker Removing user netdata from group docker OK Fri 06 Oct 2023 07:00:49 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from nginx group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata nginx gpasswd: group 'nginx' does not exist in /etc/group FAILED [/tmp/netdata-kickstart-epqP7itTRW]# delgroup netdata nginx /usr/sbin/delgroup: The group `nginx' does not exist. FAILED Fri 06 Oct 2023 07:00:49 PM EEST : ERROR: netdata-uninstaller.sh: Failed to delete user netdata from group nginx ! Fri 06 Oct 2023 07:00:49 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from varnish group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata varnish gpasswd: group 'varnish' does not exist in /etc/group FAILED [/tmp/netdata-kickstart-epqP7itTRW]# delgroup netdata varnish /usr/sbin/delgroup: The group `varnish' does not exist. FAILED Fri 06 Oct 2023 07:00:49 PM EEST : ERROR: netdata-uninstaller.sh: Failed to delete user netdata from group varnish ! Fri 06 Oct 2023 07:00:49 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from haproxy group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata haproxy gpasswd: group 'haproxy' does not exist in /etc/group FAILED [/tmp/netdata-kickstart-epqP7itTRW]# delgroup netdata haproxy /usr/sbin/delgroup: The group `haproxy' does not exist. FAILED Fri 06 Oct 2023 07:00:49 PM EEST : ERROR: netdata-uninstaller.sh: Failed to delete user netdata from group haproxy ! Fri 06 Oct 2023 07:00:49 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from adm group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata adm Removing user netdata from group adm OK Fri 06 Oct 2023 07:00:49 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from nsd group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata nsd gpasswd: group 'nsd' does not exist in /etc/group FAILED [/tmp/netdata-kickstart-epqP7itTRW]# delgroup netdata nsd /usr/sbin/delgroup: The group `nsd' does not exist. FAILED Fri 06 Oct 2023 07:00:50 PM EEST : ERROR: netdata-uninstaller.sh: Failed to delete user netdata from group nsd ! Fri 06 Oct 2023 07:00:50 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from proxy group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata proxy Removing user netdata from group proxy OK Fri 06 Oct 2023 07:00:50 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from squid group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata squid gpasswd: group 'squid' does not exist in /etc/group FAILED [/tmp/netdata-kickstart-epqP7itTRW]# delgroup netdata squid /usr/sbin/delgroup: The group `squid' does not exist. FAILED Fri 06 Oct 2023 07:00:50 PM EEST : ERROR: netdata-uninstaller.sh: Failed to delete user netdata from group squid ! Fri 06 Oct 2023 07:00:50 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from ceph group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata ceph gpasswd: group 'ceph' does not exist in /etc/group FAILED [/tmp/netdata-kickstart-epqP7itTRW]# delgroup netdata ceph /usr/sbin/delgroup: The group `ceph' does not exist. FAILED Fri 06 Oct 2023 07:00:50 PM EEST : ERROR: netdata-uninstaller.sh: Failed to delete user netdata from group ceph ! Fri 06 Oct 2023 07:00:50 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user from nobody group ... [/tmp/netdata-kickstart-epqP7itTRW]# gpasswd -d netdata nobody gpasswd: group 'nobody' does not exist in /etc/group FAILED [/tmp/netdata-kickstart-epqP7itTRW]# delgroup netdata nobody /usr/sbin/delgroup: The group `nobody' does not exist. FAILED Fri 06 Oct 2023 07:00:50 PM EEST : ERROR: netdata-uninstaller.sh: Failed to delete user netdata from group nobody ! Do you want to delete 'netdata' system user ? [y/n] y Fri 06 Oct 2023 07:00:50 PM EEST : INFO: netdata-uninstaller.sh: Deleting netdata user account ... [/tmp/netdata-kickstart-epqP7itTRW]# userdel -f netdata OK Do you want to delete 'netdata' system group ? [y/n] y Fri 06 Oct 2023 07:00:51 PM EEST : INFO: netdata-uninstaller.sh: Removing netdata user group ... Fri 06 Oct 2023 07:00:51 PM EEST : INFO: netdata-uninstaller.sh: Group netdata already removed in a previous step. Fri 06 Oct 2023 07:00:51 PM EEST : INFO: netdata-uninstaller.sh: Netdata files were successfully removed from your system OK Finished uninstalling the Netdata Agent.[/root]# rm -rf /tmp/netdata-kickstart-epqP7itTRW OK ```

After --uninstall

 $ systemctl status netdata
Warning: The unit file, source configuration file or drop-ins of netdata.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● netdata.service - Real time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; bad; vendor preset: enabled)
     Active: inactive (dead) since Fri 2023-10-06 19:09:13 EEST; 2min 55s ago
    Process: 31111 ExecStartPre=/bin/mkdir -p /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 31112 ExecStartPre=/bin/chown -R netdata /opt/netdata/var/cache/netdata (code=exited, status=0/SUCCESS)
    Process: 31113 ExecStartPre=/bin/mkdir -p /run/netdata (code=exited, status=0/SUCCESS)
    Process: 31114 ExecStartPre=/bin/chown -R netdata /run/netdata (code=exited, status=0/SUCCESS)
    Process: 31115 ExecStart=/opt/netdata/usr/sbin/netdata -P /run/netdata/netdata.pid -D (code=exited, status=0/SUCCESS)
   Main PID: 31115 (code=exited, status=0/SUCCESS)

$ sudo systemctl daemon-reload

$ systemctl status netdata
Unit netdata.service could not be found.
IanTrudel commented 1 year ago

@ilyam8 How do you access the uninstaller log?

ilyam8 commented 1 year ago

Just copied the script output (terminal)

ilyam8 commented 1 year ago

Thanks for reporting, @IanTrudel 👍