Open SomePersonSomeWhereInTheWorld opened 2 years ago
This might have been caused by something locking your DB for some time... did this happen just once, or are you seeing this again and again?
It happened a couple of times but I've passed responsibility of this to another sys admin. Perhaps the suggestion would be to produce a more friendly message like "possible high load check 'top' command" or something of the ilk?
Something must go terribly wrong here. In the trace you can see, that the executed query is:
UPDATE director_daemon_info SET ts_last_update = '1639563723425', ts_stopped = NULL, process_info = ...
This is a very cheap one, and Director pings systemd every 5 seconds. systemd is configured to terminate the Daemon in case it doesn't send it's heartbeat after 10 seconds. That's a VERY long time. All expensive operations (like Import, Sync, Deployment jobs) are executed as a sub-process and therefore fully asynchronous.
The main daemon does only very very light queries, but somehow they can take a long time on your system.
Something must go terribly wrong here
Is there any debugging or logging I can add? Not sure it's relavant but the system uses the old NIS authentication but not for Icinga logins. Anything else I can provide?
Describe the bug
A clear and concise description of what the bug is. icinga-director timed out causing icingacli to crash and generate a coredump over nigh,t here are the logs produced, with gdb and backtrace below:
To Reproduce
Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.
Expected behavior
A clear and concise description of what you expected to happen. Not crash and coredump
Your Environment
Include as many relevant details about the environment you experienced the problem in
icinga2 --version
):Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
System information: Platform: Fedora Linux Platform version: 35 (Server Edition) Kernel: Linux Kernel version: 5.15.7-200.fc35.x86_64 Architecture: x86_64
Build information: Compiler: GNU 11.2.1 Build host: unknown OpenSSL version: OpenSSL 1.1.1l FIPS 24 Aug 2021
Application information:
General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2
Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var
Internal paths: Package data directory: /usr/share/icing2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid
icinga2 daemon -C
):* If you run multiple Icinga 2 instances, thezones.conf
file (oricinga2 object list --type ``` Endpoint
andicinga2 object list --type Zone
) from all affected nodes.Additional context
gdb with bt full