Open nielsk opened 5 years ago
nielsk
commented
3 years ago Great!
I just updated the FreeBSD port, @nielsk and @mat813 can you please confirm that you setup does work now?
I will try to find time next week. If it doesn't work m, the rollback can be quite cumbersome. Thanks a lot already to all who worked on this.
mat813
commented
3 years ago @bsdlme With icinga2-2.12.1, it absolutely still crashes on startup on i386 boxes.
nielsk
commented
3 years ago @bsdlme With icinga2-2.12.1, it absolutely still crashes on startup on i386 boxes.
Did you try i386 or x64? I just want to be sure before I do my test.
mat813
commented
3 years ago Well, the answer is in the comment you are responding to, i386. I never had any problems on amd64.
nielsk
commented
3 years ago Thanks. I just wanted to be sure because I have seen people using i386 and x64 interchangeably.
nielsk
commented
3 years ago @bsdlme I tried to build it today on my poudriere (with a FreeBSD 11.4-jail) and it fails. I created a bug in the FreeBSD-bugzilla.
bsdlme
commented
3 years ago @nielsk Yes, but you seem to have a local patch that can't be applied correctly.
nielsk
commented
3 years ago I could now update -- I had to upgrade to 11.4 because 11.3 is not supported anymore. @bsdlme icinga2 r2.12.1-1 still crashes (x64)
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.1-1)
Copyright (c) 2012-2020 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Unknown
Platform version: Unknown
Kernel: FreeBSD
Kernel version: 11.4-RELEASE-p3
Architecture: amd64
Build information:
Compiler: Clang 10.0.0
Build host: ic-11_4-RELEASE-HEAD-job-01
OpenSSL version: OpenSSL 1.1.1h 22 Sep 2020
Application information:
General paths:
Config directory: /usr/local/etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /var/run/icinga2
Old paths (deprecated):
Installation root: /usr/local
Sysconf directory: /usr/local/etc
Run directory (base): /var/run
Local state directory: /var
Internal paths:
Package data directory: /usr/local/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /var/run/icinga2/icinga2.pidToo bad. So we're back at the beginning.
julianbrost
commented
3 years ago Random thought I just had what might be causing this (did not investigate this further, just writing it down so I don't forget): Icinga 2.11 changed the network stack to use Boost.Asio and executes coroutines on multiple worker theads. AFAIK Boost.Asio may schedule these coroutines on arbitrary worker threads, thus if a coroutine holds a mutex while it performs a yield operation, the mutex might be unlocked on a different thread.
davehayes
commented
3 years ago Confirming this bug is still here after upgrading to 12.2 and 2021Q1 packages:
# pkg info | egrep 'boost|icinga'
boost-libs-1.72.0_3 Free portable C++ libraries (without Boost.Python)
icinga2-2.12.3 Monitoring and management system for hosts, services and networksHow can I tell icinga not to use multiple threads as a workaround?
# icinga2 -V
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)
Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Unknown
Platform version: Unknown
Kernel: FreeBSD
Kernel version: 12.2-STABLE
Architecture: amd64
Build information:
Compiler: Clang 10.0.1
Build host: pkg.dream-tech.com
OpenSSL version: LibreSSL 3.2.5
Application information:
General paths:
Config directory: /usr/local/etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /var/run/icinga2
Old paths (deprecated):
Installation root: /usr/local
Sysconf directory: /usr/local/etc
Run directory (base): /var/run
Local state directory: /var
Internal paths:
Package data directory: /usr/local/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /var/run/icinga2/icinga2.pidHow can I tell icinga not to use multiple threads as a workaround?
No threads at all won't work, the best you could try would be to reduce the size of thread pools to 1, however that's not configurable at runtime.
You'd have to patch the following line to say ... m_Threads(1) ...:
https://github.com/Icinga/icinga2/blob/2cb995e93714ab82914e8cace050749ba6359213/lib/base/io-engine.cpp#L87
And ... threads = 1 ... here:
https://github.com/Icinga/icinga2/blob/2cb995e93714ab82914e8cace050749ba6359213/lib/base/threadpool.hpp#L39
davehayes
commented
3 years ago That looks potentially dangerous. Still, I may try it in desperation.
davehayes
commented
3 years ago So I tried both of the changes suggested above and icinga2 still crashes.
Al2Klimov
commented
2 years ago @nielsk And 12.x+?
nielsk
commented
2 years ago I don't know. I can't update the server because of this issue, I think I even tried once 12.x and had the same problem but it is quite a while ago.
And I am migrating currently to checkmk.
Al2Klimov
commented
2 years ago To all FreeBSD users subscribed here:
Please could you test a recent Icinga 2 version on a recent FreeBSD version and report:
henningkessler
commented
2 years ago Hello,
I am running icinga2 version r2.13.1-1 (from packages) in a FreeBSD jail (12.2p11) without problems.
henning
Am 19.01.2022 um 11:26 schrieb Alexander Aleksandrovič Klimov @.***>:
To all FreeBSD users subscribed here:
Please could you test a recent Icinga 2 version on a recent FreeBSD version and report:
Icinga 2 version FreeBSD version Architecture
7539 https://github.com/Icinga/icinga2/issues/7539 occurs?
— Reply to this email directly, view it on GitHub https://github.com/Icinga/icinga2/issues/7539#issuecomment-1016294600, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATU2IPWYLEQWS7BIU34ZTLUW2GW5ANCNFSM4I2K3HSA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.
davehayes
commented
2 years ago icinga2-2.13.2, FreeBSD 12.2-STABLE, amd64.
I am running icinga2 at a remote site that works, but has never crashed. I am using the patch to boost libraries in this bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260143
which was thankfully provided to me by #9174. This installation is now having a swap space resource exhaustion issue, but that is probably different than the relevant issue.
I could no longer use the icinga2 site that has crashed before so it has been retired.
Describe the bug
After upgrading from icinga 2.10.5 to 2.11 on FreeBSD 11.3-p3, icinga2 daemon -C shows that the configuration is correct, but starts and immediately exits when the api-feature is enabled. It works without the api-feature.
After re-running the api setup, I got it working but it crashed when I tried to send a notification.
output from running truss icinga2 daemon -x debug before 'api setup'
crash
Your Environment
Include as many relevant details about the environment you experienced the problem in
icinga2 --version): r2.11.0-1icinga2 feature list): api checker command ido-mysql mainlog notificationicinga2 daemon -C):additional context
I opened a thread on the community discourse where I might have wrote more: https://community.icinga.com/t/problems-with-upgrading-icinga-2-10-5-to-2-11-on-freebsd/2325