Icinga / icingaweb2-module-director

The Director aims to be your new favourite Icinga config deployment tool. Director is designed for those who want to automate their configuration deployment and those who want to grant their “point & click” users easy access to the configuration.
https://icinga.com/docs/director/latest
GNU General Public License v2.0
412 stars 201 forks source link

icinga-director.service: Watchdog timeout (limit 10s)!= coredump on Fedora 35 #2451

Open SomePersonSomeWhereInTheWorld opened 2 years ago

SomePersonSomeWhereInTheWorld commented 2 years ago

Describe the bug

A clear and concise description of what the bug is. icinga-director timed out causing icingacli to crash and generate a coredump over nigh,t here are the logs produced, with gdb and backtrace below:

Dec 17 05:22:12 systemd[1]: icinga-director.service: Watchdog timeout (limit 10s)!
Dec 17 05:22:12 systemd[1]: icinga-director.service: Killing process 2772786 (icingacli) with signal SIGABRT.
Dec 17 05:22:13 systemd[1]: Started Process Core Dump (PID 2830897/UID 0).
Dec 17 05:22:13 kernel: xfs filesystem being remounted at /run/systemd/unit-root/var/tmp supports timestamps until 2038 (0x7fffffff)
Dec 17 05:22:13 kernel: xfs filesystem being remounted at /run/systemd/unit-root/var/lib/systemd/coredump supports timestamps until 2038 (0x7fffffff)
Dec 17 05:22:13 kernel: xfs filesystem being remounted at /run/systemd/unit-root/var/lib/systemd/coredump supports timestamps until 2038 (0x7fffffff)
Dec 17 05:22:13 mandelbrot kernel: xfs filesystem being remounted at /run/systemd/unit-root/var/tmp supports timestamps until 2038 (0x7fffffff)
Dec 17 05:22:15 mandelbrot systemd-coredump[2830898]: Process 2772786 (icingacli) of user 964 dumped core.
Found module linux-vdso.so.1 with build-id: f47dcc8bc9855f7803fcf50e0e28781e1db3adec
Found module libuuid.so.1 with build-id: 2dee14d2566e86111e3f0a235ad58a0f27b8f826
Found module libfribidi.so.0 with build-id: a4e245984a5c2e6d842a71adea32ec7c05cb7e40
Found module libICE.so.6 with build-id: 322afa55205247e8243671021f23846fbaeb9c21
Found module libSM.so.6 with build-id: 66be9db349cf0e0925db17296fcddae2ee5b9999
Found module libltdl.so.7 with build-id: 4d1cbc08cd7060e81505ddc83d6e8a16d5f69583
Found module libXt.so.6 with build-id: a45d0501cefd82176212d7c5844a67e5aff7c574
Found module libXext.so.6 with build-id: 08e82aa07cb7910a08d8386fb16de7896ce290f4
Found module liblqr-1.so.0 with build-id: cbdcfdb1b08eaaaaf940a9e1779b2935b6b0b8fd
Found module libraqm.so.0 with build-id: 4de263dccb5f4c2d7329a263ccd19deec7a964f9
Found module liblcms2.so.2 with build-id: d75e08b6e0796e1e43778305a2cb83f7fd12d4ab
Found module libMagickCore-6.Q16.so.7 with build-id: b4d6d95f35c6cc0bfe98fa1fa7e059475c969b92
Found module libMagickWand-6.Q16.so.7 with build-id: 3b35873a02758a9dbd9f73b61ce94b188eb65c5f
Found module imagick.so with build-id: b819fa385daf47dbad3f748626c643adcf68f286
Found module xmlreader.so with build-id: 210da907640330a7f650f61b3e323358613cc9b4
Found module pdo_sqlite.so with build-id: a6813f493e583c128cb960b9179f252c75211401
Found module pdo_pgsql.so with build-id: aa3fdfd04940caebf78431877e36623798f50b0b
Found module pdo_mysql.so with build-id: 1dd133ceafe4d37d8e06e3171db5c326ce8bfa21
Found module mysqli.so with build-id: f44758ebed62339220b151c65e8b8a1b096f2b6a
Found module libmcrypt.so.4 with build-id: c93ed8ea696d31ced4485203f7fcb92ac0778851
Found module mcrypt.so with build-id: 6773c2689a1165008463c215a6bc3999bcb5ef66
Found module libexslt.so.0 with build-id: 6cf4e3c6a7b2002deecff9bb8248f837ddf249f4
Found module libxslt.so.1 with build-id: a48bf4ec778ef1330719918cff01d22b81aea57c
Found module xsl.so with build-id: ed258b6fed46d221a1aece0ab318c03b895553cc
Found module xmlwriter.so with build-id: 0a280221e1d23c05e9f6461185b8201b50873654
Found module xml.so with build-id: c8f454849fd259eec2d427a970e355c41ac418fa
Found module tokenizer.so with build-id: db2a0caf089a66fa5f676512f0fa97e08ec0eb8a
Found module sysvshm.so with build-id: 3b526801e1de6a7e3db2bf4ebd35acdb2e40a177
Found module sysvsem.so with build-id: 9f218a5981a9fdb0422f64cc7ef3be1e59732aef
Found module sysvmsg.so with build-id: 3334d038bd010186e36a8935c800a0fb89b2658a
Found module libsqlite3.so.0 with build-id: 072b1e135c03359061ea9c8346cdb61528950924
Found module sqlite3.so with build-id: 74df076e801ade79a6ed32825f0f71adf120431d
Found module libsodium.so.23 with build-id: d45082d4c7667c8e661ca266e2f620bda3e73f86
Found module sodium.so with build-id: a1ec38b4c3c2b10cb0eab59ba44fc31790221a04
Found module sockets.so with build-id: 18b309f72daa7577d4079a6155998d8e28bd05cc
Found module soap.so with build-id: 43950d7ee93adce921df1bea52afc62f41150ffc
Found module simplexml.so with build-id: 88f26833c2269060f004d183f150ef8ef10f4031
Found module shmop.so with build-id: 49705815c76c0fea01aeeb5f0a5aaba8b191532c
Found module posix.so with build-id: 91d345fa2ee9e3a851317232cd92494ca4610530
Found module phar.so with build-id: e23502caab9f9d5937dfb06c0e9a4bc03ea01a8b
Found module libpq.so.5 with build-id: f90c40399880801687011432964c5f5b58771bcc
Found module pgsql.so with build-id: a44869ebd757c828b1e972c365641222c6ef27f8
Found module pdo.so with build-id: b13f7c35f7db24209d7ac2650fe35581938cbfbf
Found module mysqlnd.so with build-id: 24293a1ae5a02e3a528b0a6232c5f3fe2afedf2a
Found module libonig.so.5 with build-id: 40c96ccce0aa3923d8e22f7d3ac2358c6143cc85
Found module mbstring.so with build-id: 280caadeb1c2ce81ed994412a79634d42e21c6fc
Found module ldap.so with build-id: 50dfb0bf603c90df6cec6e175e7d956d3e766935
Found module libicudata.so.69 with build-id: fae0a612a5535b81eaec823e8cedb06c89719d94
Found module libicuuc.so.69 with build-id: 45119735453983eea1a3fb59b0ed34a6e61d8c5a
Found module libicui18n.so.69 with build-id: 96cea623123108e425250a0c59fa6221e96cadb2
Found module libicuio.so.69 with build-id: e795db09fe20bcf5957bc8fe5730daba52b299b4
Found module intl.so with build-id: 6fd444202385b0ca8a33091a4f7b582fd00663d6
Found module iconv.so with build-id: c7f194f8fdd28c20e453528962c45fc4c8b36bf9
Found module gettext.so with build-id: 9d26e3694a5a30840f003bfe51574a36b3bb2088
Found module libstdc++.so.6 with build-id: 64dc8524cb0109c7f3d3c0ef3d4d6ec9979db75c
Found module libbrotlienc.so.1 with build-id: a221af9d7e12d8ae3717cdcd563f5aec32531a91
Found module libpcre.so.1 with build-id: fcbdb4731e50f20a47b62341a6841dd616fe495d
Found module libvmaf.so.1 with build-id: dafbc2bd7453910662879891493d17c35c5ff34a
Found module libjxl.so.0.6 with build-id: 1ecc954ff11191990d1d41ea4d5150bfedee063b
Found module libXau.so.6 with build-id: b6a2d4859848c28bf0a4fc8744d04b1935879af6
Found module libgraphite2.so.3 with build-id: 58f949a538f65a6221e442879144b368f663f7e3
Found module libglib-2.0.so.0 with build-id: 3720108994016259cac8f61f174ea32513518152
Found module libaom.so.3 with build-id: f4466d1a8aa876c8ccca9d59f5159e48832e0d10
Found module libSvtAv1Enc.so.0 with build-id: b4f43b47f8a2dd48a358c5c6930d4e406a2969f6
Found module librav1e.so.0 with build-id: 1f68156f9f85bfb88cc11cc4dda00d34d3d112ee
Found module libdav1d.so.5 with build-id: dbf6a6322b97c79478fd4a90c6a65c953d07346c
Found module libjbig.so.2.1 with build-id: 6ede8c5263061d212ab5fafc4ef06beac11d82e7
Found module libzstd.so.1 with build-id: c594c701d8f5ba174a5be306973fa9803d65e327
Found module libxcb.so.1 with build-id: 1352e33391b3079347f320218799d6e531bd42f1
Found module libgcc_s.so.1 with build-id: 88564abce789aa42536da1247a57ff6062d61dcb
Found module libgomp.so.1 with build-id: 99a8163947150be8e2c11d0329fcf4c31e961223
Found module libharfbuzz.so.0 with build-id: 620a4b6e2914d6e1301c086def09a60e58450dc3
Found module libavif.so.12 with build-id: 01721fb17c16d34fccc34410d57331b8ffa6d88f
Found module libwebp.so.7 with build-id: 5a9d1654520899320170710ce46a258b49cb1b45
Found module libtiff.so.5 with build-id: d7b3223a4850d3a1fc4bf0349e4479e1d9bcf37e
Found module libX11.so.6 with build-id: b63e8f330442522b1d5872d1b2ccf32702767d80
Found module libXpm.so.4 with build-id: 6b547a703f665c22b1fcb3f842308d80a0240581
Found module libimagequant.so.0 with build-id: 486342b904b59f8e140b2151909f42123f897cfb
Found module libjpeg.so.62 with build-id: f50d5361714c8fe58ed898123a98aa0d6547068e
Found module libfreetype.so.6 with build-id: 6d98d390da279e5c22ebe472a552b3470554655b
Found module libfontconfig.so.1 with build-id: e50d477be88a082f6a2fd42a7db8278bb88466ca
Found module libpng16.so.16 with build-id: afcd45d29b0d579615db44a29213544328adde6f
Found module libgd.so.3 with build-id: c62dae85209a0b2a871e792c1a16485afd5fe924
Found module gd.so with build-id: 93a466faa1cd2b5b4cd42b53de8259b42c267021
Found module ftp.so with build-id: 17d7d8306a298baaa29fc3412debc7f27efac42a
Found module fileinfo.so with build-id: 9c59944573433008b0dab94bf464af92cc0920b9
Found module exif.so with build-id: a575f020fca097c11672937185c3fb198e31bb8c
Found module dom.so with build-id: 741c167c10af07493600aed78d3fb09461b7b2cd
Found module libselinux.so.1 with build-id: f805394f993c704b949315b56c344d22dfad801f
Found module libbrotlicommon.so.1 with build-id: 9e92a8ec2d9efe80ec86e04fba0549cd3202ebc4
Found module libsasl2.so.3 with build-id: 4eefde24d117c2ec7bcd12d4b583ba46571fc22a
Found module libresolv.so.2 with build-id: 057b93718c09e93c39c7945cdbeb314445b21fd6
Found module libkeyutils.so.1 with build-id: 2560a16099ad1875f7ea2195ae25b97ea168a758
Found module libkrb5support.so.0 with build-id: 03649849c4d1813c307624a711b03fb15e7124cb
Found module libunistring.so.2 w
Dec 17 05:22:16 mandelbrot systemd[1]: systemd-coredump@12-2830897-0.service: Deactivated successfully.
Dec 17 05:22:16 mandelbrot systemd[1]: icinga-director.service: Main process exited, code=dumped, status=6/ABRT
Dec 17 05:22:16 mandelbrot systemd[1]: icinga-director.service: Failed with result 'watchdog'.
Dec 17 05:22:16 mandelbrot systemd[1]: icinga-director.service: Consumed 14.673s CPU time.
Dec 17 05:22:17 mandelbrot abrt-server[2830916]: Interpreter crashed, but no packaged script detected: '$'icinga::director: running, db: connected' "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""'
Dec 17 05:22:17 mandelbrot abrt-server[2830916]: 'post-create' on '/var/spool/abrt/ccpp-2021-12-17-05:22:16.468522-2772786' exited with 1
Dec 17 05:22:17 mandelbrot abrt-server[2830916]: Deleting problem directory '/var/spool/abrt/ccpp-2021-12-17-05:22:16.468522-2772786'
Dec 17 05:22:46 systemd[1]: icinga-director.service: Scheduled restart job, restart counter is at 1.
Dec 17 05:22:46 systemd[1]: Stopped Icinga Director - Monitoring Configuration.
Dec 17 05:22:46 systemd[1]: icinga-director.service: Consumed 14.673s CPU time.
Dec 17 05:22:46 systemd[1]: Starting Icinga Director - Monitoring Configuration...
Dec 17 05:22:48 systemd[1]: Started Icinga Director - Monitoring Configuration.
Dec 17 05:22:48 icingadirector[2831000]: RuntimeException in /usr/share/icingaweb2/modules/director/library/Director/Daemon/DaemonDb.php:161 with message: DB is locked by a running daemon instance, will retry
Dec 17 05:23:03 mandelbrot systemd[1]: Reloading.
.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.

  1. No way to reproduce at this point

Expected behavior

A clear and concise description of what you expected to happen. Not crash and coredump

Your Environment

Include as many relevant details about the environment you experienced the problem in

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: Fedora Linux Platform version: 35 (Server Edition) Kernel: Linux Kernel version: 5.15.7-200.fc35.x86_64 Architecture: x86_64

Build information: Compiler: GNU 11.2.1 Build host: unknown OpenSSL version: OpenSSL 1.1.1l FIPS 24 Aug 2021

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icing2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid


* Operating System and version: Fedora 35
* Enabled features (`icinga2 feature list`): ```
Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata
Enabled features: api checker command ido-mysql mainlog notification syslog

Additional context

gdb with bt full

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `icinga::director: running, db: connected                       '.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f8c2cbae73f in __GI___poll (fds=fds@entry=0x7ffe3ae40190, nfds=nfds@entry=1, timeout=86400000)
    at ../sysdeps/unix/sysv/linux/poll.c:29
29        return SYSCALL_CANCEL (poll, fds, nfds, timeout);
Missing separate debuginfos, use: dnf debuginfo-install php-pecl-imagick-im6-3.6.0-2.fc35.remi.8.0.x86_64
(gdb) bt full
#0  0x00007f8c2cbae73f in __GI___poll (fds=fds@entry=0x7ffe3ae40190, nfds=nfds@entry=1, timeout=86400000)
    at ../sysdeps/unix/sysv/linux/poll.c:29
        sc_ret = -516
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
Icinga/icinga2#1  0x0000555b4d5956c6 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffe3ae40190) at /usr/include/bits/poll2.h:48
No locals.
Icinga/icinga2#2  php_pollfd_for (timeouttv=0x7f8c2c894878, events=25, fd=<optimized out>)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/main/php_network.h:177
        p = {fd = 5, events = 25, revents = 0}
        n = <optimized out>
        p = <optimized out>
        n = <optimized out>
Icinga/icinga2#3  php_sock_stream_wait_for_data (stream=0x7f8c2c89f0e0, sock=<optimized out>)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/main/streams/xp_socket.c:139
        retval = <optimized out>
        ptimeout = 0x7f8c2c894878
        retval = <optimized out>
        ptimeout = <optimized out>
Icinga/icinga2#4  php_sockop_read (stream=0x7f8c2c89f0e0, buf=0x7f8c2c994000 "0", count=32768)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/main/streams/xp_socket.c:163
        sock = <optimized out>
        nr_bytes = 0
        err = <optimized out>
Icinga/icinga2#5  0x0000555b4d5898f3 in _php_stream_fill_read_buffer (stream=stream@entry=0x7f8c2c89f0e0, size=size@entry=4)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/main/streams/streams.c:666
        justread = 0
Icinga/icinga2#6  0x0000555b4d589a11 in _php_stream_read (stream=stream@entry=0x7f8c2c89f0e0, buf=<optimized out>,
    buf@entry=0x7ffe3ae402d4 "0", size=size@entry=4) at /usr/src/debug/php-8.0.13-1.fc35.x86_64/main/streams/streams.c:718
        toread = <optimized out>
        didread = 0
Icinga/icinga2#7  0x00007f8c27029a2e in mysqlnd_mysqlnd_vio_network_read_pub (vio=<optimized out>, buffer=<optimized out>, count=4,
    stats=0x7f8c2c93d3e0, error_info=<optimized out>) at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/mysqlnd/mysqlnd_vio.c:87
        ret = <optimized out>
        return_value = PASS
        net_stream = 0x7f8c2c89f0e0
        to_read = 4
--Type <RET> for more, q to quit, c to continue without paging--
        p = 0x7ffe3ae402d4 "0"
Icinga/icinga2#8  0x00007f8c27024f1c in mysqlnd_read_header (pfc=0x7f8c2c872500, vio=<optimized out>, header=0x7ffe3ae40600,
    conn_stats=0x7f8c2c93d3e0, error_info=<optimized out>)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/mysqlnd/mysqlnd_wireprotocol.c:247
        buffer = "0\000\000\001"
Icinga/icinga2#9  0x00007f8c27025710 in mysqlnd_read_packet_header_and_body (packet_header=packet_header@entry=0x7ffe3ae40600,
    pfc=0x7f8c2c872500, vio=0x7f8c2c881780, stats=0x7f8c2c93d3e0, error_info=error_info@entry=0x7f8c2c91f520,
    connection_state=0x7f8c2c91f778, buf=0x7f8c2c8c0000 "S\001", buf_size=4096, packet_type=PROT_RSET_HEADER_PACKET,
    packet_type_as_text=<optimized out>) at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/mysqlnd/mysqlnd_wireprotocol.c:293
No locals.
Icinga/icinga2#10 0x00007f8c2702729a in php_mysqlnd_rset_header_read (conn=<optimized out>, _packet=0x7ffe3ae40600)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/mysqlnd/mysqlnd_wireprotocol.c:1042
        packet = 0x7ffe3ae40600
        error_info = 0x7f8c2c91f520
        pfc = <optimized out>
        vio = <optimized out>
        stats = <optimized out>
        connection_state = <optimized out>
        ret = PASS
        buf_len = <optimized out>
        buf = 0x7f8c2c8c0000 "S\001"
        p = 0x7f8c2c8c0000 "S\001"
        begin = 0x7f8c2c8c0000 "S\001"
        len = <optimized out>
Icinga/icinga2#11 0x00007f8c27030a66 in mysqlnd_query_read_result_set_header (conn=0x7f8c2c91f400, s=<optimized out>)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/mysqlnd/mysqlnd_result.c:391
        ret = FAIL
        stmt = <optimized out>
        rset_header = {header = {size = 0, packet_no = 0 '\000', m = 0x7f8c2704c568 <packet_methods+168>}, field_count = 0,
          warning_count = 0, server_status = 0, affected_rows = 0, last_insert_id = 0, info_or_local_file = {s = 0x0,
            l = 0}, error_info = {error = '\000' <repeats 512 times>, sqlstate = "\000\000\000\000\000", error_no = 0,
            error_list = {head = 0x0, tail = 0x0, count = 0, size = 0, dtor = 0x0, persistent = 0 '\000',
              traverse_ptr = 0x0}, persistent = false, m = 0x0}}
        fields_eof = {header = {size = 0, packet_no = 8 '\b', m = 0x2704c538}, field_count = 10 '\n', warning_count = 0,
          server_status = 0,
          error = '\000' <repeats 35 times>, "\060.42508200", '\000' <repeats 45 times>, "\270\231XM[U\000\000\004\000\000\000\000\000\000\000\340\360\211,\214\177\000\000\004\000\000\000\000\000\000\000\340ӓ,\214\177\000\000\304\004\344:\376\177\000\--Type <RET> for more, q to quit, c to continue without paging--
000.\232\002'\214\177\000\000\000\000\000\000\000\000\000\000\355\231XM[U\000\000\005\000\000\000\000\000\000\000\340\360\211,\214\177\000\000\005\000\000\000\000\000\000\000\340ӓ,\214\177\000\000XƟ,\214\177\000\000."...,
          sqlstate = "\000\030\300\237,\214", error_no = 4084121856}
Icinga/icinga2#12 0x00007f8c2701ffaf in mysqlnd_mysqlnd_conn_data_query_pub (query_len=338,
    query=0x7f8c2c882200 "UPDATE director_daemon_info SET ts_last_update = '1639563723425', ts_stopped = NULL, process_info = '{\\\"1893086\\\":{\\\"command\\\":\\\"\\\\/usr\\\\/bin\\\\/icingacli director daemon run information\\\",\\\"running\\\":"..., conn=0x7f8c2c91f400) at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/mysqlnd/mysqlnd_connection.c:855
        this_func = <optimized out>
        ret = <optimized out>
        this_func = <optimized out>
        ret = <optimized out>
        _s = <optimized out>
        _p_s = <optimized out>
        v = <optimized out>
        _s = <optimized out>
        _p_s = <optimized out>
        v = <optimized out>

Icinga/icinga2#13 mysqlnd_mysqlnd_conn_data_query_pub (conn=0x7f8c2c91f400,
    query=0x7f8c2c882200 "UPDATE director_daemon_info SET ts_last_update = '1639563723425', ts_stopped = NULL, process_info = '{\\\"1893086\\\":{\\\"command\\\":\\\"\\\\/usr\\\\/bin\\\\/icingacli director daemon run information\\\",\\\"running\\\":"..., query_len=338) at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/mysqlnd/mysqlnd_connection.c:846
        this_func = 24
        ret = FAIL
        _s = <optimized out>
        _p_s = <optimized out>
        v = <optimized out>
        _s = <optimized out>
        _p_s = <optimized out>
        v = <optimized out>
Icinga/icinga2#14 0x00007f8c26c0d2f2 in pdo_mysql_stmt_execute (stmt=0x7f8c2c883a00)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/pdo_mysql/mysql_statement.c:320
        S = 0x7f8c2c8cd600
        H = 0x7f8c2c89b848
Icinga/icinga2#15 0x00007f8c26ffc4ef in zim_PDOStatement_execute (execute_data=<optimized out>, return_value=<optimized out>)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/ext/pdo/pdo_stmt.c:466
        input_params = <optimized out>
        ret = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        stmt = 0x7f8c2c883a00
Icinga/icinga2#16 0x0000555b4d642ec0 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/Zend/zend_vm_execute.h:1863
        retval = <optimized out>
        call = 0x7f8c2c813960
        fbc = 0x555b4e1d32b0
        ret = <optimized out>
        call = <optimized out>
        fbc = <optimized out>
        ret = <optimized out>
        retval = <optimized out>
Icinga/icinga2#17 execute_ex (ex=0x7ffe3ae40190) at /usr/src/debug/php-8.0.13-1.fc35.x86_64/Zend/zend_vm_execute.h:54208
        vm_stack_data = {orig_opline = 0x7ffe3ae42fa0, orig_execute_data = 0x0,
          hybrid_jit_red_zone = '\000' <repeats 15 times>}
Icinga/icinga2#18 0x0000555b4d644719 in zend_execute (op_array=0x7f8c2c872100, return_value=0x0)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/Zend/zend_vm_execute.h:58523
        execute_data = 0x7f8c2c813020
        object_or_called_scope = <optimized out>
        call_info = <optimized out>
Icinga/icinga2#19 0x0000555b4d5d98b0 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/Zend/zend.c:1680
        files = {{gp_offset = 40, fp_offset = 0, overflow_arg_area = 0x7ffe3ae40b70, reg_save_area = 0x7ffe3ae40b00}}
        i = 1
        file_handle = 0x41838f98
        op_array = 0x7f8c2c872100
        ret = 746666160
Icinga/icinga2#20 0x0000555b4d5752c3 in php_execute_script (primary_file=<optimized out>)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/main/main.c:2539
        realfile = "/usr/bin/icingacli", '\000' <repeats 2942 times>...
        __orig_bailout = <optimized out>
        __bailout = {{__jmpbuf = {93850638493032, -3129226186922893179, 93850630975568, 93850630975528, 93850630976440,
              93850638493024, -3129226187900166011, -9090465020181847931}, __mask_was_saved = 0, __saved_mask = {__val = {
                0 <repeats 16 times>}}}}
        prepend_file_p = 0x0
        append_file_p = 0x0
        prepend_file = {handle = {fp = 0x0, stream = {handle = 0x0, isatty = 0, reader = 0x0, fsizer = 0x0, closer = 0x0}},
          filename = 0x5da86089f36ec100 <error: Cannot access memory at address 0x5da86089f36ec100>, opened_path = 0x0,
--Type <RET> for more, q to quit, c to continue without paging--
          type = ZEND_HANDLE_FILENAME, free_filename = false, buf = 0x7ffe3ae41c10 "thread", len = 140239918091228}
        append_file = {handle = {fp = 0x7f8c2693c213, stream = {handle = 0x7f8c2693c213, isatty = 5,
              reader = 0x555b4e235e80, fsizer = 0x7f8c26843e25 <GetPolicyInfo+373>, closer = 0x6500646165726874}},
          filename = 0x64616572687400 <error: Cannot access memory at address 0x64616572687400>, opened_path = 0x0,
          type = ZEND_HANDLE_FILENAME, free_filename = false, buf = 0x0, len = 0}
        old_cwd = <optimized out>
        use_heap = false
        retval = 0
Icinga/icinga2#21 0x0000555b4d66a326 in do_cli (argc=6, argv=0x555b4df82560)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/sapi/cli/php_cli.c:948
        __orig_bailout = 0x7ffe3ae441e0
        __bailout = {{__jmpbuf = {0, 3129637341480378501, 0, 140729886458324, 93850630976440, 93850634818464,
              -3129226186916601723, -9090464884999823227}, __mask_was_saved = 0, __saved_mask = {__val = {93850630854245,
                93850630854258, 93850630854275, 93850630854296, 93850630854316, 93850630854333, 93850630854354,
                93850630854364, 93850630854378, 93850630854400, 93850630854419, 93850630854446, 93850630854475,
                93850630854503, 0, 57344}}}}
        c = <optimized out>
        file_handle = {handle = {fp = 0x555b4e122ba0, stream = {handle = 0x555b4e122ba0, isatty = 0,
              reader = 0x555b4d646fb0 <zend_stream_stdio_reader>, fsizer = 0x555b4d646fd0 <zend_stream_stdio_fsizer>,
              closer = 0x555b4d646f80 <zend_stream_stdio_closer>}}, filename = 0x555b4df825c0 "/usr/bin/icingacli",
          opened_path = 0x0, type = ZEND_HANDLE_STREAM, free_filename = false, buf = 0x7f8c2c872000 "", len = 218}
        behavior = <optimized out>
        reflection_what = <optimized out>
        request_started = 1
        php_optarg = 0x0
        orig_optarg = 0x0
        php_optind = 2
        orig_optind = 1
        exec_direct = <optimized out>
        exec_run = <optimized out>
        exec_begin = <optimized out>
        exec_end = <optimized out>
        arg_free = <optimized out>
        arg_excp = <optimized out>
        script_file = <optimized out>
        translated_path = 0x555b4e236820 "/usr/bin/icingacli"
        interactive = 0
--Type <RET> for more, q to quit, c to continue without paging--
        param_error = <optimized out>
        hide_argv = <optimized out>
Icinga/icinga2#22 0x0000555b4d43aa2b in main (argc=6, argv=0x555b4df82560)
    at /usr/src/debug/php-8.0.13-1.fc35.x86_64/sapi/cli/php_cli.c:1336
        __orig_bailout = 0x0
        __bailout = {{__jmpbuf = {0, 3129637341480378501, 0, 140729886458324, 93850630976440, 93850634818464,
              -3129226189710008187, -9090465185240948603}, __mask_was_saved = 0, __saved_mask = {__val = {2048, 64, 1048576,
                16, 64, 11, 64, 64, 32768, 0, 365072220166, 25769803780, 0, 0, 0, 0}}}}
        c = <optimized out>
        exit_status = 0
        module_started = 1
        sapi_started = 1
        php_optarg = 0x0
        php_optind = 1
        use_extended_info = 0
        ini_path_override = 0x0
        ini_entries = 0x555b4df828a0 "html_errors=0\nregister_argc_argv=1\nimplicit_flush=1\noutput_buffering=0\nmax_execution_time=0\nmax_input_time=-1\n"
        ini_entries_len = <optimized out>
        ini_ignore = 0
        sapi_module = <optimized out>
Thomas-Gelf commented 2 years ago

This might have been caused by something locking your DB for some time... did this happen just once, or are you seeing this again and again?

SomePersonSomeWhereInTheWorld commented 2 years ago

It happened a couple of times but I've passed responsibility of this to another sys admin. Perhaps the suggestion would be to produce a more friendly message like "possible high load check 'top' command" or something of the ilk?

Thomas-Gelf commented 2 years ago

Something must go terribly wrong here. In the trace you can see, that the executed query is:

UPDATE director_daemon_info SET ts_last_update = '1639563723425', ts_stopped = NULL, process_info = ...

This is a very cheap one, and Director pings systemd every 5 seconds. systemd is configured to terminate the Daemon in case it doesn't send it's heartbeat after 10 seconds. That's a VERY long time. All expensive operations (like Import, Sync, Deployment jobs) are executed as a sub-process and therefore fully asynchronous.

The main daemon does only very very light queries, but somehow they can take a long time on your system.

SomePersonSomeWhereInTheWorld commented 2 years ago

Something must go terribly wrong here

Is there any debugging or logging I can add? Not sure it's relavant but the system uses the old NIS authentication but not for Icinga logins. Anything else I can provide?