search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
1.99k stars 573 forks source link

Apply scheduled downtime bug in the api #8994

Open maelig opened 3 years ago

maelig commented 3 years ago

Describe the bug

I applied a downtime on the services of a host (apply ScheduledDowntime to Service). But some of these services are not seen in downtime with the API (value at 0 instead of 1).

To Reproduce

Example with this configuration for the services of a host yel-spr-prd-gce-bdd-lnx-mysql-003 :

apply ScheduledDowntime "DOWNTIME-SPR-BDD2" to Service {
  author = "jaubin"
  comment = "Scheduled downtime each day for backup 23h - 23h59"
  ranges = {
    monday = "23:00-23:59"
    tuesday = "23:00-23:59"
    wednesday = "23:00-23:59"
    thursday = "23:00-23:59"
    friday = "23:00-23:59"
    saturday = "23:00-23:59"
    sunday = "23:00-23:59"
  }
 assign where regex("yel-spr-prd-gce-bdd-lnx-mysql-003", host.name)
}

Result of the api request during donwtime hours (23h17) :

curl -k  -u 'XXXX:XXXXX'  -H 'Accept: application/json' -X GET 'https://locahost:5665/v1/objects/services' -d '{"attrs": [ "downtime_depth" ], "filter":"match(\"yel-spr-prd-gce-bdd-lnx-mysql-003\", host.name)"}' | python -m json.tool
{
    "results": [
        {
            "attrs": {
                "downtime_depth": 1.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 0.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 1.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 0.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 0.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 1.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 1.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Status",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 1.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available",
            "type": "Service"
        },
        {
            "attrs": {
                "downtime_depth": 1.0
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!check_backup_spr",
            "type": "Service"
        }
    ]
}

Expected behavior

The value must be 1 for all the services of the host during dowtime.

Your Environment

icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 16.04.6 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.15.0-1113-azure
  Architecture: x86_64

Build information:
  Compiler: GNU 5.4.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
julianbrost commented 3 years ago

What results do you get for the following API queries? Do they show objects for each service?

'https://localhost:5665/v1/objects/scheduleddowntimes' -d '{"filter":"match(\"yel-spr-prd-gce-bdd-lnx-mysql-003\", host.name)"}'
'https://localhost:5665/v1/objects/downtimes' -d '{"filter":"match(\"yel-spr-prd-gce-bdd-lnx-mysql-003\", host.name)"}'

The latter one is more interesting during the downtime period, but you may still check it outside it as the downtime objects are already created before their interval starts.

julianbrost commented 3 years ago

There also was a bug fix in version 2.12.5 that could potentially explain the issue: #8863/#8848

If that was the case, you should see critical/ThreadPool: Exception thrown in event handler: messages in your logs. Can you please check that?

maelig commented 3 years ago

@julianbrost , results API request outside downtime hours : 

{
    "results": [
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!ac10622d-12bd-4b33-8f34-bb0f24152b16",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 13h - 14h30",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!DOWNTIME-SPR-BDD",
                "duration": 0.0,
                "end_time": 1631190600.0,
                "entry_time": 1631098856.641932,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 57.0,
                "name": "ac10622d-12bd-4b33-8f34-bb0f24152b16",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!DOWNTIME-SPR-BDD",
                "service_name": "SPR: Check Cluster Galera State",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Galera State!ac10622d-12bd-4b33-8f34-bb0f24152b16.conf"
                },
                "start_time": 1631185200.0,
                "templates": [
                    "ac10622d-12bd-4b33-8f34-bb0f24152b16"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631098856.641964,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!ac10622d-12bd-4b33-8f34-bb0f24152b16",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!2a39aacb-24b8-4383-bfbe-3fd32b4a03b2",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 13h - 14h30",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!DOWNTIME-SPR-BDD",
                "duration": 0.0,
                "end_time": 1631190600.0,
                "entry_time": 1631098856.701418,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 24.0,
                "name": "2a39aacb-24b8-4383-bfbe-3fd32b4a03b2",
                "original_attributes": null,
                "package": "_api",
                "paused": true,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!DOWNTIME-SPR-BDD",
                "service_name": "FS Warning Root Bdd Spr",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!2a39aacb-24b8-4383-bfbe-3fd32b4a03b2.conf"
                },
                "start_time": 1631185200.0,
                "templates": [
                    "2a39aacb-24b8-4383-bfbe-3fd32b4a03b2"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631098856.701443,
                "was_cancelled": false,
                "zone": "master"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!2a39aacb-24b8-4383-bfbe-3fd32b4a03b2",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!66460daf-3305-484d-815b-0a5820acd5e6",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 13h - 14h30",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!DOWNTIME-SPR-BDD",
                "duration": 0.0,
                "end_time": 1631190600.0,
                "entry_time": 1631098856.733322,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 9.0,
                "name": "66460daf-3305-484d-815b-0a5820acd5e6",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!DOWNTIME-SPR-BDD",
                "service_name": "Process Mysqld Spr",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!66460daf-3305-484d-815b-0a5820acd5e6.conf"
                },
                "start_time": 1631185200.0,
                "templates": [
                    "66460daf-3305-484d-815b-0a5820acd5e6"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631098856.733347,
                "was_cancelled": false,
                "zone": "master"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!66460daf-3305-484d-815b-0a5820acd5e6",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!d837af3b-af08-4c44-a31c-9b46c8d18caa",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 13h - 14h30",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!DOWNTIME-SPR-BDD",
                "duration": 0.0,
                "end_time": 1631190600.0,
                "entry_time": 1631098856.671565,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 83.0,
                "name": "d837af3b-af08-4c44-a31c-9b46c8d18caa",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!DOWNTIME-SPR-BDD",
                "service_name": "SPR: Check Flag for backup",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Flag for backup!d837af3b-af08-4c44-a31c-9b46c8d18caa.conf"
                },
                "start_time": 1631185200.0,
                "templates": [
                    "d837af3b-af08-4c44-a31c-9b46c8d18caa"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631098856.671592,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!d837af3b-af08-4c44-a31c-9b46c8d18caa",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!812da099-2b3c-44e4-b92d-552bcd1d8c54",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 13h - 14h30",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!DOWNTIME-SPR-BDD",
                "duration": 0.0,
                "end_time": 1631190600.0,
                "entry_time": 1631098856.532148,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 107.0,
                "name": "812da099-2b3c-44e4-b92d-552bcd1d8c54",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!DOWNTIME-SPR-BDD",
                "service_name": "SPR: Check Cluster Galera Size",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Galera Size!812da099-2b3c-44e4-b92d-552bcd1d8c54.conf"
                },
                "start_time": 1631185200.0,
                "templates": [
                    "812da099-2b3c-44e4-b92d-552bcd1d8c54"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631098856.532179,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!812da099-2b3c-44e4-b92d-552bcd1d8c54",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!e7978bc9-45a2-4474-a6e2-07931ece4381",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 13h - 14h30",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!DOWNTIME-SPR-BDD",
                "duration": 0.0,
                "end_time": 1631190600.0,
                "entry_time": 1631098856.626434,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 51.0,
                "name": "e7978bc9-45a2-4474-a6e2-07931ece4381",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!DOWNTIME-SPR-BDD",
                "service_name": "SPR: Check Cluster Galera Ready",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Galera Ready!e7978bc9-45a2-4474-a6e2-07931ece4381.conf"
                },
                "start_time": 1631185200.0,
                "templates": [
                    "e7978bc9-45a2-4474-a6e2-07931ece4381"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631098856.626457,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!e7978bc9-45a2-4474-a6e2-07931ece4381",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!d737aa14-3400-43ad-9832-9ea956f26096",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 13h - 14h30",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!DOWNTIME-SPR-BDD",
                "duration": 0.0,
                "end_time": 1631190600.0,
                "entry_time": 1631098856.657265,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 117.0,
                "name": "d737aa14-3400-43ad-9832-9ea956f26096",
                "original_attributes": null,
                "package": "_api",
                "paused": true,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!DOWNTIME-SPR-BDD",
                "service_name": "SPR: Check Cluster Wordpress Base available",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Wordpress Base available!d737aa14-3400-43ad-9832-9ea956f26096.conf"
                },
                "start_time": 1631185200.0,
                "templates": [
                    "d737aa14-3400-43ad-9832-9ea956f26096"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631098856.65729,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!d737aa14-3400-43ad-9832-9ea956f26096",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!3ea96ab6-e65f-4d9c-8d72-517070b15935",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 23h - 23h59",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!DOWNTIME-SPR-BDD2",
                "duration": 0.0,
                "end_time": 1631224740.0,
                "entry_time": 1631134815.512669,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 152.0,
                "name": "3ea96ab6-e65f-4d9c-8d72-517070b15935",
                "original_attributes": null,
                "package": "_api",
                "paused": true,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!DOWNTIME-SPR-BDD2",
                "service_name": "SPR: Check Cluster Galera Size",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Galera Size!3ea96ab6-e65f-4d9c-8d72-517070b15935.conf"
                },
                "start_time": 1631221200.0,
                "templates": [
                    "3ea96ab6-e65f-4d9c-8d72-517070b15935"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631134815.512711,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Size!3ea96ab6-e65f-4d9c-8d72-517070b15935",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!e6e0769c-bf86-4aec-b015-0eecc94a51a7",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 23h - 23h59",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!DOWNTIME-SPR-BDD2",
                "duration": 0.0,
                "end_time": 1631224740.0,
                "entry_time": 1631134815.528748,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 153.0,
                "name": "e6e0769c-bf86-4aec-b015-0eecc94a51a7",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!DOWNTIME-SPR-BDD2",
                "service_name": "SPR: Check Cluster Galera Ready",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Galera Ready!e6e0769c-bf86-4aec-b015-0eecc94a51a7.conf"
                },
                "start_time": 1631221200.0,
                "templates": [
                    "e6e0769c-bf86-4aec-b015-0eecc94a51a7"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631134815.528769,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera Ready!e6e0769c-bf86-4aec-b015-0eecc94a51a7",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!0275a1e2-19d9-4d92-8dcb-89321f8ef307",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 23h - 23h59",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!DOWNTIME-SPR-BDD2",
                "duration": 0.0,
                "end_time": 1631224740.0,
                "entry_time": 1631134815.600285,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 158.0,
                "name": "0275a1e2-19d9-4d92-8dcb-89321f8ef307",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!DOWNTIME-SPR-BDD2",
                "service_name": "SPR: Check Cluster Wordpress Base available",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Wordpress Base available!0275a1e2-19d9-4d92-8dcb-89321f8ef307.conf"
                },
                "start_time": 1631221200.0,
                "templates": [
                    "0275a1e2-19d9-4d92-8dcb-89321f8ef307"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631134815.60031,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Wordpress Base available!0275a1e2-19d9-4d92-8dcb-89321f8ef307",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!40df6572-d12a-44d5-9cc1-6da7af08afeb",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 23h - 23h59",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!DOWNTIME-SPR-BDD2",
                "duration": 0.0,
                "end_time": 1631224740.0,
                "entry_time": 1631134815.61402,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 159.0,
                "name": "40df6572-d12a-44d5-9cc1-6da7af08afeb",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!DOWNTIME-SPR-BDD2",
                "service_name": "SPR: Check Cluster Galera State",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Cluster Galera State!40df6572-d12a-44d5-9cc1-6da7af08afeb.conf"
                },
                "start_time": 1631221200.0,
                "templates": [
                    "40df6572-d12a-44d5-9cc1-6da7af08afeb"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631134815.614044,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Cluster Galera State!40df6572-d12a-44d5-9cc1-6da7af08afeb",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!169934d3-3bfe-4881-9a68-d98ecef175a2",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 23h - 23h59",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!DOWNTIME-SPR-BDD2",
                "duration": 0.0,
                "end_time": 1631224740.0,
                "entry_time": 1631134815.643717,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 161.0,
                "name": "169934d3-3bfe-4881-9a68-d98ecef175a2",
                "original_attributes": null,
                "package": "_api",
                "paused": false,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!DOWNTIME-SPR-BDD2",
                "service_name": "SPR: Check Flag for backup",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!SPR%3A Check Flag for backup!169934d3-3bfe-4881-9a68-d98ecef175a2.conf"
                },
                "start_time": 1631221200.0,
                "templates": [
                    "169934d3-3bfe-4881-9a68-d98ecef175a2"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631134815.64374,
                "was_cancelled": false,
                "zone": "gcp"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!SPR: Check Flag for backup!169934d3-3bfe-4881-9a68-d98ecef175a2",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!c14dc26d-421e-455a-b391-7eef7397243d",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 23h - 23h59",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!DOWNTIME-SPR-BDD2",
                "duration": 0.0,
                "end_time": 1631224740.0,
                "entry_time": 1631134815.658136,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 162.0,
                "name": "c14dc26d-421e-455a-b391-7eef7397243d",
                "original_attributes": null,
                "package": "_api",
                "paused": true,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!DOWNTIME-SPR-BDD2",
                "service_name": "Process Mysqld Spr",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!c14dc26d-421e-455a-b391-7eef7397243d.conf"
                },
                "start_time": 1631221200.0,
                "templates": [
                    "c14dc26d-421e-455a-b391-7eef7397243d"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631134815.658159,
                "was_cancelled": false,
                "zone": "master"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!Process Mysqld Spr!c14dc26d-421e-455a-b391-7eef7397243d",
            "type": "Downtime"
        },
        {
            "attrs": {
                "__name": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!02dc0a7a-1c88-41da-9b40-62ecce2e6098",
                "active": true,
                "author": "jaubin",
                "authoritative_zone": "master",
                "comment": "Scheduled downtime each day for backup 23h - 23h59",
                "config_owner": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!DOWNTIME-SPR-BDD2",
                "duration": 0.0,
                "end_time": 1631224740.0,
                "entry_time": 1631134815.687079,
                "fixed": true,
                "ha_mode": 0.0,
                "host_name": "yel-spr-prd-gce-bdd-lnx-mysql-003",
                "legacy_id": 164.0,
                "name": "02dc0a7a-1c88-41da-9b40-62ecce2e6098",
                "original_attributes": null,
                "package": "_api",
                "paused": true,
                "scheduled_by": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!DOWNTIME-SPR-BDD2",
                "service_name": "FS Warning Root Bdd Spr",
                "source_location": {
                    "first_column": 0.0,
                    "first_line": 1.0,
                    "last_column": 69.0,
                    "last_line": 1.0,
                    "path": "/var/lib/icinga2/api/packages/_api/7848a1a7-a7a5-45ba-a9fd-d95bcc4179e2/conf.d/downtimes/yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!02dc0a7a-1c88-41da-9b40-62ecce2e6098.conf"
                },
                "start_time": 1631221200.0,
                "templates": [
                    "02dc0a7a-1c88-41da-9b40-62ecce2e6098"
                ],
                "trigger_time": 0.0,
                "triggered_by": "",
                "triggers": [],
                "type": "Downtime",
                "version": 1631134815.687102,
                "was_cancelled": false,
                "zone": "master"
            },
            "joins": {},
            "meta": {},
            "name": "yel-spr-prd-gce-bdd-lnx-mysql-003!FS Warning Root Bdd Spr!02dc0a7a-1c88-41da-9b40-62ecce2e6098",
            "type": "Downtime"
        }
    ]
}
maelig commented 3 years ago

I only found the message once in the last 7 days : 

/var/log/icinga2/icinga2.log.2.gz:[2021-09-07 15:30:21 +0200] critical/ThreadPool: Exception thrown in event handler:
julianbrost commented 3 years ago

There should be a message with the reason for the exception following in the next few lines. So something like grep -A10 should reveal the cause.

If that was interfering with scheduling downtimes, I'd expect it to appear more often, would be interesting to know nonetheless, might also be another issue that could be worth looking into.

maelig commented 3 years ago

@julianbrost, here is the cause reveal (livestatus) :

zgrep -A10 'Exception thrown in event handler' /var/log/icinga2/icinga2.log.2.gz
[2021-09-07 15:30:21 +0200] critical/ThreadPool: Exception thrown in event handler:
Error: Livestatus header must contain a verb.

    (0) icinga2: icinga::LivestatusQuery::LivestatusQuery(std::vector<icinga::String, std::allocator<icinga::String> > const&, icinga::String const&) (+0x274f) [0x96e7bf]
    (1) icinga2: icinga::LivestatusListener::ClientHandler(boost::intrusive_ptr<icinga::Socket> const&) (+0x16b) [0x9744db]
    (2) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x68ecdd]
    (3) icinga2: boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<bool icinga::ThreadPool::Post<std::function<void ()> >(std::function<void ()>, icinga::SchedulerPolicy)::{lambda()#1}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, std::allocator<void>*, boost::system::error_code const&, unsigned long) (+0xdf) [0x68ef9f]
    (4) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0xaf1ff6]
    (5) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x695b95]
    (6) icinga2: boost_asio_detail_posix_thread_function (+0xf) [0x5caa5f]
    (7) libpthread.so.0: <unknown function> (+0x76ba) [0x7f010d79d6ba]
julianbrost commented 3 years ago

Okay, that should be unrelated to the issue here. And probably only some invalid query sent to icinga2.

Al2Klimov commented 2 years ago

@julianbrost You’ve assigned the OP. Is that still valid?