Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2k stars 574 forks source link

Scheduled Downtime won't be changed/deleted after reload #10042

Closed Mordecaine closed 4 months ago

Mordecaine commented 6 months ago

Describe the bug

If I create a Scheduled Downtime in a global Zone via DSL, it is not possible to delete/change it afterwards

To Reproduce

Result: The old Downtime will not be changed and is still active In the Log you can find the following lines:

[2024-04-16 13:41:24 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:41:24 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/37356bcb-7b59-441c-82fc-f384fae69a60/conf.d/downtimes/msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965.conf
[2024-04-16 13:41:24 +0200] information/ConfigObjectUtility: Created and activated object 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965' of type 'Downtime'.
[2024-04-16 13:41:24 +0200] information/Downtime: Added downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965' between '2024-04-21 20:00:00' and '2024-04-21 22:00:00', author: 'Lasse', flexible with 3600s duration
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-rsyslog' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!pending_mcms_updates' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!pending_updates' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!sentinelone-agent-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!disk' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-sssd' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-sentinelone' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!swap' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!interface-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!pending_updates_security-only' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!uptime' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!disk-io' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-chronyd' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!ntp-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!memory' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!about-me' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!dmesg' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!mountpoints-status' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-vmtoolsd' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!selinux-mode' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!cpu' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:56 +0200] notice/Downtime: Checkable 'msd-lwtest01!service-crond' already in a NOT-OK state. Triggering downtime now.
[2024-04-16 13:45:57 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:01 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:05 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:08 +0200] notice/ApiListener: Received config update for object: {"config":"object Downtime \"916b75be-2881-40f4-ab62-de302e93c965\" ignore_on_error {\n\tauthor = \"Lasse\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"test 123\"\n\tconfig_owner = \"msd-lwtest01!test123\"\n\tconfig_owner_hash = \"7d7c18a918e1b2e7397f941b7080a5b0a8c1b65905d86d123838f21f69e1e7e3\"\n\tduration = 3600.000000\n\tend_time = 1713729600.000000\n\tentry_time = 1713267684.576673\n\tfixed = false\n\thost_name = \"msd-lwtest01\"\n\tparent = \"\"\n\tscheduled_by = \"msd-lwtest01!test123\"\n\tstart_time = 1713722400.000000\n\ttriggered_by = \"\"\n\tversion = 1713267684.576721\n\tzone = \"msd-private_vcloud\"\n}\n","modified_attributes":{},"name":"msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965","original_attributes":[],"type":"Downtime","version":1713267684.576721,"zone":"msd-private_vcloud"}
[2024-04-16 13:46:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:46:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:47:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:47:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:48:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:48:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:49:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:49:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:50:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:50:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:51:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:51:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'
[2024-04-16 13:52:56 +0200] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "msd-lwtest01!test123"
[2024-04-16 13:52:56 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'msd-lwtest01!test123': Error: Cannot remove downtime 'msd-lwtest01!916b75be-2881-40f4-ab62-de302e93c965'. It is owned by scheduled downtime object 'msd-lwtest01!test123'

Expected behavior

The Downtime should be updated or deleted. Furthermore, in the logfile is written, 'Checkable msd-lwtest01!cpu' already in a NOT-OK state. Triggering downtime now, but the downtime should be started in the future and not at the reload time.

Your Environment

Include as many relevant details about the environment you experienced the problem in

log1-c commented 6 months ago

Tested this on a different system and can confirm, but also with IcingaDB installed.

apply ScheduledDowntime "test123" to Host {
      author = "Test"
      comment = "test 123"

      fixed = false
      duration = 60m

      ranges = {
        "sunday 3" = "20:00-22:00"
      }

      assign where host.name == "g-ici-mon01"
    }
Object 'g-ici-mon01!test123' of type 'ScheduledDowntime':
  % declared in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * __name = "g-ici-mon01!test123"
  * author = "Test"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 2:4-2:18
  * child_options = "DowntimeNoChildren"
  * comment = "test 123"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 3:4-3:23
  * duration = 3600
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 6:4-6:17
  * fixed = false
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 5:4-5:16
  * host_name = "g-ici-mon01"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * name = "test123"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * ranges
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 8:4-10:4
    * sunday 3 = "20:00-22:00"
  * service_name = ""
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 40
    * last_line = 1
    * path = "/etc/icinga2/zones.d/master/downtime.conf"
  * templates = [ "test123" ]
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * type = "ScheduledDowntime"
  * vars = null
  * zone = "master"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
[2024-04-18 09:21:31 +0200] information/ConfigObjectUtility: Created and activated object 'g-ici-mon01!8ee97360-f554-4896-bf93-2a188859ad42' of type 'Downtime'.
[2024-04-18 09:21:31 +0200] information/Downtime: Added downtime 'g-ici-mon01!8ee97360-f554-4896-bf93-2a188859ad42' between '2024-04-21 20:00:00' and '2024-04-21 22:00:00', author: 'Test', flexible with 3600s duration

Change downtime to

apply ScheduledDowntime "test123" to Host {
          author = "Test"
          comment = "test 123"

          fixed = false
          duration = 60m

          ranges = {
            "sunday 4" = "20:00-22:00"
          }

          assign where host.vars.is_icinga2_master
        }

systemctl reload icinga2

Object 'g-ici-mon01!test123' of type 'ScheduledDowntime':
  % declared in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * __name = "g-ici-mon01!test123"
  * author = "Test"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 2:4-2:18
  * child_options = "DowntimeNoChildren"
  * comment = "test 123"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 3:4-3:23
  * duration = 3600
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 6:4-6:17
  * fixed = false
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 5:4-5:16
  * host_name = "g-ici-mon01"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * name = "test123"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * ranges
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 8:4-10:4
    * sunday 4 = "20:00-22:00"
  * service_name = ""
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 40
    * last_line = 1
    * path = "/etc/icinga2/zones.d/master/downtime.conf"
  * templates = [ "test123" ]
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
  * type = "ScheduledDowntime"
  * vars = null
  * zone = "master"
    % = modified in '/etc/icinga2/zones.d/master/downtime.conf', lines 1:0-1:40
[2024-04-18 09:26:55 +0200] information/Downtime: Added downtime 'g-ici-mon01.!498f381c-7246-42a9-815d-8d6a100f79d7' between '2024-04-28 20:00:00' and '2024-04-28 22:00:00', author: 'Test', flexible with 3600s duration
[2024-04-18 09:26:55 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'g-ici-mon01!test123': Error: Cannot remove downtime 'g-ici-mon01!8ee97360-f554-4896-bf93-2a188859ad42'. It is owned by scheduled downtime object 'g-ici-mon01!test123'

Removing the config file then yields no results via icinga2 object list but both downtimes remain visible in the webinterface. And no log entries any more like the one above. image

Icinga2 2.14.2 Icinga Web 2 Version 2.12.1 icinga/icinga-php-library 0.13.1 icinga/icinga-php-thirdparty 0.12.1 director 1.11.1
icingadb 1.1.1
incubator 0.22.0

edit: Can only get rid of the downtimes in the webinterface when delteing, reloading and then recreating the host object

log1-c commented 6 months ago

Just did another test, this time on a IDO based system.

Same/Similar behavior, this time the downtime was configured via the Icinga Director and I only modified the range, without touching the assignment condition. image

[2024-04-18 10:20:25 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 's049wl04!Wartungsfenster 20.04. EDI/ERP/BI': Error: Cannot remove downtime 's049wl04!358770b1-def1-4a09-8a7a-f2b126f5036a'. It is owned by scheduled downtime object 's049wl04!Wartungsfenster 20.04. EDI/ERP/BI'
[2024-04-18 10:20:25 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 's049as43!Wartungsfenster 20.04. EDI/ERP/BI': Error: Cannot remove downtime 's049as43!acdbc208-3e43-403d-876a-1cff6e2f5102'. It is owned by scheduled downtime object 's049as43!Wartungsfenster 20.04. EDI/ERP/BI'
[2024-04-18 10:20:25 +0200] critical/ScheduledDowntime: Exception occurred during removal of obsolete downtime for scheduled downtime 'c049vm115!Wartungsfenster 20.04. EDI/ERP/BI': Error: Cannot remove downtime 'c049vm115!8befcb1e-bb2b-4d3b-b58c-787cb0418fca'. It is owned by scheduled downtime object 'c049vm115!Wartungsfenster 20.04. EDI/ERP/BI'

"Both" downtimes visible in the webinterface: image

Mordecaine commented 6 months ago

I think all scheduled downtimes are affected: Fixed Scheduled Downtimes: yes Flexible Scheduled Downtimes: yes In a global Zone: yes In a non global Zone: yes IDO: yes IcingaDB: yes

Nayakum commented 5 months ago

Not entirely the same behaviour, but I had a similar issue a couple years back where downtimes were "copied" by changing aspects of them, e.g. adding comments: https://github.com/Icinga/icinga2/issues/9142 At that time, it was considered as working as intended.

yhabteab commented 5 months ago

ref/IP/53497