Describe the bug

Hi, I'm seeing a behaviour which I find weird and I'd like to know if it's a bug or something to expect.

I have two service templates which perform variable overrides based on the configuration/properties of the host.
The first template changes a variable (in what I assumed was some sort of apply Service scope for lack of a better word).
The second sees inconsistent values for that variable.

To Reproduce

I have the following templates:

apply Service for (site => config in host.vars.http_vhosts) {
  import "generic-service"

  check_command = "http"
  vars.http_ssl = true
  vars.http_sni = true
  vars.http_ipv4 = true

  # Never check the certificate here, we have a dedicated service below
  config.remove("http_certificate")
  vars += config
}

apply Service "certificate " for (site => config in host.vars.http_vhosts) {
  import "generic-service"

  check_command = "http"
  # the service name is derived from the string that follows "apply"
  # name = "certificate " + site

  vars.http_certificate = 10
  vars.http_ssl = true
  vars.http_sni = true
  vars.http_ipv4 = true
  vars += config
}

And the following host:

object Host "xxx" {
[…]
  vars.http_vhosts["http"] = {
    http_address = "xxx"
    http_vhost = "xxx"
    http_string = "xxx"
    http_certificate = 7
  }
[…]
}

Whenever I reload or restart Icinga 2, the value of the overridden http_certificate variable is not consistent:

# systemctl restart icinga2 && icinga2 object list -t service -n 'xxx!certificate http' | grep http_certificate
    * http_certificate = 7
# systemctl restart icinga2 && icinga2 object list -t service -n 'xxx!certificate http' | grep http_certificate
    * http_certificate = 10
# systemctl restart icinga2 && icinga2 object list -t service -n 'xxx!certificate http' | grep http_certificate
    * http_certificate = 7
# systemctl restart icinga2 && icinga2 object list -t service -n 'xxx!certificate http' | grep http_certificate
    * http_certificate = 10
# systemctl restart icinga2 && icinga2 object list -t service -n 'xxx!certificate http' | grep http_certificate
    * http_certificate = 10

Expected behavior

I believe the changes which I make to config in the first service template should not affect the second one, therefore, the value should always be 7, never 10. I could understand having the value unset because of the remove but never to have it returned to its default value.

Your Environment

Include as many relevant details about the environment you experienced the problem in

Version used (icinga2 --version):

icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Gentoo
  Platform version: Unknown
  Kernel: Linux
  Kernel version: 5.15.13-gentoo-x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 11.2.1
  Build host: localhost
  OpenSSL version: OpenSSL 1.1.1m  14 Dec 2021

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Operating System and version: Gentoo amd64 unstable
Enabled features (icinga2 feature list):

Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb ido-mysql influxdb2 opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-pgsql influxdb livestatus mainlog notification

Config validation (icinga2 daemon -C):

[2022-01-15 04:36:02 +0100] information/cli: Icinga application loader (version: r2.13.2-1)
[2022-01-15 04:36:02 +0100] information/cli: Loading configuration file(s).
[2022-01-15 04:36:02 +0100] information/ConfigItem: Committing config item(s).
[2022-01-15 04:36:02 +0100] information/ApiListener: My API identity: xxx
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 72 Services.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 LivestatusListener.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 3 Users.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 6 Zones.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 6 ScheduledDowntimes.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 3 HostGroups.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 79 Notifications.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 12 Downtimes.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 Dependency.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 7 Hosts.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 5 Endpoints.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 251 CheckCommands.
[2022-01-15 04:36:02 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2022-01-15 04:36:02 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-01-15 04:36:02 +0100] information/cli: Finished validating the configuration file(s).

If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes → this happens on the master node.

Additional context

Changing the first service to do this (instead of removing http_certificate from config) fixes the issue. However, I'd like to know if this is the expected behaviour or if this is a bug.

  vars += config
  vars.remove("http_certificate")

Icinga / icinga2

Applying the same config for different service templates does not work consistently when one service edits the config #9178

Describe the bug

To Reproduce

Expected behavior

Your Environment

Additional context