"icinga2 object list" does not reflect changes made via API

oxzi commented 1 month ago

Describe the bug

Changes performed via the Icinga 2 API on an object defined from config are not reflected in the icinga2 object list output.

To Reproduce

I am using an IcingaDB object for this example, as this is were I came across this bug.

Starting with a default IcingaDB object as:

$ cat /etc/icinga2/features-enabled/icingadb.conf
object IcingaDB "icingadb" {
  host = "redis-master1"
  //port = 6380
  //password = "xxx"
}

Then, perform a (questionable) change via the API. This will only affect the connection after a restart, see #10102.

$ curl -k -s -S -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/objects/IcingaDBs/icingadb' -d '{"attrs": {"port": 9999}, "pretty":
true}'

{
    "results": [
        {
            "code": 200,
            "name": "icingadb",
            "status": "Attributes updated.",
            "type": "IcingaDB"
        }
    ]
}

Check the state…

via icinga2 objects list

$ icinga2 daemon -C --dump-objects
[snip]
$ icinga2 object list --type icingadb | grep port
* port = 6380

via and the API.

$ curl -k -s -u root:icinga -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' 'https://localhost:5665/v1/objects/icingadbs' -d '{"pretty": true}'
{
"results": [
    {
        "attrs": {
            "__name": "icingadb",
            [snip]
            "original_attributes": {
                "port": 6380
            },
            [snip]
            "port": 9999,
    }
]
}

While the Icinga 2 API represents the changed port as well as the original_attributes field of the Common Runtime Attributes, the output of icinga2 object list is outdated.

Neither CLI command: Object nor Configuration Troubleshooting, List Configuration Objects document this behavior as a limitation.

Expected behavior

I would expect icinga2 object list to show the updated port attribute.

Screenshots

N/A

Your Environment

Include as many relevant details about the environment you experienced the problem in

Version used (icinga2 --version): v2.14.0-293-g4f690a6fd
Operating System and version: Debian GNU/Linux 11
Enabled features (icinga2 feature list): api checker icingadb notification
Icinga Web 2 version and modules (System - About): N/A
Config validation (icinga2 daemon -C): :ballot_box_with_check:
If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

Additional context

N/A

yhabteab commented 1 month ago

Check the state…

via icinga2 objects list

$ icinga2 daemon -C --dump-objects
[snip]
$ icinga2 object list --type icingadb | grep port
  * port = 6380

How long did you wait after changing the attribute before dumping the objects with ... --dump-objects? The icinga2 daemon -C --dump-objects command does not dump the in memory state of the objects, but reads it from the individual configuration files. However, the /var/lib/icinga2/modified_attributes.conf is only flushed every 5m, so you must at least wait for the Dumping program state to file ... log entry after the API request, which indicates that the modified attributes and the Icinga 2 state files have actually been flushed.

oxzi commented 1 month ago

How long did you wait after changing the attribute before dumping the objects with ... --dump-objects?

I initially realized this after restarting Icinga 2, as the changed did not take effect during the prior runtime (as stated above). Afterwards, I played a bit with the API and so on, which should have taken longer than five minutes, but I cannot say so with certainty.

To test it, I just gave it another shot.

I updated the port again.

[2024-09-23 11:52:54 +0000] information/HttpServerConnection: Request POST /v1/objects/IcingaDBs/icingadb (from [::ffff:172.24.0.1]:58248, user: root, agent: curl/8.7.1, status: OK) took 0ms.
For the moment, both the API and icinga2 object list behave as documented in the initial post.
Sipping coffee while waiting :-)
Icinga 2 is dumping the program state.

[2024-09-23 11:57:33 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'

icinga2 object list still shows the outdated port:

$ icinga2 daemon -C --dump-objects
[snip]

$ icinga2 object list --type icingadb
Object 'icingadb' of type 'IcingaDB':
 % declared in '/etc/icinga2/features-enabled/icingadb.conf', lines 1:0-1:25
 * __name = "icingadb"
[snip]
 * port = 6380
[snip]

Btw, the modified-attributes.conf reflects my change:

$ cat /var/lib/icinga2/modified-attributes.conf
var obj = get_object("IcingaDB", "icingadb")
if (obj) {
       obj.modify_attribute("port", 9999.000000)
       obj.version = 1727092374.937358
}

Thus, it seem like icinga2 object list ignores the modified-attributes.conf, at least in this case.

yhabteab commented 1 month ago

5. icinga2 object list still shows the outdated port:

Ah, you actually have to dump the objects again here with the --dump-objects flag :).

oxzi commented 1 month ago

icinga2 object list still shows the outdated port:

Ah, you actually have to dump the objects again here with the --dump-objects flag :).

Sorry, I pasted the icinga2 object list --type icingadb command twice. The first command was icinga2 daemon -C --dump-objects. I have edited the code block in the previous comment.

oxzi commented 1 month ago

Just had another idea and gave it another shot.

$ curl -k -s -S -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/objects/IcingaDBs/icingadb' -d '{"attrs": {"port": 9999}, "pretty": true}'
{
    "results": [
        {
            "code": 200,
            "name": "icingadb",
            "status": "Attributes updated.",
            "type": "IcingaDB"
        }
    ]
}

The Icinga 2 log honors this change:

[2024-09-23 12:17:57 +0000] information/HttpServerConnection: Request POST /v1/objects/IcingaDBs/icingadb (from [::ffff:172.24.0.1]:52684, user: root, agent: curl/8.7.1, status: OK) took 0ms.

Having a look at the /var/cache/icinga2/icinga2.debug file:

$ stat /var/cache/icinga2/icinga2.debug
[snip]
Change: 2024-09-23 12:12:09.980045950 +0000
[snip]

$ grep -o '"port":6380' /var/cache/icinga2/icinga2.debug
"port":6380

Now, Icinga 2 has dumped its state again:

[2024-09-23 12:22:33 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'

Let's verify:

$ icinga2 daemon -C --dump-objects
[snip]

$ stat /var/cache/icinga2/icinga2.debug
[snip]
Change: 2024-09-23 12:23:11.318267722 +0000
[snip]

$ grep -o '"port":6380' /var/cache/icinga2/icinga2.debug
"port":6380
$ grep -o '"port":9999' /var/cache/icinga2/icinga2.debug
$ grep -o ':9999' /var/cache/icinga2/icinga2.debug

So.. the port is not changed after the state dump in this file.

yhabteab commented 1 month ago

It seems to be a bug as the modified attributes are restored while activating the objects and not committing them and the icinga2 daemon -C --dump-objects does only the later.. See https://github.com/Icinga/icinga2/issues/8699

Al2Klimov commented 1 month ago

It seems to be a bug as the modified attributes are restored while activating the objects and not committing them

I think #8155 should fix exactly this due to 689f28082dbf6fc94b2b8b5aa7cd5a10f051de8c

Icinga / icinga2