Changing a label/annotation used in token substitution can require a restart of sensu-backend

[ichilton]

The labels/annotations seem to be cached on the backend and even when updated (and sensuctl showing the correct new values), the old values are still used in checks until the backend is forcefully restarted to clear.

Expected Behavior

Changes to a label or annotation should just take effect straight away.

Current Behavior

Sometimes (though not always), I can update a label or annotation. The sensu web ui / api show the new value, but the old value keeps being used for checks - even hours later. I have to restart sensu-backend for the change to be picked up, at which point the new value starts being used and the alerts clear straight away.

Steps to Reproduce (for bugs)

I've not quite managed to pin down why this happens some times and not others.

See below for an example of where it just happened, but i've seen it a number of times in recent weeks.

Context

An example of it just happening though - I have some proxy entities created via the API which have an annotation containing an IP address.

I then have checks to ping & ssh to that IP, i.e:

/usr/lib/nagios/plugins/check_ping -H {{ .annotations.ip_address }} -w 200,10% -c 500.0,75% -p 5 -t 5

and

/usr/lib/nagios/plugins/check_ssh -p 22 {{ .annotations.ip_address }}

I added an entity - with yaml piped in to sensuctl create with the wrong IP. It alerted. I then changed the IP, piped the yaml back in to sensuctl create. sensuctl info reported the correct IP, but the check continued to use the original IP so kept failing. A few hours later, I restated sensu-backend and the alerts cleared on the next check.

Your Environment

Sensu Go 5.20.2 running on Debian 10, using the official Debian packages (5.20.2-12959).

[mcatngena]

I can confirm the same issue.

[portertech]

@ichilton is this still happening with a more recent release? i.e. 6.x?

[cynici]

FWIW, I started with Sensu Go v6.2.7 about two month ago and upgraded to v6.3.0. I have not encountered this issue. cc @portertech