st2 ha: packs/register command doesn't execute sensors reload on every node

[engineer-roman]

SUMMARY

I have HA installation of Stackstorm with three nodes and pipeline which deploys my packs over all the nodes. Same pipeline executes API call POST -> /api/v1/packs/register with all the packs and as far as I understand, this command causes sensors reloading. But this is not what happens all the time - sometimes sensor might be reloaded on some node and sometimes not. This leads to cases when:

• some node is running sensors with new code.
• some node is running sensors with old code, but actions with new code.

Such behavior leads to completely upredictable behavior of your workflow based on sensors and actions.

STACKSTORM VERSION

st2 3.3.0, on Python 3.6.9

OS, environment, install method

VM, Ubuntu 18.04, custom install.

Steps to reproduce the problem

• Run HA installation with 3 nodes
• Get some packs with sensors installed
• Register packs via API call POST -> /api/v1/packs/register
• Update sensors code
• Register packs via API call POST -> /api/v1/packs/register
• Check logs of st2sensorcontainer for word "reloaded". You may need to do that few times to catch the issue.

Expected Results

Single API call POST -> /api/v1/packs/register causes sensors reloading on every node in HA installation.

Actual Results

Sometimes one or two nodes continue running sensors (i can see it by updated log messages in sensor's code) with old code and logs of st2sensorcontainer doesn't contain anything about reloading

[engineer-roman]

I just realized it might be expected behavior since (if I understood it right) every node doesn't know about other nodes. Am i right? Issue is still actual but it would make whole case a bit more clear.