cmacknz
commented
1 year ago Fleet and agent currently do not expect that the policy on the agent side can be modified unexpectedly, so there is no mechanism to detect that the agent policy is out of sync with what Fleet previously sent and correct that situation.
We would likely need to include the policy ID and revision in the checkin request made from the agent to fleet server:
Fleet server does not show the current state of running elastic-agents
This is an issue / bug related to what policies are displayed in the fleet interface VS what is actually running on the agent.
For context: In our environment, VM snapshots are used frequently here to test and rollback changes, sometimes days apart.
Setup Make an elastic-agent apply policy 'a' and take a VM snapshot, change that agent to use policy 'b' in the fleet interface, wait for it to be applied, then revert the VM snapshot.
Result This will cause the fleet server to still report the agent as running policy 'b' even though policy 'a' is loaded and running on the elastic agent. The only way I have found to correct this behaviour is to go into the fleet interface (or API), and switch the agents policy and then correct it. The agent will also report it's status as healthy in the fleet user interface.
I have confirmed this behaviour by running the elastic-agent diagnostics collect command and comparing what is there against what policy is displayed in fleet.
This is currently deployed in a production setting and causing us to miss events because we can't trust what is displayed in fleet.