Agent comes back online after unenroll due to too many authentication failures

[AndersonQ]

Users have reported to have several agents were not coming back online after a Fleet Server outage. Those agents had tried to unenroll themselves due to many authentication failures, they logged: retrieved an invalid api key error '7' times. Starting to unenroll the elastic agent.. However restarting the agents brought them back online. That shows the agent did not persist its unenroll.

It happens because as it's a self-inflicted unenroll the agent does not want to trigger a policyChange ack and sets the action to nil

• https://github.com/elastic/elastic-agent/blob/b4877f52a39c53591634394be57e156d79ed57e0/internal/pkg/agent/application/actions/handlers/handler_action_unenroll.go#L86-L89

however a nil action isn't persisted. Actually the agent panics when trying to persist a nil action

• https://github.com/elastic/elastic-agent/blob/b4877f52a39c53591634394be57e156d79ed57e0/internal/pkg/agent/application/actions/handlers/handler_action_unenroll.go#L95-L101
• https://github.com/elastic/elastic-agent/blob/b4877f52a39c53591634394be57e156d79ed57e0/internal/pkg/agent/storage/store/state_store.go#L166-L188

as the action ins't persisted, next time the agent For confirmed bugs, please report:

• Version: 8.6.2, 8.8.2, most likely on main too
• Operating System: all
• Discuss Forum URL: n/a
• Steps to Reproduce:
  • enroll an agent
  • have fleet-server return 7 401 - unauthorised
  • confirm the agent will unenroll itself by finding the log retrieved an invalid api key error '7' times. Starting to unenroll the elastic agent.
  • confirm the agent isn't checking-in with fleet server anymore
  • restart the agent
  • the agent should be back online

[cmacknz]

We should just not unenroll instead: https://github.com/elastic/elastic-agent/issues/5428

[ycombinator]

Closing in favor of https://github.com/elastic/elastic-agent/issues/5428