Re-applying a failed `Plan` does not re-trigger signalling

jnummelin commented 8 months ago

Before creating an issue, make sure you've checked the following:

[X] You are running the latest released version of k0s
[X] Make sure you've searched for existing issues, both open and closed
[X] Make sure you've searched for PRs too, a fix might've been merged already
[X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Not relevant

Version

v1.28.5

Sysinfo

`k0s sysinfo`

➡️ Please replace this text with the output of `k0s sysinfo`. ⬅️

What happened?

Applying autopilot Planwith e.g. a wrong SHA for k0s binary makes it stall. Even if user re-applies with a correct SHA it's still "stuck" on failed state.

status:
  commands:
    - id: 0
      k0supdate:
        controllers:
          - lastUpdatedTimestamp: '2024-02-02T09:56:19Z'
            name: master1
            state: SignalPending
          - lastUpdatedTimestamp: '2024-02-02T09:56:19Z'
            name: master2
            state: SignalApplyFailed
          - lastUpdatedTimestamp: '2024-02-02T09:56:19Z'
            name: master3
            state: SignalPending
      state: ApplyFailed
  state: ApplyFailed

The steps to workaround:

Delete the old plan object
Delete the autopilot annotation on related node object(s), including ControlNodes

Docs do not really say anything how to recover from these kinds of cases. 😞

Steps to reproduce

Apply Plan with a wrong SHA for k0s download
Observe the Plan getting to failed state
Re-apply with correct SHA
Observe the Plan still being in failed state

Expected behavior

Re-applying with a correct SHA should make autopilot to proceed. Autopilot should be able to figure out the Plan is modified and re-trigger the signalling for failed nodes. Maybe we could add the plan resourceVersion into the signalling data so AP can figure out if it should re-trigger the failed signal?

Actual behavior

No response

Screenshots and logs

time="2024-02-02 10:22:41" level=error msg="Unable to download 'https://github.com/k0sproject/k0s/releases/download/v1.29.1+k0s.0/k0s-v1.29.1+k0s.0-amd64': checksum mismatch" component=a
utopilot controller=ControlNode leadermode=false object=ControlNode reconciler=downloading signalnode=master2

The