Support rolling updates

[fhunleth]

Instead of requesting all devices to update simultaneously, it's useful to roll out the updates. This has two benefits:

1. Should an update break a device or cause a support call, it gives time to pause the deployment.
2. It can reduce the pressure on any network bottlenecks since the .fw files are large.

[joshk]

I've spoken to @fhunleth and @lawik about this topic. We discussed something akin to a CI deployment pipeline with different stages and success criteria to progress further.

I'm going to keep this issue open as this would be a major improvement to rolling out new firmware to large device fleets.

[oestrich]

fwiw we do support rolling updates now as described above. That’s what the deployment orchestrator does

[joshk]

Thanks, @oestrich. From my understanding, the current rolling updates don't include a way to monitor health after an update or to automatically stop the update if conditions aren't met. Is that correct?

[oestrich]

It will, the orchestrator hands out the inflight update and then the device channel resolves the update which uses the failure rate to block out updates. So it's not entirely straight forward but its there

[joshk]

Ok, so if 100 devices can't install the lastest update in a deployment the deployment orchestrator will stop updating all other devices?

And I'm assuming this is based on cases where there was a failure to download the fwup file and use fwup to install the update, but not cases where a device goes into an unhealthy state and has to rollback?