Add jitter to devices' setting their deployment on deployment change

[oestrich]

If we have a large fleet of devices, we could cause cascading ecto pool issues if thousands+ of devices go to set their deployment at the same time. Jitter over 5 minutes to help avoid slamming the pool. Also space out resolving a deployment for attached devices over the same time frame.

[joshk]

Question: would this potentially mean that Deployments with just 1 Device in them might have a 5-minute wait until the Device updates (max jitter time for that device)?

If so, I think this jitter should not apply when the device count for a deployment is under a certain threshold, and maybe it should be smaller than 5 minutes for cases with different device counts.

[oestrich]

It could take 5 minutes for 1 device if that’s all you have. There’s no way of knowing how many devices the deployment is targeting until you let something calculate attachment and that’s a very heavy thing to do. I also don’t know that it’s worth having branches for something like this. There was already a minute jitter for devices that were attached so this makes things more even

[joshk]

In this case, I would prefer this to be a runtime config.

When a deployment has A LOT of devices in it, well, this makes sense. For the 99% of NervesHub installs, this jitter could feel like an unexpected delay. And with Lars and I doing more demos and conf talks, including showing off how deployments work, having features feel snappy can lead to people getting to an 'aha' moment.

[fhunleth]

I don't have strong opinions here, but I really like the idea of a runtime config to allow the large scale deployment needs to not degrade small and demo deployments.