Ideas for reducing our number of pulls from Docker when using Shepherd

[cpswan]

1. We could RUN_ONCE_AND_EXIT every time we push a new image to the registry. I've mentioned webhooks, but in practice it's probably easiest to SSH from the GitHub Action and just run the script.
2. When the script tests for [ "$previousImage" == "$currentImage" ] we could simply exit rather than continuing to iterate through every other service that's going to have the same outcome (because we're running a consistent fleet).
3. The reason why Shepherd does so many pulls is it updates the service then asks "is this version of the image the same as the last version of the image?", and Docker does a pull each time to get the latest image_with_digest to see if there's a new one - and it does this for EVERY service instance (in our case every secondary), so effectively it's asking the same question AGAIN and AGAIN and AGAIN. But once we have the answer we should update using image_with_digest rather than image - then Docker shouldn't have to do a pull to see what the latest digest is, because it already knows which digest we're demanding.

What I envisage here is an outer loop, which uses a singleton canary service to see if there's a new image_with_digest for a given tag, and then an inner loop that updates all the other services to use that image_with_digest when there is a new one.

If we wanted to keep this generic (so it would still work for a mixed fleet) then perhaps:

1. Iterate across the fleet to extract the image names, and compose a consolidated list
2. Check each item in the list (using a singular service instance) for updates
3. Where there is an update, use image_with_digest to modify all the other service instances using that image

[cconstab]

Solved/mitigated with the introduction of a pull through cache registry

[cconstab]

Pull through caching registry solved issues for now continuing to use shepherd, sheepdog on backlog