Remove promotion/re-tagging feature

[skovhus]

The promoting of images (re-tagging images matching a hash) was introduced in https://github.com/tmrowco/brick/pull/57. Back then it showed an improvement for first build on a branch from 8 minutes to 2 minutes.

But an unforeseen side-effect it that promoting an image steals the latest tag. This means that master builds needs cannot reuse images that matches the hash.

I suggest we either:

• try out removing the promotion of images feature (this PR)
• or we do not use the latest tag, but a random tag matching the dependency hash. It would mean that branch builds might steal from each other instead (but as we have a lot of tags it should be fine).

[madsnedergaard]

Hmm, wouldn't this mean all work on new branches will slow down 4x again, or have we made improvements since then that would help? 🤔

What's the reason merge->master builds use the latest tag, could we do something about that instead of the other way around?

[skovhus]

Hmm, wouldn't this mean all work on new branches will slow down 4x again, or have we made improvements since then that would help? 🤔

The initial build on a branch will be slower. Not sure how much slower... We would need to test that out.

What's the reason merge->master builds use the latest tag, could we do something about that instead of the other way around?

So the dependency hashing can be seen here. It finds images that has the same label as what we are currently building AND that they also have the same dependency hash. On master and on branches we build latest images.

Let me try to illustrate this a bit. So currently this a flow illustrating how the promotion works (newest event first).

With promotion of latest tags from other branches:

• master: builds my_image:latest for the first time (no images matching dependencies hash)
• feature-branch1: promotes my_image:latest to my_image:feature-branch1 (as the dependency hash matched) 🐢
• feature-branch2: cannot find a matching images for my_image (as feature-branch1 stole the latest tag), so it builds it again (the docker cache helps a lot here, so this is fairly fast) 🐇
• feature-branch1: my_image:feature-branch1 is build and dependency hash finds it 🐇
• master: cannot cache my_image as there is no latest tag 🐢

Without promotion:

• master: builds my_image:latest for the first time (no images matching dependencies hash)
• feature-branch1: needs to create my_image:feature-branch1 again (docker cache helps) 🐢
• feature-branch2: needs to create my_image:feature-branch2 again (docker cache helps) 🐢
• feature-branch1: my_image:feature-branch1 is build and dependency hash finds it 🐇
• master: can use a cache version of an_image:latest 🐇

With promotion of any non-latest tags from other branches:

• master: builds my_image:latest for the first time (no images matching dependencies hash)
• feature-branch1: promotes my_image:old-feature-branch to my_image:feature-branch1 (as the dependency hash matched) 🐢
• feature-branch2: promotes my_image:an-even-older-feature-branch to my_image:feature-branch2 (as the dependency hash matched) 🐢
• feature-branch1: my_image:feature-branch1 is build and dependency hash finds it 🐇
• master: cannot cache my_image as there is no latest tag 🐢

Maybe worth explaining in person! :D

[madsnedergaard]

Haha yeah, it's a bit complicated without a whiteboard - but you did a pretty good job anyway! 💪

We can also talk about this in person tomorrow, but posting here nonetheless so that I at least remember it 😄

A few questions/ideas:

1. Could we "duplicate"/copy from an image with a new feature-branch tag instead of "stealing" latest? (like creating new branch from another branch in git)

Example:

• master: builds my_image:latest for the first time (no images matching dependencies hash)
• feature-branch1: creates my_image:feature-branch1 based on my_image:latest (as the dependency hash matched)
• feature-branch2: creates my_image:feature-branch2 based on my_image:latest (as the dependency hash matched)
• feature-branch1: my_image:feature-branch1 is build and dependency hash finds it
• master: can still find my_image as there is a latest tag

2. What if master didn't rely on :latest tag, but instead used :master labels to find latest image?

Example:

• master: builds my_image:master for the first time (no images matching dependencies hash)
• feature-branch1: promotes my_image:latest to my_image:feature-branch1 (as the dependency hash matched)
• feature-branch2: cannot find a matching images for my_image (as feature-branch1 stole the latest tag), so it builds it again (the docker cache helps a lot here, so this is fairly fast)
• feature-branch1: my_image:feature-branch1 is build and dependency hash finds it
• master: uses my_image:latest (if dep. hash matches)

3. How does all of this play together with what we do in projects/deploy.sh > tag_and_push_docker_compose() where we tag all used images as :production? Is that only "locally" on instance-1 that we do it?

[skovhus]

I haven't observed the issue here since we upgraded our Docker version.