parallel build requests do not synchronize pushing identical layer blobs

[hholst80]

To me it looks like the push, pushes the same layers, but does the exact job twice and concurrently?

You see the arena01 and p10000 images below, they are being done concurrently. but it is the same layer. it does not make sense to send the same data twice because the registry will cache the layers between repositories.

can this redundant push be avoided, or do I have to build the two different images in sequence?

[tonistiigi]

Where's this output from? Is it from docker buildx bake?

[hholst80]

Yes box it is.

[hholst80]

Same problem it seems with pull. Two identical concurrent pulls of the same layer. When you're on a 3G connection, stuff like that really matters. When bugs like this are multiplied out over the total usage of docker build in the world right now, the impact becomes significant.

[tonistiigi]

@hholst80 Pulls should not have this issue.

[hholst80]

Edit: The video below did not capture this supposed observation so let's assume I got it wrong and it's something related to the push only.

I might be reading it wrong, but it sure looks like that.

https://user-images.githubusercontent.com/6200749/175700152-6b947cf6-b6da-4eea-be92-c440b9556d6a.mp4

https://user-images.githubusercontent.com/6200749/175700457-0e2662cd-7127-49df-b605-47cca7647b60.mov

Edit: I don't seem to be very successful in inlining the video that shows the thing I was looking at, but maybe if you open up the link directly it will work.

[hholst80]

I thought I saw the problem again so I digged into this a bit.

Docker engine uses by default a limit of 3 layer pulls per image. Is this a limit that is enforced globally for a compose build as well? It looks like it is pulling down 5 layers, but 2 layers are duplicates. So in total 3 unique layers are being pulled down . The async reporting makes it difficult to understand what is going on here because the update is slightly out of sync.

[hholst80]