Cache misses from differing commit hash leading to unnecessary builds

[mthalman]

There are a few Dockerfiles which are not being recognized as a cache hit to an already published version of the image.

From the build log (internal link):

Image info's Dockerfile commit: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/507c1c3291de8f64d9592f72ae92f8f9f0b45353/src/ubuntu/22.04/coredeps/Dockerfile
Latest Dockerfile commit: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/70a3638e31ff3f7f69a9373b0b4c06773f5eaee6/src/ubuntu/22.04/coredeps/Dockerfile
Dockerfile commits match: False
CACHE MISS

Image info's Dockerfile commit: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/3fc5553956a6e28906d81758fc2852c6a1fd93cc/src/ubuntu/18.04/crossdeps/Dockerfile
Latest Dockerfile commit: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/70a3638e31ff3f7f69a9373b0b4c06773f5eaee6/src/ubuntu/18.04/crossdeps/Dockerfile
Dockerfile commits match: False
CACHE MISS

Image info's Dockerfile commit: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/507c1c3291de8f64d9592f72ae92f8f9f0b45353/src/ubuntu/22.04/coredeps/Dockerfile
Latest Dockerfile commit: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/70a3638e31ff3f7f69a9373b0b4c06773f5eaee6/src/ubuntu/22.04/coredeps/Dockerfile
Dockerfile commits match: False
CACHE MISS

These are being detected as being different commit hashes than what Dockerfile was last used to publish. But that's false because the different commit hashes are actually pointing to the same version of the Dockerfile. For example, with that first one (https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/70a3638e31ff3f7f69a9373b0b4c06773f5eaee6/src/ubuntu/22.04/coredeps/Dockerfile), if you go to the URL in your browser, you'll see the webpage refer to that version using the short hash of the equivalent hash:

It's not clear why there are differences in the values, nor why the publishing process doesn't update the image info JSON content with the value that we're detecting the file to have.

[lbussell]

[Triage] My suspicion is that something is up with this command: https://github.com/dotnet/docker-tools/blob/8fa03e26afbf7676865f2335c0b2f78baf5dbfe9/src/Microsoft.DotNet.ImageBuilder/src/GitHelper.cs#L43

Something must be causing it to retrieve the wrong commit. In the commit it returned, 70a3638e31ff3f7f69a9373b0b4c06773f5eaee6, there were no changes to the coredeps Dockerfile.

[lbussell]

Transferring to Docker tools, seems like a bug in ImageBuilder...

Looks to me like ImageBuilder is always updating the file with the latest commit from the repo, not the commit referring to the last time that the Dockerfile in question was touched. It can be confusing because when you click on the link, GitHub tells you which commit refers to the last time the Dockerfile was edited (at the top of the page - circled in Matt's image). But in reality you are viewing the file tree from the latest commit - in this case 70a3638...

[lbussell]

git log -1 --format=format:%{format} {filePath} Something must be causing it to retrieve the wrong commit.

I wonder if this could be due to the pipeline shallow-cloning the repo, if it does that.

[lbussell]

[Triage] This seems to cause CBL Mariner images to be rebuilt more frequently then they need to. There are some large builds in that dependency graph. Given that this could be a relatively simple fix, we should definitely invest in the initial investigation here.

[lbussell]

[Triage] Related: This might cause issues if we need to move back to a single pipeline due to this issue: https://github.com/dotnet/docker-tools/issues/1417

[mthalman]

Confirmed this was caused by a shallow fetch of the git repo. This was configured in the AzDO UI for several pipelines. I've set them all to not do any shallow fetch.

[lbussell]

Thanks for investigating this @mthalman!