Checksum Validation failure for moby-buildx

[paralllax]

Describe the issue

When sycning packages from the Focal amd64 repository, validation fails due to checksum mismatch.

A file located at the url https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd64.deb
failed validation due to checksum. Expected '0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03', 
Actual '707cfe4a1890e2e368e9b6b845c6d5359221287b2c4bb31873efc60a0ebbd15c'

When did the issue occur?

We do nightly repository syncs, this error started occurring ~3 days ago

2024-02-12T07:54:01.824593Z
A file located at the url https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd64.deb failed validation due to checksum. Expected '0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03', Actual '707cfe4a1890e2e368e9b6b845c6d5359221287b2c4bb31873efc60a0ebbd15c'

2024-02-11T06:56:24.987970Z
A file located at the url https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd64.deb failed validation due to checksum. Expected '0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03', Actual '707cfe4a1890e2e368e9b6b845c6d5359221287b2c4bb31873efc60a0ebbd15c'

2024-02-10T07:22:48.690632Z
A file located at the url https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd64.deb failed validation due to checksum. Expected '0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03', Actual '707cfe4a1890e2e368e9b6b845c6d5359221287b2c4bb31873efc60a0ebbd15c'

If applicable, what package did you attempt to install, and from which repo?

Steps to Reproduce You can manually verify this by pulling the package and getting the checksum, then comparing to the repository metadata

wget -O - https://packages.microsoft.com/ubuntu/20.04/prod/dists/focal/main/binary-amd64/Packages 2>/dev/null | grep -A12 --no-group-separator ' 0.12.1' | yq '.SHA256'
0d1c7ac216d2a7d825b6fb0dbdec0f81d88540f1cc074d77abf4c0e7988f3e03

wget -O - https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-ubuntu20.04u1_amd
64.deb 2>/dev/null | sha256sum
707cfe4a1890e2e368e9b6b845c6d5359221287b2c4bb31873efc60a0ebbd15c  -

Actual Result

The checksums do not match

Expected Result

The checksums should match

[mbearup]

@paralllax thanks for reporting this issue, it should now be resolved. For clarity, our content delivery cache retains packages indefinitely (since their contents should not generally change) but it looks like an updated version of this package (same name/version/arch, but different checksum) was published recently. We have work planned for the coming months to gracefully purge packages from cache when they are updated centrally.

[paralllax]

@mbearup, that was remarkably fast! Thank you for the fast turnaround and context/background information.

[sozercan]

@paralllax thanks for the report!

[offbyone]

@mbearup we're still experiencing this on our image pulls here on a different dist, but not in a way we can consistently reproduce:

$ wget -O- https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/(wget -O - https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/dists/bullseye/main/binary-amd64/Packages 2>/dev/null | ggrep -A15 --no-group-separator ' 0.12.1' | yq '.Filename') 2>/dev/null | sha256sum
72ba1ec05f3db690e87bb342e94c5bf20e413c1ef8541c1efb7cce562d1a70d4  -
$ wget -O- https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-debian11u1_amd64.deb | sha256sum
--2024-02-13 14:17:00--  https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-debian11u1_amd64.deb
Resolving packages.microsoft.com (packages.microsoft.com)... 40.118.250.56
Connecting to packages.microsoft.com (packages.microsoft.com)|40.118.250.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34140374 (33M) [application/octet-stream]
Saving to: ‘STDOUT’

-                                   100%[=================================================================>]  32.56M  9.54MB/s    in 3.6s

2024-02-13 14:17:04 (9.00 MB/s) - written to stdout [34140374/34140374]

0cc51f3c1b330a86a93488d87971b131be497e101ea4aee99dc7482fd0cc730c  -

This is from a successful one:

$ wget -O - https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/dists/bullseye/main/binary-amd64/Packages 2>/dev/null | ggrep -A15 --no-group-separator ' 0.12.1' 2>/dev/null | yq '.SHA256'
72ba1ec05f3db690e87bb342e94c5bf20e413c1ef8541c1efb7cce562d1a70d4
$ wget -O- https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-debian11u1_amd64.deb | sha256sum
--2024-02-13 14:24:42--  https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/pool/main/m/moby-buildx/moby-buildx_0.12.1-debian11u1_amd64.deb
Resolving packages.microsoft.com (packages.microsoft.com)... 104.42.185.173
Connecting to packages.microsoft.com (packages.microsoft.com)|104.42.185.173|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34084902 (33M) [application/octet-stream]
Saving to: ‘STDOUT’

-                                   100%[=================================================================>]  32.51M  9.20MB/s    in 4.1s

2024-02-13 14:24:47 (7.86 MB/s) - written to stdout [34084902/34084902]

72ba1ec05f3db690e87bb342e94c5bf20e413c1ef8541c1efb7cce562d1a70d4  -

So, we are definitely seeing the correct artifact, but not every time.

[jasonzio]

What IP address(es) are you seeing when you resolve packages.microsoft.com? I suspect you're hitting different mirrors and one of them didn't handle the purge request properly.

[offbyone]

That's in the output I included 😁 40.118.250.56 is the one that has an invalid checksum

[jasonzio]

Thanks. We re-ran the purge and I've confirmed the packages pulled from those two mirrors are identical and they have the correct checksum as shown in the repo metadata.

We're not sure how this one particular mirror didn't correctly execute the purge operation we performed yesterday, but that's what happened.

Thanks for the detailed bug report - made it really easy to smack this one down.