Closed DanceParty closed 2 years ago
@DanceParty I'm not currently able to reproduce this. It's possible it was related to a blip in the apt feed. Please retry.
I'd also suggest taking a look at the dev container CLI (or just the docker CLI) once you've gotten things set up so you can pre-build dependencies into your image to avoid this happening in the future.
Hmm, this has been happening since Friday so not sure there is a blip here. Very possible I could have done something wrong here but not sure :/
It's difficult to know without the entire Dockerfile. Try using the docker-in-docker definition to see if you experience an issue there. That's what I did to try to reproduce.
One possible cuplrit is it looks like you have a Debian 8/Jessie feed in your setup:
#17 0.357 Hit:3 http://repo.mysql.com/apt/debian jessie InRelease
In theory that could cause all sorts of problems since you're mixing distribution versions.
@Chuxel, is it possible that one of the mirrors of the https://packages.microsoft.com/repos/microsoft-debian-buster-prod/pool/main/m/moby-buildx/moby-buildx_0.7.0+azure-1_amd64.deb
package has inconsistent metadata around the expected hashes? Not sure why else that hash mismatch would occur. It seems like an issue with the package feed.
I stripped down my dockerfile as much as possible and still having the same issue:
Dockerfile in current form:
FROM ruby:3.0.2-buster
WORKDIR /workspace/project-name
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get clean && rm -rf /var/lib/apt/lists/* && apt-get update
# Docker in docker setup
COPY .devcontainer/library-scripts/*.sh /tmp/library-scripts/
ENV DOCKER_BUILDKIT=1
RUN apt-get update && /bin/bash /tmp/library-scripts/docker-in-docker-debian.sh
ENTRYPOINT ["/usr/local/share/docker-init.sh"]
VOLUME [ "/var/lib/docker" ]
CMD ["sleep", "infinity"]
Switching from ruby:3.0.2-buster
(even ruby:3.0.2
seems to use buster
by default) to ruby:3.0.2-bullseye
seemed to fix this issue which is strange that this past friday was the day it stopped working (it worked fine on buster before friday Nov 12)
@Chuxel, is it possible that one of the mirrors of the https://packages.microsoft.com/repos/microsoft-debian-buster-prod/pool/main/m/moby-buildx/moby-buildx_0.7.0+azure-1_amd64.deb package has inconsistent metadata around the expected hashes? Not sure why else that hash mismatch would occur. It seems like an issue with the package feed.
It would have to be regional if so. I can't repro this in Codespace or my local machine. @DanceParty where are you located? @jkeech @joshspicer, can you repro?
OK - I see it in WestUS2 specifically (not in EastUS where I'm at). Will shoot a note to the internal package feed team to see if this is known.
They will be re-pushing the package soon - hopefully that will resolve it!
@DanceParty This should be resolved now. Can you retry?
@Chuxel I have the same problem and it won't resolved yet I looked the packages dates and the last one is from November 11
@alangroszTP They were redeployed not rebuilt, so the update date won't change. It only occurred in certain geographies.
Where are you located? WestUS2 seems to be working, so it's possible something is up in another region. I'll ping the team and see if it was redeployed everywhere or just the west coast.
Also, try to rebuild without cache to make sure you aren't experiencing some sort of caching challenge.
Switched the main image back to ruby:3.0.1
which is what was originally failing for me and it worked fine!
@alangroszTP They were redeployed not rebuilt, so the update date won't change. It only occurred in certain geographies.
Where are you located? WestUS2 seems to be working, so it's possible something is up in another region. I'll ping the team and see if it was redeployed everywhere or just the west coast.
Also, try to rebuild without cache to make sure you aren't experiencing some sort of caching challenge.
@Chuxel I'm from Argentina
Still seeing this problem currently.
The Linux package team is looking into this right now. It seems like what happened is West US started working, but other regions then broke. The team is trying to determine why this is happening and remediate.
Near term, if you are using Codespaces, you can force your Codespace into the West US region. Go to https://github.com/settings/codespaces and you can specify choosing "select manually":
Okay - the package feed team found the problem and believes they've resolved it. Checked East US, West US, West Europe, and Southeast Asia all appear to look good now.
If you still see this error, let us know where you are located so we can relay that to the package team. (Also include the OS in your container not Debian 10/buster - run cat /etc/os-release
to find out.) Otherwise things should work now.
Closing as resolved. Let us know if you are still hitting issues.
@Chuxel
I'm still seeing this error:
#18 69.30 Get:4 https://packages.microsoft.com/repos/microsoft-debian-buster-prod buster/main arm64 moby-cli arm64 20.10.11+azure-1 [9213 kB]
#18 70.30 Get:5 https://packages.microsoft.com/repos/microsoft-debian-buster-prod buster/main arm64 moby-runc arm64 1.0.2+azure-1 [4879 kB]
#18 70.83 Get:6 https://packages.microsoft.com/repos/microsoft-debian-buster-prod buster/main arm64 moby-containerd arm64 1.4.12+azure-1 [19.3 MB]
#18 72.91 Get:7 https://packages.microsoft.com/repos/microsoft-debian-buster-prod buster/main arm64 moby-engine arm64 20.10.11+azure-1 [14.0 MB]
#18 74.43 Fetched 68.6 MB in 8s (8778 kB/s)
Failed to fetch https://packages.microsoft.com/repos/microsoft-debian-buster-prod/pool/main/m/moby-buildx/moby-buildx_0.7.0+azure-1_arm64.deb Hash Sum mismatch
#18 74.43 Hashes of expected file:
#18 74.43 - SHA512:e9d5b2a378cef5b407ebf4dadb72044a5c0846e05d1b02e73b912b18d84f103268f9d062c34ba5880cb5df8603a678b4b4be1da506f91a0a7965b30e12325385
#18 74.43 - SHA256:2931feca352c35f9bd0861800c21dfee40b4f012d38bf2c1825ab679bbec4cfc
#18 74.43 - SHA1:661b66a246058509167a3ce875c1e82beb35e642 [weak]
#18 74.43 - MD5Sum:1b1626c8ad558f2dc3b53ee76b8471c8 [weak]
#18 74.43 - Filesize:21084004 [weak]
#18 74.43 Hashes of received file:
#18 74.43 - SHA512:0d62dac0a3837a40ad3560a026b07cce3f985966f3fd1f239bce71c30ea7c8cbb6a78810ccdbfd4c574f31b79435b3f4e7e3e7264ff9a52fbfec093c5029e180
#18 74.43 - SHA256:e09924028e6ec11d9cfce0af80921c7b6ce973b66ed9a1a0cfe4fbf73813b573
#18 74.43 - SHA1:5fd7ef4417f99695825e4e00e091cc88d7a5549e [weak]
#18 74.43 - MD5Sum:181010a473b481e8f5630da08b240904 [weak]
#18 74.43 - Filesize:21084004 [weak]
#18 74.43 Last modification reported: Thu, 11 Nov 2021 17:43:42 +0000
#18 74.43 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
#18 ERROR: process "/bin/sh -c apt-get update && /bin/bash /tmp/library-scripts/docker-in-docker-debian.sh && apt-get clean -y" did not complete successfully: exit code: 100
I'm prebuilding this image in a self-hosted GitHub action runner in AWS eu-central-1.
@woehrl01 Is this happening for linux/amd64 as well or just arm64?
@Chuxel Just arm64, linux/amd64 is working fine.
Reopening to track arm64.
@woehrl01 One question for you here - did this originally work?
Also, are you using buildx to build the image on x86 hardware? I know that buster does not work well in emulated ARM64 because of an OS bug. We recommend anyone building arm64 images move to bullseye because of it (and our pre-built images are only x86_64 for this reason).
This isn't showing up like the typical segfault we'd see there, but Debian 10/buster and Ubuntu 20.04/focal do not work well on Apple Silicon macs because of this.
@Chuxel
I actually don't know. I'm currently migrating to arm64 and stumbled over that issue. Thank you for the hint, I'll try bullseye.
Yes, I'm using buildx to build on x86.
@Chuxel I just would like to confirm that it's working great with debian/bullseye
. Thank you.
Ok - this should be resolved now. As before, let us know the region you are located if you still experience this error.
@Chuxel -- This issue appears to have started again here this evening, was working earlier today but something is now out of sync.
CodeSpace Region - East US2 (Virginia) & West US2 (California)
VS Code (VPN Connect -- CentralUS Connected && London Connected)
Reproduce: Add new Development Container Configuration File Configuration Python 3 3.10-bullseye (default) -- lts/* (default) Feature - Docker (Moby) support (Docker-in-Docker) - latest version
#7 32.04 - Filesize:26966076 [weak]
#7 32.04 Last modification reported: Wed, 12 Jan 2022 19:11:38 +0000
#7 32.07 Get:7 https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye/main amd64 moby-engine amd64 20.10.11+azure-3 [21.2 MB]
#7 38.82 E: Failed to fetch https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/pool/main/m/moby-containerd/moby-containerd_1.4.12%2bazure-1_amd64.deb Hash Sum mismatch
#7 38.82 Hashes of expected file:
#7 38.82 - SHA512:7f0059afb6124d756508792fbdc972fdc1e6c8f178ace40d999acb4441d9e2a282a3e07798d3e622dfd5af309a3c1549dc7280ac42c58ea2bbe1748b271271b2
#7 38.82 - SHA256:b1aa4dc0b8116bcc98b7e9d5270f154a382549081bb4df1c20475c567813dd08
#7 38.82 - SHA1:4736318f2973caf3cc0263b03659a928225471fb [weak]
#7 38.82 - MD5Sum:7bbf6db138a768c15b23a73f971a89cf [weak]
#7 38.82 - Filesize:26966076 [weak]
#7 38.82 Hashes of received file:
#7 38.82 - SHA512:fab0f7992983f4cb633c7b77fa55cfb424d57d2ca79cd15458934461b5890a1a68226e7d822a51cbb5c395aaa6b5e8c49beab63ed1afb5faebe953e4b9da6949
#7 38.82 - SHA256:064e2676231cdb018c34936c41589ef8a53bf4059139ebbd250517a0cdfa4d02
#7 38.82 - SHA1:d7fb6bb3fc78caaa50a686bc607ebf86e8d59d57 [weak]
#7 38.82 - MD5Sum:65fce0295bd00e762e84351d89807828 [weak]
#7 38.82 - Filesize:26966076 [weak]
#7 38.82 Last modification reported: Wed, 12 Jan 2022 19:11:38 +0000
#7 38.82 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
#7 38.82 Fetched 87.2 MB in 26s (3326 kB/s)
We have the same problem, a workaround fix that worked for us is to add moby-containerd=1.4.11+azure-1
inside /library-scripts/docker-in-docker-debian.sh
file like this:
apt-get -y install --no-install-recommends moby-cli moby-containerd=1.4.11+azure-1 moby-buildx moby-engine
In addition, this problem happens only when using https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/dists/bullseye/. If you try replacing it to "buster", it works.
Confirmed. We've contacted the MS package feed team.
Fix is propagating. Things should start working soon if they are not already.
This should be resolved.
Steps to Reproduce:
docker-in-docker-debian
to my Dockerfiledocker build -t <name> -f <path> .
The above steps get me this error: