Open ca-mtg opened 2 years ago
chrmarti
commented
2 years ago This appears to be a known issue with BuildKit: https://github.com/moby/buildkit/issues/971
We could add a setting to disable the use of BuildKit, though long-term we might not be able to support all features without using BuildKit.
ca-mtg
commented
2 years ago I think there is probably a legitimate use case for some people to want to disable buildkit for things like this, even if they need to accept a more limited functionality when doing so.
Strangely, I built a clean VM and it works fine on there. All three machines (my machine, the new dev's machine, and my clean VM) are using buildkit v0.8.2 according to the docker buildx version command.
My old machine is using docker-ce v19.03.1, my new "clean" VM is using docker-ce v20.10.17, and the new dev's machine is using docker-ce v20.10.16, all according to the docker version command.
All three are using the Remote-Containers version 0.241.3 in VS Code version 1.68.1
Here are my logs in case it's actually a different issue than the proxy. From what I can tell, it's failing to pull my base image from my local registry (which is in the no_proxy variable), even though the error message is referring to the mybuilder image rather than my local image. It seems to me like it's ignoring my no_proxy variable and trying to access my local registry through the proxy when it should be skipping the proxy for this request.
Here's my error log, redacted as needed:
[2022-07-11T16:14:46.778Z] Remote-Containers 0.241.3 in VS Code 1.68.1 (30d9c6cd9483b2cc586687151bcbcd635f373630).
[2022-07-11T16:14:46.778Z] Start: Resolving Remote
[2022-07-11T16:14:46.803Z] Setting up container for folder or workspace: <WORKSPACEFOLDER>
[2022-07-11T16:14:46.823Z] Start: Check Docker is running
[2022-07-11T16:14:46.823Z] Start: Run: docker version --format {{.Server.APIVersion}}
[2022-07-11T16:14:46.910Z] Stop (87 ms): Run: docker version --format {{.Server.APIVersion}}
[2022-07-11T16:14:46.911Z] Server API version: 1.40
[2022-07-11T16:14:46.911Z] Stop (88 ms): Check Docker is running
[2022-07-11T16:14:46.912Z] Start: Run: docker volume ls -q
[2022-07-11T16:14:46.992Z] Stop (80 ms): Run: docker volume ls -q
[2022-07-11T16:14:46.999Z] Start: Run: docker ps -q -a --filter label=vsch.local.folder=<WORKSPACEFOLDER> --filter label=vsch.quality=stable
[2022-07-11T16:14:47.065Z] Stop (66 ms): Run: docker ps -q -a --filter label=vsch.local.folder=<WORKSPACEFOLDER> --filter label=vsch.quality=stable
[2022-07-11T16:14:47.066Z] Start: Run: /usr/share/code/code /home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /home/<USERNAME>/.config/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder <WORKSPACEFOLDER> --workspace-mount-consistency cached --id-label devcontainer.local_folder=<WORKSPACEFOLDER> --log-level debug --log-format json --config /Path/To/Code/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-07-11T16:14:47.246Z] remote-containers 0.241.3.
[2022-07-11T16:14:47.246Z] Start: Run: docker buildx version
[2022-07-11T16:14:47.397Z] Stop (151 ms): Run: docker buildx version
[2022-07-11T16:14:47.397Z] Start: Resolving Remote
[2022-07-11T16:14:47.404Z] Start: Run: git rev-parse --show-cdup
[2022-07-11T16:14:47.409Z] Stop (5 ms): Run: git rev-parse --show-cdup
[2022-07-11T16:14:47.410Z] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=<WORKSPACEFOLDER>
[2022-07-11T16:14:47.470Z] Stop (60 ms): Run: docker ps -q -a --filter label=devcontainer.local_folder=<WORKSPACEFOLDER>
[2022-07-11T16:14:47.476Z] Start: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /Path/To/Code/.devcontainer/Dockerfile -t <tag> /Path/To/Code/.devcontainer
[2022-07-11T16:14:47.651Z] [+] Building 0.0s (0/1)
=> [internal] booting buildkit 0.0s
[2022-07-11T16:14:47.801Z] [+] Building 0.2s (0/1)
=> [internal] booting buildkit 0.2s
=> => pulling image mybuilder 0.2s
[2022-07-11T16:14:47.958Z] [+] Building 0.3s (0/1)
=> [internal] booting buildkit 0.3s
=> => pulling image mybuilder 0.3s
[2022-07-11T16:14:48.105Z] [+] Building 0.5s (0/1)
=> [internal] booting buildkit 0.5s
=> => pulling image mybuilder 0.5s
[2022-07-11T16:14:48.255Z] [+] Building 0.6s (0/1)
=> [internal] booting buildkit 0.6s
=> => pulling image mybuilder 0.6s
[2022-07-11T16:14:48.406Z] [+] Building 0.8s (0/1)
=> [internal] booting buildkit 0.8s
=> => pulling image mybuilder 0.8s
[2022-07-11T16:14:48.485Z] [+] Building 0.8s (1/1) FINISHED
=> ERROR [internal] booting buildkit 0.8s
=> => pulling image mybuilder 0.8s
------
> [internal] booting buildkit:
------
error: Error response from daemon: pull access denied for mybuilder, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
[2022-07-11T16:14:48.494Z] Stop (1018 ms): Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /Path/To/Code/.devcontainer/Dockerfile -t <tag> /Path/To/Code/.devcontainer
[2022-07-11T16:14:48.496Z] Error: Command failed: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /Path/To/Code/.devcontainer/Dockerfile -t <tag> /Path/To/Code/.devcontainer
[2022-07-11T16:14:48.497Z] at GR (/home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:240:1436)
[2022-07-11T16:14:48.497Z] at processTicksAndRejections (node:internal/process/task_queues:96:5)
[2022-07-11T16:14:48.497Z] at async Hu (/home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:239:2796)
[2022-07-11T16:14:48.497Z] at async tb (/home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:239:1779)
[2022-07-11T16:14:48.497Z] at async QR (/home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:2006)
[2022-07-11T16:14:48.497Z] at async cb (/home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:3110)
[2022-07-11T16:14:48.497Z] at async i_ (/home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:13194)
[2022-07-11T16:14:48.497Z] at async n_ (/home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js:245:12950)
[2022-07-11T16:14:48.503Z] Stop (1437 ms): Run: /usr/share/code/code /home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /home/<USERNAME>/.config/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder <WORKSPACEFOLDER> --workspace-mount-consistency cached --id-label devcontainer.local_folder=<WORKSPACEFOLDER> --log-level debug --log-format json --config /Path/To/Code/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-07-11T16:14:48.504Z] Exit code 1
[2022-07-11T16:14:48.505Z] Command failed: /usr/share/code/code /home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /home/<USERNAME>/.config/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder <WORKSPACEFOLDER> --workspace-mount-consistency cached --id-label devcontainer.local_folder=<WORKSPACEFOLDER> --log-level debug --log-format json --config /Path/To/Code/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2022-07-11T16:14:48.506Z] Exit code 1And here's my log from the successful build, redacted and truncated as needed:
[30 ms] Remote-Containers 0.241.3 in VS Code 1.68.1 (30d9c6cd9483b2cc586687151bcbcd635f373630).
[29 ms] Start: Resolving Remote
[37 ms] Setting up container for folder or workspace: <WORKSPACEFOLDER>
[48 ms] Start: Check Docker is running
[48 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[109 ms] Server API version: 1.41
[109 ms] Start: Run: docker volume ls -q
[164 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=<WORKSPACEFOLDER> --filter label=vsch.quality=stable
[198 ms] Start: Run: /usr/share/code/code /home/<USERNAME>/.vscode/extensions/ms-vscode-remote.remote-containers-0.241.3/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /home/<USERNAME>/.config/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder <WORKSPACEFOLDER> --workspace-mount-consistency cached --id-label devcontainer.local_folder=<WORKSPACEFOLDER> --log-level debug --log-format json --config /Path/To/Code/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[409 ms] remote-containers 0.241.3.
[409 ms] Start: Run: docker buildx version
[527 ms] Start: Resolving Remote
[530 ms] Start: Run: git rev-parse --show-cdup
[536 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=<WORKSPACEFOLDER>
[577 ms] Start: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /Path/To/Code/.devcontainer/Dockerfile -t <vsc-tag> /Path/To/Code/.devcontainer
[+] Building 127.7s (20/20) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.15kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for <localregistry>:<port>/<image1> 0.0s
=> [internal] load metadata for <localregistry>:<port>/<image2> 0.0s
=> [stage-1 1/12] FROM <localregistry>:<port>/<image1> 66.5s
=> => resolve <localregistry>:<port>/<image1>@sha256 0.0s
... <Successful docker pull of image1> ...
=> [internal] load build context 0.0s
=> => transferring context: 1.86kB 0.0s
=> [stage1 1/1] FROM <localregistry>:<port>/<image2>@sha256 67.9s
=> => resolve <localregistry>:<port>/<image2>@sha256 0.0s
... <Successful docker pull of image1> ...
... <Successful docker build of devcontainer> ...
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:8eb08911bef22069aaed1e65344a39b8d021f1727ac19 0.0s
=> => naming to docker.io/library/<vsc-tag> 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export 0.0s
[128497 ms] Start: Run: docker inspect --type image <vsc-tag>
[128597 ms] Start: Run: docker build -f <TMPDIR>/updateUID.Dockerfile-0.241.3 -t <vsc-tag>-uid --build-arg BASE_IMAGE=<vsc-tag> --build-arg REMOTE_USER=<RUSER> --build-arg NEW_UID=<UID> --build-arg NEW_GID=<GID> --build-arg IMAGE_USER=<RUSER> <TMPDIR>Unrelated sidenote: The log above that says naming to docker.io/library/<vsc-tag> scared me for a second - I thought I was accidentally pushing things to docker hub.
crairs
commented
1 year ago We are a big Microsoft customer and RedHat Linux customer. We supplanted use of Docker with Podman on all of our servers years ago. We wish to make use of internal remote devcontainer servers but forcing the use of BuildKit may be a killer of this plan.
chrmarti
commented
1 year ago @crairs We detect if BuildKit is available or not and do work without it.
crairs
commented
1 year ago @chrmarti I was concerned because we get an error "[15397 ms] Error: unrecognized command podman buildx version". I am assuming that if I was using docker the error would state "Error: unrecognized command docker buildx version". It seems a warning or info level message would be more appropriate if you fallback to docker or podman if BuildKit doesn't exist.
chrmarti
commented
1 year ago The string "Error" comes from Podman. I kept the command output unchanged to have all the available information in the log.
krisfoster
commented
4 months ago I have a similar issue in that I would like to stable buildit support within a devcontianer b build. Is there anyway that it can be disabled through the dev containers.json file? Thanks
TLDR: Looking for a way to disable buildx for devcontainer builds.
We're using the remote-containers plugin extensively here. I'm behind a notoriously difficult corporate proxy. It took a lot of work to get it to work properly behind our proxy while also being able to ignore the proxy to pull images / push images to/from our local registry. For about 2 years or so my devcontainers setup worked well.
We have multiple new devs starting, and as they start they end up with newer versions of docker than the one we've been using for 2 years (used to use 19.03.x, new devs are downloading newest version available which I think is 20.10.x. The new versions come with buildx by default. I also manually installed buildx yesterday for some multi-arch testing that I was doing.
I noticed today that the new devs as well as myself were unable to build/launch devcontainers. The errors seem to be a combination of permissions errors and known proxy-related errors (buildkit seems to ignore docker's proxy settings, another issue altogether...). I saw the buildx errors and thought it had something to do with a new VSCode update, considering it affected us all at once. After spending most of the day today troubleshooting this between meetings, I noticed that I can disable buildx (by taking the executable permissions off of the docker-buildx file in the libraries directory) and functionality goes back to normal.
I tried everything I can think of to soft-disable buildx for the devcontainers, but there just doesn't seem to be any way to do this. I tried setting DOCKER_BUILDKIT=0 in every possible way that the devcontainers allow you to change environment variables, and none of those had an affect (as they're all applied AFTER the docker build), I tried adding DOCKER_BUILDKIT to my bashrc, which was ignored, I tried running "docker buildx uninstall" to set the builder default back to docker, but still nothing. It seems like the devcontainers might be forcing buildkit whenever its installed.
I could use the workaround I mentioned above to disable buildx altogether, but after the aforementioned testing yesterday I'd actually like to use it for multi-arch containers so I'd rather only disable it for devcontainers if possible.
So my question is: Does anybody know a decent way to disable buildx/buildkit for devcontainer builds only, or alternatively is there a way that anybody knows of to fix proxy settings for buildkit? I was able to get buildx working through thte proxy yesterday but it involved making a custom buildkit image and passing that to the buildx create command, which doesn't really seem to be an option for devcontainer builds.
Thanks in advance!!
Environment: CentOS 7.9, 16 GB RAM, 4 CPU Cores