devcontainer build fails with authentication error

[fasmat]

Hi everyone,

I've been using the build feature of the devcontainer CLI before and didn't have any issues until recently.

Now when I try to build my devcontainer configuration I receive the following error:

Building app
[+] Building 1.4s (5/5) FINISHED                                                
 => [internal] load build definition from Dockerfile-with-features         0.0s
 => => transferring dockerfile: 3.85kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/dev_container_fe  1.3s
 => [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/b  0.3s
 => [auth] library/dev_container_feature_content_temp:pull token for regi  0.0s
------
 > [internal] load metadata for docker.io/library/dev_container_feature_content_temp:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
ERROR: Service 'app' failed to build : Build failed
[4407 ms] Error: Command failed: docker-compose --project-name workspace_devcontainer -f /workspace/.devcontainer/docker-compose.yml -f /tmp/docker-compose/docker-compose.devcontainer.build-1655998380158.yml build --no-cache --pull app
[4407 ms]     at buildAndExtendDockerCompose (/usr/local/share/nvm/versions/node/v16.15.1/lib/node_modules/@vscode/dev-container-cli/dist/spec-node/devContainersSpecCLI.js:19967:51)
[4407 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[4407 ms]     at async doBuild (/usr/local/share/nvm/versions/node/v16.15.1/lib/node_modules/@vscode/dev-container-cli/dist/spec-node/devContainersSpecCLI.js:21358:7)
[4407 ms]     at async build (/usr/local/share/nvm/versions/node/v16.15.1/lib/node_modules/@vscode/dev-container-cli/dist/spec-node/devContainersSpecCLI.js:21269:18)
[4415 ms] Exit code 1

I'm logged into Docker hub and can pull other images with no problem. Any ideas what could be the issue here?

[fvelcker]

Similar issue here. It was fine last month (I rebuild the image every month) I build the container in Gitlab CI with something like: devcontainer build --no-cache --image-name "$IMAGE_NAME" "$DEVCONTAINER_PATH". And get this at some point:

[...]
Removing intermediate container 74f20fc549df
 ---> ff2862a8bcfd
Step 15/22 : FROM dev_container_feature_content_temp as dev_containers_feature_content_source
time="2022-06-25T12:15:51.053634117Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
time="2022-06-25T12:15:51.053748136Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
pull access denied for dev_container_feature_content_temp, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
[833340 ms] Error: Command failed: docker build -f /tmp/vsch/container-features/0.240.0-1656158518176/Dockerfile-with-features -t vsc-php7.4-ebd414c7648d77d06b3bbc734dc8ffc6 --no-cache --pull --build-arg VARIANT=7.4 --build-arg NODE_VERSION=lts/* --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /builds/Reassured-devops/docker/devcontainers
[833341 ms]     at buildAndExtendImage (/usr/local/lib/node_modules/@vscode/dev-container-cli/dist/spec-node/devContainersSpecCLI.js:20711:11)
[833342 ms]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[833342 ms]     at async buildNamedImageAndExtend (/usr/local/lib/node_modules/@vscode/dev-container-cli/dist/spec-node/devContainersSpecCLI.js:20620:12)
[833342 ms]     at async doBuild (/usr/local/lib/node_modules/@vscode/dev-container-cli/dist/spec-node/devContainersSpecCLI.js:21335:36)
[833342 ms]     at async build (/usr/local/lib/node_modules/@vscode/dev-container-cli/dist/spec-node/devContainersSpecCLI.js:21269:18)
[833353 ms] Exit code 1

[bamurtaugh]

Thanks for opening and for the feedback @fasmat and @fvelcker. It appears you're both using the @vscode/dev-container-cli.

I recently commented in https://github.com/devcontainers/cli/issues/66 about using @devcontainers/cli vs @vscode/dev-container-cli, and general guidance is:

You should use @devcontainers/cli, which is the CLI from this repo. It serves as the reference implementation for the dev container specification. @vscode/dev-container-cli is an older "Remote-Containers CLI" that could be installed within the Remote-Containers extension or through the command line.

We're currently tracking renaming the older Remote-Containers CLI (issue), and replacing the Remote-Containers CLI with the newer one in current tooling (issue).

Would you be able to try a devcontainer build using the CLI from this repo, @devcontainers/cli?

cc @chrmarti @stuartleeks

[fasmat]

Hi @bamurtaugh,

I tried your suggestion:

vscode ➜ /workspace $ npm i -g @devcontainers/cli
vscode ➜ /workspace $ devcontainer build --workspace-folder . --image-name local-env:latest --no-cache
[13 ms] @devcontainers/cli 0.6.0.
[1004 ms] Start: Run: docker-compose -f /workspace/.devcontainer/docker-compose.yml config
[1449 ms] services:
  app:
    build:
      context: /workspace/.devcontainer
      dockerfile: Dockerfile
    command: sleep infinity
    entrypoint: /usr/local/share/docker-init.sh
    user: vscode
    volumes:
    - ~/.ssh:/home/vscode/.ssh:ro
    - /var/run/docker.sock:/var/run/docker-host.sock:rw
    - ..:/workspace:cached
version: '3'

[1461 ms] Preparing to parse declared features and fetch remote features.
[1461 ms] Detected local feature set. Continuing...
[1461 ms] Detected local feature set. Continuing...
[1461 ms] Detected local feature set. Continuing...
[1461 ms] Detected local feature set. Continuing...
[1461 ms] Detected local feature set. Continuing...
[1462 ms] Detected local feature set. Continuing...
[1462 ms] Detected local feature set. Continuing...
[1467 ms] local container features stored at: /usr/local/share/nvm/versions/node/v16.15.1/lib/node_modules/@devcontainers/cli/dist/node_modules/vscode-dev-containers/container-features
[1468 ms] Start: Run: tar --no-same-owner -x -f -
[1529 ms] Start: Run: docker build -t dev_container_feature_content_temp -f /tmp/vsch/container-features/0.6.0-1656355510590/Dockerfile.buildContent /tmp/vsch/container-features/0.6.0-1656355510590
[+] Building 0.1s (5/5) FINISHED                                                
 => [internal] load build definition from Dockerfile.buildContent          0.0s
 => => transferring dockerfile: 101B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 270.75kB                                      0.0s
 => CACHED [1/1] COPY . /tmp/build-features/                               0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:4eef675b07d4033f3f26ad2437468d3e2070577593675  0.0s
 => => naming to docker.io/library/dev_container_feature_content_temp      0.0s
[1938 ms] Start: Run: docker-compose --project-name workspace_devcontainer -f /workspace/.devcontainer/docker-compose.yml -f /tmp/docker-compose/docker-compose.devcontainer.build-1656355511076.yml build --no-cache --pull app
Building app
[+] Building 1.4s (4/4) FINISHED                                                
 => [internal] load build definition from Dockerfile-with-features         0.0s
 => => transferring dockerfile: 3.85kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/dev_container_fe  1.3s
 => [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/b  0.2s
------
 > [internal] load metadata for docker.io/library/dev_container_feature_content_temp:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
ERROR: Service 'app' failed to build : Build failed
Error: Command failed: docker-compose --project-name workspace_devcontainer -f /workspace/.devcontainer/docker-compose.yml -f /tmp/docker-compose/docker-compose.devcontainer.build-1656355511076.yml build --no-cache --pull app
    at buildAndExtendDockerCompose (/usr/local/share/nvm/versions/node/v16.15.1/lib/node_modules/@devcontainers/cli/dist/spec-node/dockerCompose.js:259:66)
    at async doBuild (/usr/local/share/nvm/versions/node/v16.15.1/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:311:13)
    at async build (/usr/local/share/nvm/versions/node/v16.15.1/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:226:20)
{"outcome":"error","message":"Command failed: docker-compose --project-name workspace_devcontainer -f /workspace/.devcontainer/docker-compose.yml -f /tmp/docker-compose/docker-compose.devcontainer.build-1656355511076.yml build --no-cache --pull app","description":"An error occurred building the Docker Compose images."}

I'm logged into Docker Hub and can pull images from there, but devcontainer keeps complaining it cannot authorize.

As mentioned by @fvelcker this problem only occurred recently and the same command would not error a few weeks ago.

[fvelcker]

@bamurtaugh Yes I did that before I raised the other question about what package to use. At first I thought it was working but then, I had a similar error about the access denied.

[979505 ms] Removing intermediate container 368b38d045a6
 ---> bdecdf562737
Step 15/22 : FROM dev_container_feature_content_temp as dev_containers_feature_content_source
time="2022-06-27T15:37:56.879823879Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
time="2022-06-27T15:37:56.879973550Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
[981072 ms] pull access denied for dev_container_feature_content_temp, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
[981078 ms] Exit code 1
Error: Command failed: docker build -f /tmp/vsch/container-features/0.6.0-1656343295957/Dockerfile-with-features -t vsc-php7.4-ebd414c7648d77d06b3bbc734dc8ffc6 --no-cache --pull --build-arg VARIANT=7.4 --build-arg NODE_VERSION=lts/* --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /builds/Reassured-devops/docker/devcontainers
    at buildAndExtendImage (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-node/singleContainer.js:199:15)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async buildNamedImageAndExtend (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-node/singleContainer.js:97:16)
    at async doBuild (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:285:42)
    at async build (/usr/local/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:226:20)
{"outcome":"error","message":"Command failed: docker build -f /tmp/vsch/container-features/0.6.0-1656343295957/Dockerfile-with-features -t vsc-php7.4-ebd414c7648d77d06b3bbc734dc8ffc6 --no-cache --pull --build-arg VARIANT=7.4 --build-arg NODE_VERSION=lts/* --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp /builds/Reassured-devops/docker/devcontainers","description":"An error occurred building the image."}

Edit: Retried with v0.7.0 and same (I was hoping that PR https://github.com/devcontainers/cli/pull/64 would get this fixed)

[fvelcker]

Found this: https://github.com/microsoft/vscode-remote-release/issues/6851#issuecomment-1163303463. I tried to remove the features from the devcontainer.json, and, indeed, the build passed.

[chrmarti]

I see pull access denied for dev_container_feature_content_temp. I guess it might be missing this temporary image in the local image cache. We build locally when needed, but maybe there is a case where this is missing.

[chrmarti]

You could install BuildKit for your Docker install as a workaround. (If you already have BuildKit installed, let me know which version.)

docker build --pull tries to pull the local image, docker buildx build --pull avoids that. Investigating a fix.

[fvelcker]

Thanks for looking into it. I'll try BuildKit then

[chrmarti]

Actually the workaround of using BuildKit will not work for Docker Compose at the moment (that won't switch to using buildx). Preparing a fix that will simply skip --pull when we use that temporary image with docker build.

Another workaround is to not use --no-cache (--pull is only added when that is set).

[fvelcker]

ah, well, good to know, I ended up with some issue with BuildKit, so I won't get further into that :)

I am actually already using --no-cache. I am building the image in Gitlab CI, and use that command

devcontainer build --no-cache --workspace-folder "$DEVCONTAINER_PATH" --image-name "$IMAGE_NAME" "$DEVCONTAINER_PATH" 

[chrmarti]

Fixed in Remote-Containers 0.241.2-pre-release and Dev Containers CLI 0.7.1.

(The workaround would be to not use --no-cache.)

[fvelcker]

Fixed in Remote-Containers 0.241.2-pre-release and Dev Containers CLI 0.7.1.

(The workaround would be to not use --no-cache.)

Ah, I missed that keyword!

Thanks for the fix, I'll try it now, would be a nice finish for the week

[fvelcker]

That works perfectly now, thanks a lot

[fasmat]

Fixed in Remote-Containers 0.241.2-pre-release and Dev Containers CLI 0.7.1.

(The workaround would be to not use --no-cache.)

Just tested CLI version 0.7.1. Works again, thanks for fixing! ❤️