Dev Container 0.255.2 : Compose: Rebuild failed

leo-ossais commented 2 years ago

services: web: build: dockerfile: /var/folders/15/0k4w8b016wndrjhq99frm1wr0000gn/T/devcontainercli/container-features/0.19.0-1665488528250/Dockerfile-with-features target: dev_containers_target_stage args:

BUILDKIT_INLINE_CACHE=1
_DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label
_DEV_CONTAINERS_IMAGE_USER=root
_DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp

[3643 ms] Start: Run: docker-compose --project-name mamita -f /Users/leonardocosta/Repos/bitbucket/mamita/docker-compose.yml -f /Users/leonardocosta/Repos/bitbucket/mamita/.devcontainer/docker-compose.yml -f /Users/leonardocosta/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data/docker-compose/docker-compose.devcontainer.build-1665488528804.yml build [+] Building 1.3s (10/10) FINISHED
=> [mamita-web internal] load build definition from Dockerfile-with-feat 0.0s => => transferring dockerfile: 1.53kB 0.0s => [mamita-db internal] load build definition from Dockerfile-db 0.0s => => transferring dockerfile: 34B 0.0s => [mamita-web internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [mamita-db internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => ERROR [mamita-web internal] load metadata for docker.io/library/dev_c 1.1s => CANCELED [mamita-web internal] load metadata for docker.io/library/py 1.1s => CANCELED [mamita-db internal] load metadata for docker.io/library/pos 1.1s => [auth] library/postgres:pull token for registry-1.docker.io 0.0s => [auth] library/python:pull token for registry-1.docker.io 0.0s => [auth] library/dev_container_feature_content_temp:pull token for regi 0.0s

[mamita-web internal] load metadata for docker.io/library/dev_container_feature_content_temp:latest:

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed [5409 ms] Error: Command failed: docker-compose --project-name mamita -f /Users/leonardocosta/Repos/bitbucket/mamita/docker-compose.yml -f /Users/leonardocosta/Repos/bitbucket/mamita/.devcontainer/docker-compose.yml -f /Users/leonardocosta/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data/docker-compose/docker-compose.devcontainer.build-1665488528804.yml build [5410 ms] at pd (/Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:259:426) [5410 ms] at process.processTicksAndRejections (node:internal/process/task_queues:96:5) [5410 ms] at async ZF (/Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:259:2424) [5410 ms] at async XF (/Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:241:2391) [5410 ms] at async bD (/Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:303:2193) [5410 ms] at async ys (/Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:303:3182) [5410 ms] at async UL (/Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:423:10319) [5410 ms] at async ML (/Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js:423:10075) [5413 ms] Exit code 1 [5418 ms] Command failed: /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --ms-enable-electron-run-as-node /Users/leonardocosta/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /Users/leonardocosta/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-containers/data --workspace-folder /Users/leonardocosta/Repos/bitbucket/mamita --workspace-mount-consistency cached --id-label devcontainer.local_folder=/Users/leonardocosta/Repos/bitbucket/mamita --log-level debug --log-format json --config /Users/leonardocosta/Repos/bitbucket/mamita/.devcontainer/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true [5418 ms] Exit code 1

afischer-opentext-com commented 2 years ago

I confirm this issue with setups using docker-compose. (observed on OpenSuse Tumbleweed using release 0.255.2 and vscode 1.72.1.) In my case opening the dev container fails with the cryptic popup "An error occurred setting up the container" and no further hints are provided. Checking the output of the Log (Window) it says

[2022-10-11 18:44:32.371] [renderer1] [error] Command failed: /usr/share/code/code --ms-enable-electron-run-as-node /home/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /home/user/project --log-level debug --log-format json --config /home/user/project/.devcontainer/devcontainer.json --include-merged-configuration --mount-workspace-git-root true: Error: Command failed: /usr/share/code/code --ms-enable-electron-run-as-node /home/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /home/afischer/project --log-level debug --log-format json --config /home/user/project/.devcontainer/devcontainer.json --include-merged-configuration --mount-workspace-git-root true
    at n._executeContributedCommand (/usr/share/code/resources/app/out/vs/workbench/api/node/extensionHostProcess.js:90:111510)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)

afischer-opentext-com commented 2 years ago

Dupe of #7305 as it looks. Going back to 0.251.0 workarounds the issue for me.

chrmarti commented 2 years ago

What do you get for docker buildx ls? If that doesn't show default as the active builder try running docker buildx use default to switch to that and retry.

Continuing in https://github.com/microsoft/vscode-remote-release/issues/7305.

afischer-opentext-com commented 2 years ago

For completeness.

user@host:~> docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'

This is because the docker package is installed, but the docker-buildx package is not. Is this a new requirement?

chrmarti commented 2 years ago

@afischer-opentext-com Could you append the full log from when this fails? (F1 > Dev Containers: Show Container Log)

afischer-opentext-com commented 2 years ago

Here you go:

[29179 ms] Dev Containers 0.255.2 in VS Code 1.72.1 (129500ee4c8ab7263461ffe327268ba56b9f210d).
[29179 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[29208 ms] 1.41
[29216 ms] Start: Run: /usr/share/code/code --ms-enable-electron-run-as-node /home/user/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.2/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /home/user/project --log-level debug --log-format json --config /home/user/project/.devcontainer/devcontainer.json --include-merged-configuration --mount-workspace-git-root true --terminal-columns 80 --terminal-rows 30
[29391 ms] @devcontainers/cli 0.19.0. Node.js v16.14.2. linux 5.17.2-1-default x64.
[29391 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/home/user/project
[29425 ms] Start: Run: docker-compose version --short
[29485 ms] Docker Compose version: 2.11.2
[29485 ms] Start: Run: docker-compose -f /home/user/project/.devcontainer/docker-compose.yml --profile * config
[29548 ms] name: devcontainer
services:
  localstack:
    environment:
      FORCE_NONINTERACTIVE: "1"
      LAMBDA_EXECUTOR: local
      PORT_WEB_UI: "8081"
      SERVICES: codecommit,dynamodb,ec2,ecr,ecs,elbv2,iam,kms,lambda,logs,rds,route53,s3,secretsmanager,sns
    hostname: localstack
    image: artifactory.otxlab.net/dockerhub/localstack/localstack:1.2.0
    networks:
      default: null
  orch:
    build:
      context: /home/user/project
      dockerfile: docker/Dockerfile
    domainname: rmdevcloud2.int
    environment:
      DEVCONTAINER_WORKSPACE: /root/workspace
      PYTHONDONTWRITEBYTECODE: "1"
    hostname: amdorc01
    networks:
      default: null
    stdin_open: true
    tty: true
    volumes:
    - type: bind
      source: /home/user/project
      target: /root/workspace
      bind:
        create_host_path: true
    - type: bind
      source: /home/user/.aws
      target: /root/.aws
      read_only: true
      bind:
        create_host_path: true
  redis:
    hostname: redis
    image: artifactory.otxlab.net/dockerhub/redis:6-alpine
    networks:
      default: null
networks:
  default:
    name: devcontainer_default
[29552 ms] Start: Run: docker inspect --type image "artifactory.company.net/dockerhub/debian:testing-slim"
[29597 ms] Failed to get registry auth token with error: Error: getaddrinfo ENOTFOUND "artifactory.company.net
[29598 ms] Error fetching image details: No manifest found for "artifactory.company.net/dockerhub/debian:testing-slim".
[29598 ms] Start: Run: docker pull "artifactory.company.net/dockerhub/debian:testing-slim"
invalid reference format
[29625 ms] []
[29625 ms] Error response from daemon: no such image: "artifactory.company.net/dockerhub/debian:testing-slim": invalid reference format

[29625 ms] Command failed: docker inspect --type image "artifactory.company.net/dockerhub/debian:testing-slim"
[29630 ms] Exit code 1

That's interesting. It complains about missing credentials when using our artifactory cache for dockerhub. This is no issue when calling docker-compose directly

user@host:~> docker pull artifactory.company.net/dockerhub/debian:testing-slim
testing-slim: Pulling from dockerhub/debian
dd0227bc7b1d: Already exists 
Digest: sha256:e7fe35a2c6fa835004c44872fb0a48514ab04a4cec7af82e92de0aa25f49932d
Status: Downloaded newer image for artifactory.company.net/dockerhub/debian:testing-slim
artifactory.company.net/dockerhub/debian:testing-slim
user@host:~>

user@host:~> docker inspect --type image "artifactory.company.net/dockerhub/debian:testing-slim"
[
    {
        "Id": "sha256:d0f250baa43d8037e7ed276036de7106472bde0ba0bf2c7f350a8a888cb1ce48",
        "RepoTags": [
            "artifactory.company.net/dockerhub/debian:testing-slim"
        ],
        "RepoDigests": [
            "artifactory.company.net/dockerhub/debian@sha256:e7fe35a2c6fa835004c44872fb0a48514ab04a4cec7af82e92de0aa25f49932d"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2022-10-04T23:28:28.109878494Z",
        "Container": "05d8ee4d6a13b56a4a5cefdf350023246fdacc599d962f257e408c9de64b43dc",
        "ContainerConfig": {
            "Hostname": "05d8ee4d6a13",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"bash\"]"
            ],
            "Image": "sha256:c5ef58f4b022f39550a97f0592bed44466e7d0467d0d14ab8a0fbc2094ac6984",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "20.10.12",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "bash"
            ],
            "Image": "sha256:c5ef58f4b022f39550a97f0592bed44466e7d0467d0d14ab8a0fbc2094ac6984",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 75355631,
        "VirtualSize": 75355631,
        "GraphDriver": {
            "Data": {
                "MergedDir": "/data/docker/overlay2/dd9b3cbb06cf68207caf53244d1f8b04961d85604bf85044e273162fe7ec5303/merged",
                "UpperDir": "/data/docker/overlay2/dd9b3cbb06cf68207caf53244d1f8b04961d85604bf85044e273162fe7ec5303/diff",
                "WorkDir": "/data/docker/overlay2/dd9b3cbb06cf68207caf53244d1f8b04961d85604bf85044e273162fe7ec5303/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f652595e3b0ae75ff6f255d986adc0a8f0bd9e483ed33dcbc8a2c53b3b79a208"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]
user@host:~>

chrmarti commented 2 years ago

Does your Dockerfile use double-quotes around the image name after FROM? That might cause the invalid reference format error.

leo-ossais commented 2 years ago

@chrmarti This is my return for docker buildx ls ~ docker buildx ls NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS default * docker default default running 20.10.17 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6 desktop-linux docker desktop-linux desktop-linux running 20.10.17 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

And I executed docker buildx use default, but still the error persists.

I'm going back to 0.251.0 workarounds the issue for me.

I believe it is an error added in the new versions.

janhurst commented 2 years ago

also experiencing this issue here. we use a devcontainer launched from a WSL remote session and whenever running the rebuild container task receive the same cryptic error

ryboe commented 2 years ago

I'm also experiencing the issue. It's trying to fetch metadata for the dev_container_feature_content_temp from Docker Hub and failing.

=> ERROR [internal] load metadata for docker.io/library/dev_container_fe  1.1s

I'm using Docker Desktop 4.12.0 on macOS 12.6.

afischer-opentext-com commented 2 years ago

@chrmarti the docker file starts with this

ARG TEMURIN_IMAGE="artifactory.company.net/dockerhub/eclipse-temurin:17-jre"
ARG BASE_IMAGE="artifactory.company.net/dockerhub/debian:testing-slim"

FROM "${TEMURIN_IMAGE}" AS TEMURIN
FROM "${BASE_IMAGE}"

When removing the quotes from the FROM instructions, the setup starts to work.

ARG TEMURIN_IMAGE="artifactory.company.net/dockerhub/eclipse-temurin:17-jre"
ARG BASE_IMAGE="artifactory.company.net/dockerhub/debian:testing-slim"

FROM ${TEMURIN_IMAGE} AS TEMURIN
FROM ${BASE_IMAGE}

chrmarti commented 1 year ago

This fix is available in Dev Containers 0.264.0-pre-release. Let me know if that fixes this issue for you. Thanks.

aeschli commented 1 year ago

I verified that quotes around the FROM value are not causing issues.

ARG IMAGE="mcr.microsoft.com/vscode/devcontainers/typescript-node"

FROM "${IMAGE}"

microsoft / vscode-remote-release

Dev Container 0.255.2 : Compose: Rebuild failed #7323

[mamita-web internal] load metadata for docker.io/library/dev_container_feature_content_temp:latest: