devspace build not working on WSL2

[lee-aaron]

What happened? This runs fine on my MacOS but I can't seem to build devspace on WSL2. Getting the output below.

Ensuring image pull secret for registry: DESKTOP-R2R7EL6:5000...
Couldn't retrieve username for registry DESKTOP-R2R7EL6:5000 from docker store
Couldn't retrieve password for registry DESKTOP-R2R7EL6:5000 from docker store
local-registry: Starting Local Image Registry
local-registry: Port forwarding to local registry started on: 32424 -> 5000
Ensuring image pull secret for registry: DESKTOP-R2R7EL6:5000...
Couldn't retrieve username for registry DESKTOP-R2R7EL6:5000 from docker store
Couldn't retrieve password for registry DESKTOP-R2R7EL6:5000 from docker store
build:objectstorage Rebuild image localhost:32424/objectstorage: because it was not found in the local registry
build:objectstorage Building image 'localhost:32424/objectstorage:kcHuFXj' with engine 'docker'
build:objectstorage Execute docker cli command with: docker build --tag localhost:32424/objectstorage:kcHuFXj --file Dockerfile --platform=linux/amd64 -
#1 [internal] load remote build contextxt to Docker daemon  557.1kB
build:objectstorage #1 sha256:64d03271ddd574b62c0f129a578a5bec9fb8dcf391e2aab6da89449abad7fb1e
build:objectstorage #1 ERROR: invalid response status 403
build:objectstorage ------
build:objectstorage  > [internal] load remote build context:
build:objectstorage ------
build:objectstorage failed to read downloaded context: failed to load cache key: invalid response status 403
build_images: build images: error building image localhost:32424/objectstorage:kcHuFXj: error executing 'docker build --tag localhost:32424/objectstorage:kcHuFXj --file Dockerfile --platform=linux/amd64 -':
fatal exit status 1

What did you expect to happen instead? Pretty much expect to be able to authenticate without coming across a 403 forbidden error. I should see building / deploying working fine.

How can we reproduce the bug? (as minimally and precisely as possible) On a Windows machine, you should have WSL2 with Ubuntu. I have a droplet running on DigitalOcean where I reverse tunneled to port 5000 and should be able to authenticate with the registry there.

Note: My registry address resolves to 127.0.1.1 instead of 127.0.0.1

My devspace.yaml (trimmed some of it for privacy reasons):

version: v2beta1
name: localenv

vars:
  IMAGE: ${REGISTRY}/objectstorage

images:
  objectstorage:
    image: "${IMAGE}"
    dockerfile: Dockerfile
    docker:
      args:
        - "--platform=linux/amd64"

dev:
  objectstorage:
    labelSelector:
      app.kubernetes.io/component: objectstorage
    ports:
      - port: "4000:4000"

pipelines:
  dev:
    run: |
      create_deployments objectstorage
      echo "objectstorage deployed"
  connect:
    run: |
      start_dev objectstorage

  build:
    run: |
      make build
      build_images objectstorage
      echo "objectstorage built"

Local Environment:

• DevSpace Version: 6.2.1
• Operating System: windows
• ARCH of the OS: AMD64 Kubernetes Cluster:
• Cloud Provider: digitalocean
• Kubernetes Version: Client Version: v1.25.4 Kustomize Version: v4.5.7 Server Version: v1.23.8+k3s1

Anything else we need to know?

I set REGISTRY in my env vars to be my hostname:5000

[lizardruss]

@lee-aaron Hello! It looks like our new local registry feature is interfering with your existing registry. Could you try disabling it by adding this to your devspace.yaml config?

localRegistry:
  enabled: false

[lizardruss]

@lee-aaron It also looks like DevSpace wasn’t able to find your registry credentials according to:

Couldn't retrieve password for registry DESKTOP-R2R7EL6:5000 from docker store

Is this a registry you would need to run docker login with first?

[lee-aaron]

Hi @lizardruss I don't think I need to run docker login but it works fine on Mac without needing to do so.

Running devspace build -b shows this now

build_images: build images: error building image DESKTOP-R2R7EL6:5000/objectstorage:wtzoAGC: Error during image registry authentication: Error response from daemon: login attempt to http://DESKTOP-R2R7EL6:5000/v2/ failed with status: 403 Forbidden

[lizardruss]

@lee-aaron Looks like it's unrelated to the new local registry feature since you were able to turn that off.

Are you able to build the image using docker by itself? The command DevSpace is running would be similar to:

docker build --tag DESKTOP-R2R7EL6:5000/objectstorage:wtzoAGC --file Dockerfile --platform=linux/amd64 .

If that completes successfully with docker but fails with devspace build -b, please run it again with the --debug flag and share the log output.

[lee-aaron]

@lizardruss

10:56:41 debug Run pipeline:
name: build
run: |
    make build
    build_images objectstorage
    echo "objectstorage built"

10:56:41 GOOS=linux GOARCH=amd64 go build -o build/objectstorage .
10:56:41 if [ -n "" ]; then\
10:56:41        patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 build/objectstorage;\
10:56:41 fi
10:56:41 build_images objectstorage
10:56:41 Marked project excluded: ostorinabox
10:56:41 Ensuring image pull secret for registry: DESKTOP-R2R7EL6:5000...
10:56:41 Couldn't retrieve username for registry DESKTOP-R2R7EL6:5000 from docker store
10:56:41 Couldn't retrieve password for registry DESKTOP-R2R7EL6:5000 from docker store
10:56:42 local-registry: Starting Local Image Registry
10:56:42 local-registry: Wait for local registry node port to be assigned...
10:56:42 local-registry: Wait for running local registry pod...
10:56:43 local-registry: Check for running local registry
10:56:43 local-registry: Starting local registry port forwarding
10:56:43 local-registry: Port forwarding to local registry started on: 32424 -> 5000
10:56:43 local-registry: Waiting for local registry to become ready...
10:56:43 Ensuring image pull secret for registry: DESKTOP-R2R7EL6:5000...
10:56:43 Couldn't retrieve username for registry DESKTOP-R2R7EL6:5000 from docker store
10:56:43 Couldn't retrieve password for registry DESKTOP-R2R7EL6:5000 from docker store
10:56:43 build:objectstorage Rebuild image localhost:32424/objectstorage: because it was not found in the local registry
10:56:43 build:objectstorage Building image 'localhost:32424/objectstorage:sCFgWdE' with engine 'docker'
10:56:44 build:objectstorage Execute docker cli command with: docker build --tag localhost:32424/objectstorage:sCFgWdE --file Dockerfile --platform=linux/amd64 -
#1 [internal] load remote build contextild context to Docker daemon  557.1kB
10:56:44 build:objectstorage #1 sha256:0e6e15b4a9ec374e041f59929f3db747e38631348914ba4bec95b71a437c76fd
10:56:45 build:objectstorage #1 ERROR: invalid response status 403
10:56:45 build:objectstorage ------
10:56:45 build:objectstorage  > [internal] load remote build context:
10:56:45 build:objectstorage ------
10:56:45 build:objectstorage failed to read downloaded context: failed to load cache key: invalid response status 403
10:56:45 build_images: build images: error building image localhost:32424/objectstorage:sCFgWdE: error executing 'docker build --tag localhost:32424/objectstorage:sCFgWdE --file Dockerfile --platform=linux/amd64 -':
10:56:45 fatal exit status 1

I also contacted Docker support and they released 4.15.0 today with fixes for this. I upgraded docker and ran it with the above. I'll try to restart my machine and see if this still has issues.

[lee-aaron]

Hi @lizardruss I've been still having this issue. Docker support said they don't see anything wrong with my logs from diagnostics so I'm wondering if there's some way for you to test this on your end. I'm still getting 403 as above.

[lee-aaron]

Turns out having containerd on via docker extensions was causing the issues.

[igoooor]

@lee-aaron could you explain how you fixed it? I believe I am facing a similar issue

[lee-aaron]

I actually have not fixed it. Still discussing with Docker Support on what to do

[lee-aaron]

It might have to do with how WSL2 resolves the host. Trying to see if there's some way to override how docker resolves a hostname.

[lizardruss]

@lee-aaron Have you tried with the latest version of DevSpace? Some of our recent fixes might help with this.

[lee-aaron]

Yes I'm on devspace 6.2.5 but I still haven't figured out how to put in a proper hostname entry either in my /etc/hosts or my /etc/resolv.conf. It still fails though

[withinboredom]

For anyone still running into this issue.

1. Edit with nano ~/.docker/config.json in WSL
2. Remove the reference to "credhelper.exe" (if this is your first edit, you should have an empty json file now).
3. Run docker login in WSL
4. Everything should work now.
5. Every single time you reboot or start docker desktop, you must remove the reference to credhelper.

The issue is that devspace doesn't know how to use the WSL credential helper.