Run App on Local Cloud Run Emulator fails on ARM64 / M1 Mac

[esteluk]

Environment: (Hint: "Report Extension Issue on Github" command will fill these out for you.) ^ This command doesn't seem to exist. The "Report Bug" option offers a lot of logs, but none seem entirely relevant to this section.

Version information

Cloud Code Extension version: v2.7.1

VSCode version: 1.88.0

OS: MacOS 14.3.1

Cloud SDK: 471.0.0

Skaffold: 2.9.0

Kubectl: 1.26.15

Description:

I have a trivial Cloud Run application, and when trying to "Run on a cloud run emulator" (or Debug), the resulting image inevitably fails to deploy.

I don't have a strong golang/GCP background, so apologies if I'm being dumb. (I suspect this may be the case, given that I'd have assumed someone else would have encountered this issue if it did exist in this way.

launch.json:

{
    "configurations": [
        {
            "name": "Cloud Run: Run/Debug Locally",
            "type": "cloudcode.cloudrun",
            "request": "launch",
            "build": {
                "docker": {
                    "path": "Dockerfile"
                }
            },
            "image": "backend-app",
            "service": {
                "name": "backend-app",
                "containerPort": 8080,
                "resources": {
                    "limits": {
                        "memory": "256Mi"
                    }
                }
            },
            "target": {
                "minikube": {}
            },
            "watch": true
        }
    ]

}

The detailed log are:

Running skaffold command: /Users/nathan/Developer/tools/google-cloud-sdk/bin/skaffold init --artifact {"image":"backend-app","builder":"Docker","payload":{"path":"Dockerfile"}} --kubernetes-manifest /var/folders/61/mg54nw594w3g3cb90qp4c3zm0000gn/T/cloud-code-cloud-run-TqaZAo/pods_and_services.yaml --filename /var/folders/61/mg54nw594w3g3cb90qp4c3zm0000gn/T/cloud-code-cloud-run-TqaZAo/skaffold.yaml --force
Setting build tagPolicy to SHA256.
Setting statusCheckDeadlineSeconds to 5 minutes
Setting build environment to 'local'
Created the file /var/folders/61/mg54nw594w3g3cb90qp4c3zm0000gn/T/cloud-code-cloud-run-TqaZAo/skaffold.yaml.
Running gcloud command: /Users/nathan/Developer/tools/google-cloud-sdk/bin/gcloud alpha code export --readiness-probe --service-name backend-app --image backend-app --no-skaffold-file --dockerfile= --kubernetes-file /var/folders/61/mg54nw594w3g3cb90qp4c3zm0000gn/T/cloud-code-cloud-run-TqaZAo/pods_and_services.yaml --env-vars K_SERVICE=backend-app,K_REVISION=local,K_CONFIGURATION=backend-app --memory 256Mi --quiet --verbosity warning
Created the file /var/folders/61/mg54nw594w3g3cb90qp4c3zm0000gn/T/cloud-code-cloud-run-TqaZAo/pods_and_services.yaml.
Running: skaffold dev -v info --port-forward --rpc-http-port 50985 --filename /var/folders/61/mg54nw594w3g3cb90qp4c3zm0000gn/T/cloud-code-cloud-run-TqaZAo/skaffold.yaml --kube-context cloud-run-dev-internal --minikube-profile cloud-run-dev-internal --status-check\=true
Activated kube-context \"cloud-run-dev-internal\"
starting gRPC server on port 51036
starting gRPC HTTP server on port 50985 (proxying to 51036)
Skaffold &{Version:v2.9.0 ConfigVersion:skaffold/v4beta8 GitVersion: GitCommit:6071a3f7574702c8666a243d89254e9b0d8ff4d7 BuildDate:2023-11-07T13:20:11Z GoVersion:go1.21.0 Compiler:gc Platform:darwin/arm64 User:}
Loaded Skaffold defaults from \"/Users/nathan/.skaffold/config\"
map entry found when executing locate for &{backend-app . <nil> {0x14000727ad0 <nil> <nil> <nil> <nil> <nil> <nil>} [] {[] []} [] } of type *latest.Artifact and pointer: 1374399222864
Using kubectl context: cloud-run-dev-internal
Using minikube docker daemon at tcp://127.0.0.1:64411
no kpt renderer or deployer found, skipping hydrated-dir creation
build concurrency first set to 1 parsed from *runner.pipelineBuilderWithHooks[0]
final build concurrency value is 1
Generating tags...
 - backend-app -> Tags generated in 147.083µs
backend-app:latest
Checking cache...
 - backend-app: Not found. Building
Starting build...
Found [cloud-run-dev-internal] context, using local docker daemon.
Building [backend-app]...
Target platforms: [linux/arm64]
Cache check completed in 17.104625ms
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.21kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/ubuntu:22.04
#3 DONE 0.0s

#4 [internal] load metadata for docker.io/library/golang:1.22
#4 DONE 0.0s

#5 [builder 1/6] FROM docker.io/library/golang:1.22
#5 DONE 0.0s

#6 [stage-1 1/3] FROM docker.io/library/ubuntu:22.04
#6 DONE 0.0s

#7 [internal] load build context
#7 transferring context: 3.34kB done
#7 DONE 0.0s

#8 [builder 2/6] WORKDIR /app
#8 CACHED

#9 [builder 3/6] COPY go.* ./
#9 CACHED

#10 [builder 4/6] RUN go mod download
#10 CACHED

#11 [builder 5/6] COPY . ./
#11 DONE 0.0s

#12 [builder 6/6] RUN CGO_ENABLED=0 GOOS=linux go build -mod=readonly -v -o server
<SNIP>
#12 DONE 38.6s

#13 [stage-1 2/3] COPY --from=builder /app/server /server
#13 CACHED

#14 [stage-1 3/3] COPY --from=builder /bin/sleep /bin/sleep
#14 CACHED

#15 exporting to image
#15 exporting layers done
#15 writing image sha256:afe815995131ed6d7a39414ab281f7c3f39864232d85f85e2f4bb2de1e899736 done
#15 naming to docker.io/library/backend-app:latest done
#15 DONE 0.0s

View build details: docker-desktop://dashboard/build/default/default/uxkodbdc79ae8ogaqwjachxkc

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
Build [backend-app] succeeded
Build completed in 39.312 seconds
Starting render...
starting render process
Render completed in 1.817583ms
Tags used in deployment:
 - backend-app -> backend-app:afe815995131ed6d7a39414ab281f7c3f39864232d85f85e2f4bb2de1e899736
Starting deploy...
 - Warning: spec.template.spec.containers[0].env[4]: hides previous definition of "K_CONFIGURATION"
 - Warning: spec.template.spec.containers[0].env[5]: hides previous definition of "K_REVISION"
 - Warning: spec.template.spec.containers[0].env[6]: hides previous definition of "K_SERVICE"
 - deployment.apps/backend-app created
 - service/backend-app created
Waiting for deployments to stabilize...
 - deployment/backend-app: container backend-app-readiness-probe terminated with exit code 1
    - pod/backend-app-66cf688dd5-9jqsr: container backend-app-readiness-probe terminated with exit code 1
      > Error retrieving logs for pod backend-app-66cf688dd5-9jqsr: exit status 1.
Try `kubectl logs backend-app-66cf688dd5-9jqsr -n default -c backend-app-readiness-probe`
 - deployment/backend-app failed. Error: container backend-app-readiness-probe terminated with exit code 1.
Cleaning up...
 - deployment.apps "backend-app" deleted
 - service "backend-app" deleted
Cleanup completed in 87.57075ms
1/1 deployment(s) failed
Skaffold exited with code 1.
Cleaning up...
Deleted the temporary directory /var/folders/61/mg54nw594w3g3cb90qp4c3zm0000gn/T/cloud-code-cloud-run-TqaZAo.

The readiness check certainly fails to execute as expected. When running the recommended command it returns:

❯ kubectl --context "cloud-run-dev-internal" logs backend-app-77c95f5bcb-l62ss -n default -c backend-app-readiness-probe
exec /bin/sleep: exec format error

The output of gcloud alpha code export --readiness-probe --service-name backend-app --image backend-app --no-skaffold-file --dockerfile= --kubernetes-file temp/pods_and_services.yaml --env-vars K_SERVICE=backend-app,K_REVISION=local,K_CONFIGURATION=backend-app --memory 256Mi --quiet --verbosity warn provides a Kubernetes file with a readiness probe:

kind: Deployment
metadata:
  labels:
    service: backend-app
  name: backend-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend-app
  template:
    metadata:
      labels:
        app: backend-app
    spec:
      containers:
      - env:
        - name: PORT
          value: '8080'
        - name: K_CONFIGURATION
          value: dev
        - name: K_REVISION
          value: dev-0001
        - name: K_SERVICE
          value: backend-app
        - name: K_CONFIGURATION
          value: backend-app
        - name: K_REVISION
          value: local
        - name: K_SERVICE
          value: backend-app
        image: backend-app
        name: backend-app-container
        ports:
        - containerPort: 8080
        resources:
          limits:
            memory: 268435456
      - args:
        - infinity
        command:
        - sleep
        image: gcr.io/gcp-runtimes/ubuntu_18_0_4:latest
        name: backend-app-readiness-probe
        readinessProbe:
          exec:
            command:
            - grep
            - :1F90
            - /proc/net/tcp
            - /proc/net/tcp6
          periodSeconds: 1
      terminationGracePeriodSeconds: 0
---
apiVersion: v1
kind: Service
metadata:
  name: backend-app
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: backend-app
  type: LoadBalancer

But the Ubuntu 18.0.4 image doesn't seem to be multi-arch - the readiness probe is never able to successfully start. If I manually update the image to ubuntu:22.04 and run Skaffold manually (using the generated skaffold.yaml) the service deploys as expected.

Repro step:

1. Build minimum Cloud Run project
2. Run the Cloud Code: Run on Cloud Run Emulator
3. Fails

[davidponder]

Hello!

Apologies to hear you're hitting this friction (and sorry that it took some time to get back to you!) I'll bounce this over to the broader team for an investigation and one of us will get back to you ASAP! Feel free to keep us posted on anything else you may notice in the meantime.

Thanks!

[kevingoh]

Hi, have the same problem here.

[sphr2k]

@davidponder It would be really nice if cloud-code could optionally use an existing local Kubernetes cluster, like k3d, as minikube occasionally causes problems, at least on macOS.

[olliewild]

Any updates or workarounds for this? I am also running into this issue.

[glouischandra]

Hello! @esteluk I'm curious, what version of minikube do you have installed in the cloud code environment? you should be able to use the vscode terminal and issue 'minikube version'. Unfortunately currently we only support minikube for the cloud run local dev scenario

[esteluk]

Hey @glouischandra, thanks for replying! It's v1.32.0

[servefast-cto]

Same problem here on windows