search

GoogleContainerTools / skaffold

Easy and Repeatable Kubernetes Development
https://skaffold.dev/
Apache License 2.0
15.05k stars 1.62k forks source link

support cross-platform build on Google Cloud Build #7555

Closed j-windsor closed 2 years ago

j-windsor commented 2 years ago

Expected behavior

Build completes

Actual behavior

Build Fails with:

#8 [4/5] RUN npm install
#8 0.309 exec /bin/sh: exec format error
#8 ERROR: executor failed running [/bin/sh -c npm install]: exit code: 1

Information

Steps to reproduce the behavior

  1. git clone https://github.com/GoogleCloudPlatform/cloud-code-samples
  2. Open nodejs/nodejs-hello-world
  3. minikube start
  4. skaffold run --profile cloudbuild --default-repo gcr.io/my-repo

Full Logs

➜  hello-world-38 skaffold run --profile cloudbuild --default-repo gcr.io/jaywindsor-test-gke-1 -vdebug
DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v1.38.0 ConfigVersion:skaffold/v2beta28 GitVersion: GitCommit:89b789ddcfe00d2fe7626fd86ef39a3eb6b455c5 BuildDate:2022-04-06T15:04:33Z GoVersion:go1.17.7 Compiler:gc Platform:darwin/arm64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/Users/jameswindsor/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] config version out of date: upgrading to latest "skaffold/v2beta28"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /Users/jameswindsor/hello-world-38/skaffold.yaml  subtask=-1 task=DevLoop
INFO[0000] applying profile: cloudbuild                  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field Build  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field artifacts  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field insecureRegistries  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field tagPolicy  subtask=-1 task=DevLoop
INFO[0000] no values found in profile for field TagPolicy, using original config values  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field platforms  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field BuildType  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field Test   subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field Deploy  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field DeployType  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field docker  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field helm   subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field kpt    subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field kubectl  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field kustomize  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field statusCheck  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field kubeContext  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field logs   subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field prefix  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field jsonParse  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field fields  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field PortForward  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field ResourceSelector  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field allow  subtask=-1 task=DevLoop
DEBU[0000] overlaying profile on config for field deny   subtask=-1 task=DevLoop
INFO[0000] map entry found when executing locate for &{nodejs-hello-world . <nil> {0x140005e02c0 <nil> <nil> <nil> <nil> <nil> <nil>} [] {[] []} []} of type *v1.Artifact and pointer: 1374395814592  subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: minikube               subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [{"commit":"362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7","minikubeVersion":"v1.25.2"}
]  subtask=-1 task=DevLoop
DEBU[0000] Running command: [/Users/jameswindsor/Library/Application Support/cloud-code/installer/google-cloud-sdk/bin/minikube docker-env --shell none -p minikube --user=skaffold]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://127.0.0.1:53299
DOCKER_CERT_PATH=/Users/jameswindsor/.minikube/certs
MINIKUBE_ACTIVE_DOCKERD=minikube
]  subtask=-1 task=DevLoop
DEBU[0000] setting Docker user agent to skaffold-v1.38.0  subtask=-1 task=DevLoop
INFO[0000] Using minikube docker daemon at tcp://127.0.0.1:53299  subtask=-1 task=DevLoop
DEBU[0000] CLI platforms provided: ""                    subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0000] platforms detected from active kubernetes cluster nodes: "linux/arm64"  subtask=-1 task=DevLoop
DEBU[0000] platforms selected for artifact "nodejs-hello-world": "linux/arm64"  subtask=-1 task=DevLoop
DEBU[0000] Using builder: google cloud                   subtask=-1 task=DevLoop
INFO[0000] build concurrency first set to 0 parsed from *gcb.Builder[0]  subtask=-1 task=DevLoop
INFO[0000] final build concurrency value is 0            subtask=-1 task=DevLoop
Generating tags...
 - nodejs-hello-world -> gcr.io/jaywindsor-test-gke-1/nodejs-hello-world:latest
INFO[0000] Tags generated in 127.625µs                   subtask=-1 task=Build
Checking cache...
DEBU[0002] Found dependencies for dockerfile: [{package-lock.json /hello-world true 7 7} {package.json /hello-world true 7 7} {. /hello-world true 11 11}]  subtask=-1 task=DevLoop
DEBU[0002] Could not import artifact from Docker, building instead (import of missing images disabled)  subtask=-1 task=Build
 - nodejs-hello-world: Not found. Building
INFO[0002] Cache check completed in 1.317 second         subtask=-1 task=Build
Starting build...
Building [nodejs-hello-world]...
Target platforms: [linux/arm64]
DEBU[0002] Running command: [gcloud auth print-access-token --format=json]  subtask=nodejs-hello-world task=Build
DEBU[0002] Command output: [{
  "token": "ya29.A0ARrdaM_iwcftFNwnKmDbfS4NwB0bhT-r-mNux4ktreUY_ZksnMI1BBn3UWmPTYxza-2g0KvdYi47mtjuoHuRBOZU_3nkNyMMYIo5pUvfq7783_xouCcQg9BSwKpsCrfOgTM1DdOc6lHM3vPutkCdJiAp67893tcdRfW80QYUNnWUtBVEFTQVRBU0ZRRl91NjFWOGh5QmppekFBTy1pSWc4RFI5R1BGQQ0173"
}
]  subtask=nodejs-hello-world task=Build
INFO[0002] unable to retrieve token: oauth2: token expired and refresh token is not set  subtask=-1 task=DevLoop
INFO[0002] falling back to application default credentials  subtask=-1 task=DevLoop
DEBU[0002] project id set to jaywindsor-test-gke-1       subtask=nodejs-hello-world task=Build
Pushing code to gs://jaywindsor-test-gke-1_cloudbuild/source/jaywindsor-test-gke-1-1ea18020-8854-49bf-b32a-5b55f5931008.tar.gz
Logs are available at 
https://console.cloud.google.com/m/cloudstorage/b/jaywindsor-test-gke-1_cloudbuild/o/log-9f910934-4138-457c-9ffc-64c81b2e29af.txt
DEBU[0005] current offset 0                              subtask=nodejs-hello-world task=Build
starting build "9f910934-4138-457c-9ffc-64c81b2e29af"

FETCHSOURCE
Fetching storage object: gs://jaywindsor-test-gke-1_cloudbuild/source/jaywindsor-test-gke-1-1ea18020-8854-49bf-b32a-5b55f5931008.tar.gz#1655329961304718
DEBU[0008] current offset 220                            subtask=nodejs-hello-world task=Build
Copying gs://jaywindsor-test-gke-1_cloudbuild/source/jaywindsor-test-gke-1-1ea18020-8854-49bf-b32a-5b55f5931008.tar.gz#1655329961304718...
/ [1 files][768.6 KiB/768.6 KiB]                                                
Operation completed over 1 objects/768.6 KiB.                                    
DEBU[0012] current offset 604                            subtask=nodejs-hello-world task=Build
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 320B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/node:18-slim
DEBU[0015] current offset 920                            subtask=nodejs-hello-world task=Build
#3 DONE 0.6s

#6 [internal] load build context
#6 transferring context: 1.09MB 0.0s done
#6 DONE 0.1s

#4 [1/5] FROM docker.io/library/node:18-slim@sha256:0739e03851228cc1380f60e...
#4 resolve docker.io/library/node:18-slim@sha256:0739e03851228cc1380f60e9dc14c192bd9d22d02eab364de609b6b8efb94174 0.0s done
#4 sha256:bd9810e5faec7cfc54e044883788f27d4150ed254ebdf4c876f1f4569a910ab8 1.37kB / 1.37kB done
#4 sha256:7d501d4fb4ae777ff44cc4217177fc509cd7fc0dbe95e068fb2d34799d48eee8 7.04kB / 7.04kB done
#4 sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba 0B / 30.07MB 0.1s
#4 sha256:2c57a1409c9cb0b971d9244c4edb67c41a3d019f2ff034c05bd0b4fd7a6c4c17 0B / 4.05kB 0.1s
#4 sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 0B / 45.78MB 0.1s
#4 sha256:0739e03851228cc1380f60e9dc14c192bd9d22d02eab364de609b6b8efb94174 1.21kB / 1.21kB done
#4 sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba 3.15MB / 30.07MB 0.3s
#4 sha256:2c57a1409c9cb0b971d9244c4edb67c41a3d019f2ff034c05bd0b4fd7a6c4c17 4.05kB / 4.05kB 0.2s done
#4 sha256:0a4bc6a3b78990d80ae4baabc546332300f8eaf05c05f5810a349f684e447cd1 0B / 2.77MB 0.3s
#4 sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba 8.39MB / 30.07MB 0.4s
#4 sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 6.29MB / 45.78MB 0.4s
#4 sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba 24.12MB / 30.07MB 0.6s
#4 sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 19.92MB / 45.78MB 0.6s
#4 sha256:0a4bc6a3b78990d80ae4baabc546332300f8eaf05c05f5810a349f684e447cd1 2.77MB / 2.77MB 0.4s done
#4 sha256:6642fbc6b4b412a15896f3a794f4f0cd7198986ca5e3207a175a77dd2e11a597 0B / 454B 0.6s
#4 sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba 30.07MB / 30.07MB 0.7s
#4 sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 30.41MB / 45.78MB 0.7s
#4 sha256:6642fbc6b4b412a15896f3a794f4f0cd7198986ca5e3207a175a77dd2e11a597 454B / 454B 0.6s done
#4 sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 45.78MB / 45.78MB 0.9s
#4 sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba 30.07MB / 30.07MB 0.9s done
#4 sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 45.78MB / 45.78MB 1.1s done
#4 extracting sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba
#4 extracting sha256:dc1f00a5d701e86e2cd2568b860c61b393d66be1341e7267d07e6e57448ceeba 1.8s done
#4 extracting sha256:2c57a1409c9cb0b971d9244c4edb67c41a3d019f2ff034c05bd0b4fd7a6c4c17 done
#4 extracting sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 0.1s
DEBU[0018] current offset 3624                           subtask=nodejs-hello-world task=Build
#4 extracting sha256:8ef80393755852da08e7bcb85e88e8e5087bc8fcae083d67f90c2a53e03b50cf 2.9s done
#4 extracting sha256:0a4bc6a3b78990d80ae4baabc546332300f8eaf05c05f5810a349f684e447cd1
#4 extracting sha256:0a4bc6a3b78990d80ae4baabc546332300f8eaf05c05f5810a349f684e447cd1 0.2s done
#4 extracting sha256:6642fbc6b4b412a15896f3a794f4f0cd7198986ca5e3207a175a77dd2e11a597
#4 extracting sha256:6642fbc6b4b412a15896f3a794f4f0cd7198986ca5e3207a175a77dd2e11a597 done
#4 DONE 6.7s

#5 [2/5] WORKDIR /hello-world
DEBU[0022] current offset 4123                           subtask=nodejs-hello-world task=Build
#5 DONE 1.4s

#7 [3/5] COPY package*.json ./
#7 DONE 0.0s

#8 [4/5] RUN npm install
#8 0.309 exec /bin/sh: exec format error
#8 ERROR: executor failed running [/bin/sh -c npm install]: exit code: 1
------
 > [4/5] RUN npm install:
------
executor failed running [/bin/sh -c npm install]: exit code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
DEBU[0024] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0024] Command output: [256
]                        subtask=-1 task=DevLoop
build [nodejs-hello-world] failed:  cloud build failed: FAILURE
DEBU[0024] exporting metrics                             subtask=-1 task=DevLoop
DEBU[0025] metrics uploading complete in 771.199458ms    subtask=-1 task=DevLoop
gsquared94 commented 2 years ago

investigated offline with @j-windsor, the issue seems to be due to not having docker configured with binfmt_misc extensions. AI should be to update documentation here to highlight this requirement.

j-windsor commented 2 years ago

For the Cloud Build case, it sounds like we are going to add a new field under googleCloudBuild: binfmtImage to control what image is run to setup qemu before the build step occurs, if any. It will be set to nil initially, and skaffold will add an actionable error message to suggest setting the field when a build fails with an error that looks like it could be fixed with this.