search

karmada-io / karmada

Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
https://karmada.io
Apache License 2.0
4.45k stars 881 forks source link

Improve `hack/local-up-karmada.sh` execution time #1729

Closed RainbowMango closed 2 years ago

RainbowMango commented 2 years ago

What would you like to be added: The hack/local-up-karmada.sh is used for users and developers to set up the Karmada environment quickly. But the execution time suddenly gets very long, especially after we change the way building images in #927.

# hack/local-up-karmada.sh
...
real    14m12.453s
user    0m28.787s
sys 0m13.848

Now it takes about 14 minutes on my side and it just needs 3~5 minutes before.

We can get the execution time change from CI logs as well(from the job setup e2e test environment):

Why is this needed: This change has affected users to evaluate Karmada. See #1719

cc @gy95 @ikaven1024 (I'm not blaming you, just want to notice that we might have to make changes based on your work.) /help

karmada-bot commented 2 years ago

@RainbowMango: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/karmada-io/karmada/issues/1729): >**What would you like to be added**: >The `hack/local-up-karmada.sh` is used for users and developers to set up the Karmada environment quickly. >But the execution time suddenly gets very long, especially after we change the way building images in #927. > >``` ># hack/local-up-karmada.sh >... >real 14m12.453s >user 0m28.787s >sys 0m13.848 >``` >Now it takes about `14` minutes on my side and it just needs `3~5` minutes before. > >We can get the execution time change from CI logs as well(from the job `setup e2e test environment`): >- Before #927, it takes about `8` minutes. [click me for full logs](https://github.com/karmada-io/karmada/runs/6190287719?check_suite_focus=true#step:4:1) >- After #927, it takes about `32` minutes. [click me for full logs](https://github.com/karmada-io/karmada/runs/6203898128?check_suite_focus=true#step:4:1) > >**Why is this needed**: >This change has affected users to evaluate Karmada. See #1719 > >cc @gy @ikaven1024 >(I'm not blaming you, just want to notice that we might have to make changes based on your work.) >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
wuyingjun-lucky commented 2 years ago

I found there is make images stage in local-up.sh But there are no make images stage in remote-up can we set the make containers optional ? I guess it will work in private environment but ci pipeline will always costs time

RainbowMango commented 2 years ago

The original intention of hack/local-up-karmada.sh is to build Karmada from the source.

can we set the make containers optional ?

How to do that?

Maybe we should try to find the root cause first before making any decisions.

wuyingjun-lucky commented 2 years ago

to 1: I used to skip the make container in local-up manually

2: You mean Why we do make build binary and put it into docker container faster then we make binary in docker compile context and then put it into docker execution context?

gy95 commented 2 years ago

/kind help how to fasten make images now the multi-stage docker image build cost too much time, In CI, it takes about 32 minutes, but before https://github.com/karmada-io/karmada/pull/927 (NOT using multi-stage: just compile karmada components with local environment, and copy it into docker image) it only takes 8 minutes.

karmada-bot commented 2 years ago

@gy95: The label(s) kind/help cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/karmada-io/karmada/issues/1729#issuecomment-1119297230): >/kind help >how to fasten `make images` >now the multi-stage docker image build cost too much time, In CI, it takes about 32 minutes, >but before https://github.com/karmada-io/karmada/pull/927 (NOT using multi-stage: just compile karmada components with local environment, and copy it into docker image) it only takes 8 minutes. > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
hanweisen commented 2 years ago

I found there is make images stage in local-up.sh But there are no make images stage in remote-up can we set the make containers optional ? I guess it will work in private environment but ci pipeline will always costs time

I agree with @wuyingjun-lucky. I compare two logs. The longer one build binary package as follows:

logs ``` Step 1/9 : FROM golang:1.17-alpine3.15 AS builder 1.17-alpine3.15: Pulling from library/golang df9b9388f04a: Already exists 52dc419b0ee2: Pulling fs layer 25bc292e5bac: Pulling fs layer 447a8d7240ef: Pulling fs layer f1849c0ca460: Pulling fs layer f1849c0ca460: Waiting 52dc419b0ee2: Verifying Checksum 25bc292e5bac: Download complete f1849c0ca460: Verifying Checksum f1849c0ca460: Download complete 52dc419b0ee2: Pull complete 25bc292e5bac: Pull complete 447a8d7240ef: Verifying Checksum 447a8d7240ef: Download complete 447a8d7240ef: Pull complete f1849c0ca460: Pull complete Digest: sha256:5c2fcfeb58ad9d4948d94e7b2d0432a9bc38bee0f8dfb41d383f38a18b75c38d Status: Downloaded newer image for golang:1.17-alpine3.15 ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Running in b2ad859ef646 Removing intermediate container b2ad859ef646 ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Running in 29a2a0c2caab Removing intermediate container 29a2a0c2caab ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "${GO_LDFLAGS}" -o /bin/karmada-aggregated-apiserver cmd/aggregated-apiserver/main.go ---> Running in bc4b2d4cec46 Removing intermediate container bc4b2d4cec46 ---> 9601dd4439ae Step 6/9 : FROM alpine:3.15.1 ``` ``` VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-controller-manager fatal: No names found, cannot describe anything. Sending build context to Docker daemon 101.8MB Step 1/9 : FROM golang:1.17-alpine3.15 AS builder ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Using cache ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> Using cache ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Using cache ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "$GO_LDFLAGS" -o /bin/karmada-controller-manager cmd/controller-manager/controller-manager.go ---> Running in 91bab97734f2 Removing intermediate container 91bab97734f2 ---> dde240e8c10d Step 6/9 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 7/9 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 3ebc41f1f4e9 Step 8/9 : COPY --from=builder /bin/karmada-controller-manager /bin/karmada-controller-manager ---> c73969154742 Step 9/9 : CMD ["/bin/karmada-controller-manager"] ---> Running in 1f9acc208d8c Removing intermediate container 1f9acc208d8c ---> 6b44ef31f548 Successfully built 6b44ef31f548 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-scheduler fatal: No names found, cannot describe anything. Sending build context to Docker daemon 101.8MB Step 1/9 : FROM golang:1.17-alpine3.15 AS builder ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Using cache ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> Using cache ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Using cache ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "$GO_LDFLAGS" -o /bin/karmada-scheduler cmd/scheduler/main.go ---> Running in fadd32516006 Removing intermediate container fadd32516006 ---> 8df9ffadf54b Step 6/9 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 7/9 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 3ebc41f1f4e9 Step 8/9 : COPY --from=builder /bin/karmada-scheduler /bin/karmada-scheduler ---> bd97953f0092 Step 9/9 : CMD ["/bin/karmada-scheduler"] ---> Running in 498c1bd9bfce Removing intermediate container 498c1bd9bfce ---> f38976e6f3e4 Successfully built f38976e6f3e4 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-descheduler fatal: No names found, cannot describe anything. Sending build context to Docker daemon 101.8MB Step 1/9 : FROM golang:1.17-alpine3.15 AS builder ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Using cache ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> Using cache ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Using cache ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "$GO_LDFLAGS" -o /bin/karmada-descheduler cmd/descheduler/main.go ---> Running in 3ba82f6b686f Removing intermediate container 3ba82f6b686f ---> a4596dfc84ba Step 6/9 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 7/9 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 3ebc41f1f4e9 Step 8/9 : COPY --from=builder /bin/karmada-descheduler /bin/karmada-descheduler ---> 770d301134be Step 9/9 : CMD ["/bin/karmada-descheduler"] ---> Running in a44ad38aa0a7 Removing intermediate container a44ad38aa0a7 ---> b6d69e30abad Successfully built b6d69e30abad Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-webhook fatal: No names found, cannot describe anything. Sending build context to Docker daemon 101.8MB Step 1/9 : FROM golang:1.17-alpine3.15 AS builder ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Using cache ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> Using cache ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Using cache ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "$GO_LDFLAGS" -o /bin/karmada-webhook cmd/webhook/main.go ---> Running in d21c8e470731 Removing intermediate container d21c8e470731 ---> 277f39cfff4d Step 6/9 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 7/9 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 3ebc41f1f4e9 Step 8/9 : COPY --from=builder /bin/karmada-webhook /bin/karmada-webhook ---> ab8da6622790 Step 9/9 : CMD ["/bin/karmada-webhook"] ---> Running in 18dc63b2127d Removing intermediate container 18dc63b2127d ---> e48728b0e2fd Successfully built e48728b0e2fd Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-agent fatal: No names found, cannot describe anything. Sending build context to Docker daemon 101.8MB Step 1/9 : FROM golang:1.17-alpine3.15 AS builder ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Using cache ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> Using cache ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Using cache ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "$GO_LDFLAGS" -o /bin/karmada-agent cmd/agent/main.go ---> Running in 5fd10b04de17 Removing intermediate container 5fd10b04de17 ---> c32e4be3fd4e Step 6/9 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 7/9 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 3ebc41f1f4e9 Step 8/9 : COPY --from=builder /bin/karmada-agent /bin/karmada-agent ---> 64f04a41591e Step 9/9 : CMD ["/bin/karmada-agent"] ---> Running in a1403045ba15 Removing intermediate container a1403045ba15 ---> 6400f739d007 Successfully built 6400f739d007 Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-agent:latest VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-scheduler-estimator fatal: No names found, cannot describe anything. Sending build context to Docker daemon 101.8MB Step 1/9 : FROM golang:1.17-alpine3.15 AS builder ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Using cache ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> Using cache ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Using cache ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "$GO_LDFLAGS" -o /bin/karmada-scheduler-estimator cmd/scheduler-estimator/main.go ---> Running in 3562bd9c05e0 Removing intermediate container 3562bd9c05e0 ---> e61bb3f84a25 Step 6/9 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 7/9 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 3ebc41f1f4e9 Step 8/9 : COPY --from=builder /bin/karmada-scheduler-estimator /bin/karmada-scheduler-estimator ---> 2cdb6e490d17 Step 9/9 : CMD ["/bin/karmada-scheduler-estimator"] ---> Running in a0263cc3897e Removing intermediate container a0263cc3897e ---> a18e4b23c79c Successfully built a18e4b23c79c Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-interpreter-webhook-example fatal: No names found, cannot describe anything. Sending build context to Docker daemon 101.8MB Step 1/9 : FROM golang:1.17-alpine3.15 AS builder ---> 3156fdeb9740 Step 2/9 : WORKDIR /go/src/github.com/karmada-io/karmada ---> Using cache ---> 99a73794d3a5 Step 3/9 : COPY . /go/src/github.com/karmada-io/karmada ---> Using cache ---> 4b20e67f56fc Step 4/9 : ARG GO_LDFLAGS ---> Using cache ---> 3c19d81793f0 Step 5/9 : RUN CGO_ENABLED=0 go build -ldflags "$GO_LDFLAGS" -o /bin/karmada-interpreter-webhook-example examples/customresourceinterpreter/webhook/main.go ---> Running in b089301ab3b8 Removing intermediate container b089301ab3b8 ---> 0be8be764650 Step 6/9 : FROM alpine:3.15.1 ---> e9adb5357e84 Step 7/9 : RUN apk add --no-cache ca-certificates ---> Using cache ---> 3ebc41f1f4e9 Step 8/9 : COPY --from=builder /bin/karmada-interpreter-webhook-example /bin/karmada-interpreter-webhook-example ---> 997c3ab6a9d8 Step 9/9 : CMD ["/bin/karmada-interpreter-webhook-example"] ---> Running in 6b7b9d046713 Removing intermediate container 6b7b9d046713 ---> 3499829c75dd Successfully built 3499829c75dd Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-interpreter-webhook-example:latest make: Leaving directory '/home/runner/work/karmada/karmada' Waiting for the host clusters to be ready... Waiting for kubeconfig file /home/runner/.kube/karmada.config and clusters karmada-host to be ready... Context "kind-karmada-host" renamed to "karmada-host". Cluster "kind-karmada-host" set. Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest" with ID "sha256:6b44ef31f5481058663a08557fd8a7983dc3f04e9142557b6c13f49e360663b8" not yet present on node "karmada- host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler:latest" with ID "sha256:f38976e6f3e4ae9253f6e66013f414100c76e6be957cd6a021e54d2e5d21873c" not yet present on node "karmada-host-control- plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-descheduler:latest" with ID "sha256:b6d69e30abad5d7d522d780cf9e75cd67fe754ab10dceb1a2d9fc4307d3a8617" not yet present on node "karmada-host- control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-webhook:latest" with ID "sha256:e48728b0e2fd3794623916eee7c96f828f7dc8ce93d3c0d49d9743b5a97058e4" not yet present on node "karmada-host-control- plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-scheduler-estimator:latest" with ID "sha256:a18e4b23c79c9f16b6d280c4bcda4d773f4bc67db464a8e99c7f5bc4cfe3b8b6" not yet present on node "karmada- host-control-plane", loading... Image: "swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest" with ID "sha256:6884543244fa41e7bd0562eb29bd8c89ff11e2cf68520505e1512be6096a9bf3" not yet present on node "karmada- host-control-plane", loading... go install github.com/cloudflare/cfssl/cmd/...@v1.5.0 go: downloading github.com/cloudflare/cfssl v1.5.0 go: downloading github.com/go-sql-driver/mysql v1.4.0 go: downloading github.com/lib/pq v1.3.0 go: downloading github.com/mattn/go-sqlite3 v1.10.0 go: downloading github.com/cloudflare/go-metrics v0.0.0-20151117154305-6a9aea36fb41 go: downloading github.com/zmap/zlint/v2 v2.2.1 go: downloading github.com/jmoiron/sqlx v1.2.0 go: downloading github.com/GeertJohan/go.rice v1.0.0 go: downloading golang.org/x/crypto v0.0.0-20201012173705-84dcc777aaee go: downloading github.com/kisielk/sqlstruct v0.0.0-20150923205031-648daed35d49 go: downloading github.com/google/certificate-transparency-go v1.0.21 go: downloading github.com/cloudflare/redoctober v0.0.0-20171127175943-746a508df14c .... Karmada scheduler estimator of cluster member1 has been deployed. Note: To enable scheduler estimator, please deploy other scheduler estimators of all clusters. After that, specify the option '--enable-scheduler-estimator=true' of karmada-scheduler. cluster(member2) is joined successfully Switched to context "member2". ```

Remove above logs they are almost the same.

huone1 commented 2 years ago

it will download the image : golang:1.17-alpine3.15 repeatedly in each build

ikaven1024 commented 2 years ago

I test building binaries on host (out of docker) as Before. Firstly aggregated-apiserver spend 27s, while controller-manager 9s (faster much than aggregated-apiserver).

# commit aa39b35dedff7e1feb950f0a7583b480f7bb29ae (HEAD -> master)

➜  time CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
                                                              -ldflags "-X github.com/karmada-io/karmada/pkg/version.gitVersion= -X github.com/karmada-io/karmada/pkg/version.gitCommit=9d0c1fb679690e9fd7bb2a973f8e70e1f8ac57fb -X github.com/karmada-io/karmada/pkg/version.gitTreeState="clean" -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-04-27T08:45:12Z" \
                                                              -o karmada-aggregated-apiserver \
                                                              cmd/aggregated-apiserver/main.go

________________________________________________________
Executed in   27.30 secs    fish           external
   usr time  131.76 secs    0.20 millis  131.76 secs
   sys time   17.43 secs    1.15 millis   17.43 secs

➜  time CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
                                                              -ldflags "-X github.com/karmada-io/karmada/pkg/version.gitVersion= -X github.com/karmada-io/karmada/pkg/version.gitCommit=9d0c1fb679690e9fd7bb2a973f8e70e1f8ac57fb -X github.com/karmada-io/karmada/pkg/version.gitTreeState="clean" -X github.com/karmada-io/karmada/pkg/version.buildDate=2022-04-27T08:45:12Z" \
                                                              -o karmada-controller-manager \
                                                              cmd/controller-manager/controller-manager.go

________________________________________________________
Executed in    9.34 secs    fish           external
   usr time   32.15 secs   12.76 millis   32.14 secs
   sys time    5.66 secs    3.05 millis    5.66 secs

Then build in docker. bothaggregated-apiserver and controller-manager spend about 40s(calculate by date).

# commit cd05ffee6e3f79858147f08454e3c2eac0fbdb30 (HEAD -> master)

➜  karmada git:(master) ✗ time VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-aggregated-apiserver
Sending build context to Docker daemon  114.4MB
[1/2] STEP 1/7: FROM golang:1.17 AS builder
[1/2] STEP 2/7: WORKDIR /go/src/github.com/karmada-io/karmada
--> Using cache 6ae9f471a8110b2985aea152cdb319005351954aac915ec435349f604bccbba9
--> 6ae9f471a81
[1/2] STEP 3/7: COPY . /go/src/github.com/karmada-io/karmada
--> 6570120c969
[1/2] STEP 4/7: ARG GO_LDFLAGS
--> 4a5780f3285
[1/2] STEP 5/7: RUN date
Fri May  6 14:02:40 UTC 2022                         <======== START
--> 33ddbaa6b76
[1/2] STEP 6/7: RUN CGO_ENABLED=0 go build     -ldflags "${GO_LDFLAGS}"     -o /bin/karmada-aggregated-apiserver     cmd/aggregated-apiserver/main.go
--> 7c77f31bef4
[1/2] STEP 7/7: RUN date
Fri May  6 14:03:22 UTC 2022                         <======== END: takes 42s
--> b57cd1e7af4
[2/2] STEP 1/3: FROM alpine:3.15.1
[2/2] STEP 2/3: COPY --from=builder /bin/karmada-aggregated-apiserver /bin/karmada-aggregated-apiserver
--> dd2025ea7ca
[2/2] STEP 3/3: CMD ["/bin/karmada-aggregated-apiserver"]
[2/2] COMMIT swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest
--> 98ae42adff7
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest
98ae42adff74349a19aad2ded6ecbbf488f7e0f1f7de889f923374ff97a14ca8
Successfully built 98ae42adff74
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

________________________________________________________
Executed in   51.00 secs      fish           external
   usr time  355.42 millis    0.21 millis  355.20 millis
   sys time  900.51 millis    1.09 millis  899.43 millis

➜  karmada git:(master) ✗
➜  karmada git:(master) ✗ time VERSION=latest REGISTRY=swr.ap-southeast-1.myhuaweicloud.com/karmada hack/docker.sh karmada-controller-manager
Sending build context to Docker daemon  114.4MB
[1/2] STEP 1/7: FROM golang:1.17 AS builder
[1/2] STEP 2/7: WORKDIR /go/src/github.com/karmada-io/karmada
--> Using cache 6ae9f471a8110b2985aea152cdb319005351954aac915ec435349f604bccbba9
--> 6ae9f471a81
[1/2] STEP 3/7: COPY . /go/src/github.com/karmada-io/karmada
--> 977fa38908a
[1/2] STEP 4/7: ARG GO_LDFLAGS
--> be30ab342aa
[1/2] STEP 5/7: RUN date
Fri May  6 14:04:01 UTC 2022                         <======== START
--> 4a19032ebb8
[1/2] STEP 6/7: RUN CGO_ENABLED=0 go build     -ldflags "$GO_LDFLAGS"     -o /bin/karmada-controller-manager     cmd/controller-manager/controller-manager.go
--> 8501a6bbb80
[1/2] STEP 7/7: RUN date
Fri May  6 14:04:47 UTC 2022                         <======== END: takes 46s
--> 6b4d5f46bdc
[2/2] STEP 1/3: FROM alpine:3.15.1
[2/2] STEP 2/3: COPY --from=builder /bin/karmada-controller-manager /bin/karmada-controller-manager
--> 918f0dbf7aa
[2/2] STEP 3/3: CMD ["/bin/karmada-controller-manager"]
[2/2] COMMIT swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest
--> 6a6568686cb
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest
6a6568686cb95d50aa887e1c3bde3bf149bcdb57ebbd4f9a60b5069b6fb564ef
Successfully built 6a6568686cb9
Successfully tagged swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-controller-manager:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

________________________________________________________
Executed in   55.08 secs      fish           external
   usr time  349.99 millis    0.23 millis  349.77 millis
   sys time  854.03 millis    1.16 millis  852.87 millis

Probably guess: when build on host, GOCACHE helps reduce the building time.

ikaven1024 commented 2 years ago

(https://pkg.go.dev/cmd/go)

GOCACHE
    The directory where the go command will store cached
    information for reuse in future builds.

But i have no ideal how to share GOCAHE betweens docker build. There is no -v option when in docker build

========= TEST FOR GOCACHE ========

Step 1: go build karmada-aggregated-apiserver

$ time CGO_ENABLED=0 go build -o /bin/karmada-aggregated-apiserver     cmd/aggregated-apiserver/main.go

real    0m44.364s
user    1m30.948s
sys 0m15.446s

Step 2: go build karmada-controller-manager. It spends 15s

$ time CGO_ENABLED=0 go build -o /bin/karmada-controller-manager cmd/controller-manager/controller-manager.go

real    0m14.896s
user    0m25.336s
sys 0m5.824s

Step 3: clean GOCACHE and rebuild karmada-controller-manager again. It takes 51s

$ rm -rf `go env GOCACHE`
$ time CGO_ENABLED=0 go build -o /bin/karmada-controller-manager cmd/controller-manager/controller-manager.go

real    0m51.287s
user    1m45.802s
sys 0m19.012s
ikaven1024 commented 2 years ago

I didn't find an effective way. kubernetes did not build image in multi stage either. Maybe we shall revert？ @gy95 @RainbowMango

RainbowMango commented 2 years ago

No, we need multi-stage for multi-platform images. But, can we revert the single platform image(prefix with image-)?

ikaven1024 commented 2 years ago

multi-stage is not requirement for multi-platform. In kubernetes, build multi-platform image without multi-stage build.

https://github.com/kubernetes/kubernetes/blob/3c1a3dea59c31ab9f5e1ff2350eca719af3c2b2c/test/images/image-util.sh#L188

RainbowMango commented 2 years ago

Well, good to know. What're the images look like between different platforms? I mean if the image has the same name?

RainbowMango commented 2 years ago

I guess no, there is an os suffix. https://github.com/kubernetes/kubernetes/blob/c8b3b293c047ff9971cb5542a446c9c525877f50/test/images/image-util.sh#L77

And this image is built for testing.

wuyingjun-lucky commented 2 years ago

I guess no, there is an os suffix. https://github.com/kubernetes/kubernetes/blob/c8b3b293c047ff9971cb5542a446c9c525877f50/test/images/image-util.sh#L77

And this image is built for testing.

Multi-stage is not necessary. But go build in docker image seems necessary. When I clone the repo and make containers ,I found the containers can not work on my k8s because my working os is mac and my k8s is linux. But now it resolved. When I clone and deploy Kubesphere it will not happen. https://github.com/kubesphere/kubesphere/blob/master/build/ks-controller-manager/Dockerfile

ikaven1024 commented 2 years ago

In build.sh:

GOOS=linux GOARCH=${GOARCH} go build \
        -o linux/${GOARCH}/xxx \
        cmd/xxx/main.go

In Dockerfile:

FROM alpine:3.15.1
...
ADD ${{TARGETPLATFORM}}/xxx      /bin/xxx
...

Then we can build multi platform:

GOARCH=amd64  build.sh
GOARCH=arm64  build.sh
docker buildx build --platform="linux/amd64,linux/arm64" --push ....
ikaven1024 commented 2 years ago

Another schema。

Step1 build and push image with suffix -<arch>

docker buildx build --platform=linux/amd64 --push -t karmada/xxx-amd64:latest
docker buildx build --platform=linux/arm64 --push -t karmada/xxx-arm64:latest

Step2 merge manifest

docker manifest create karmada/xxx:latest \
        karmada/xxx-amd64:latest \
        karmada/xxx-amd64:latest

docker manifest push karmada/xxx:latest

But docker manifest is experimental. FYI: https://docs.docker.com/engine/reference/commandline/manifest/ https://github.com/kubernetes/kubernetes/blob/3c1a3dea59c31ab9f5e1ff2350eca719af3c2b2c/test/images/image-util.sh#L234-L249

ikaven1024 commented 2 years ago

/assign

wuyingjun-lucky commented 2 years ago

In build.sh:

GOOS=linux GOARCH=${GOARCH} go build \
      -o linux/${GOARCH}/xxx \
      cmd/xxx/main.go

In Dockerfile:

FROM alpine:3.15.1
...
ADD ${{TARGETPLATFORM}}/xxx      /bin/xxx
...

Then we can build multi platform:

GOARCH=amd64  build.sh
GOARCH=arm64  build.sh
docker buildx build --platform="linux/amd64,linux/arm64" --push ....

I do not aggress with the schema with reversion. I think we should focus on how to reduce the costs when building on docker building stage but not trying to move the building stage outside of docker context The dockerfile should define how to make the docker container precisely and anyone can use one command just as docker build to make the container. But in the schema. Firstly we should set os and make a binary , then find the binary and place it to a right position ,finally we make the container.