Closed henrjk closed 2 years ago
henrjk
commented
2 years ago I relaxed locally build-and-push-images.sh so it continues with the next image if one fails. The other build that fails is for the ods-node16-typescript-toolset.
See build-and-push-ts-image.log for the full log.
henrjk
commented
2 years ago Until finding a fix so that images run natively on M1 added fix so that helm and typescript images are build for platform linux/amd64 (see commit c45c2e68) for details.
Building for linux/amd64 on M1 is much slower than directly on a Intel based Mac:
However after a make install-ods-tasks-kind to deploy the tasks make test-tasks now the TestTaskODSDeployHelm PASSes. All other tests fail as they require bitbucket, sonarqube or nexus. See related issues linked above.
michaelsauter
commented
2 years ago Thanks for experimenting with this and reporting all of it. That doesn't look too promising :(
Let me try to summarise:
Correct?
I guess for (1) we could try to adapt the Dockerfile to make them build also for arm64. I think this should be doable. For example, we can install sops via go install go.mozilla.org/sops/v3/cmd/sops@v3.7.1 and Helm offers a arm64 release. The Dockerfile could automatically select the correct binary based on a build arg maybe? We could use ARGs aligned with https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope, even if we do not use buildx (locally we could but we also need to ensure it still works within the BuildConfig context).
For (2) we could open issues in the GitHub repos of the official images to see if support for arm64 is possible/planned? For example for Nexus, I found https://github.com/klo2k/nexus3-docker, which uses the official Dockerfile with a few alterations.
henrjk
commented
2 years ago @michaelsauter Your summary is correct. I agree with your proposals but not sure yet whether a build-arg is needed -- need to educate myself first a bit more. But perhaps others have more multi platform docker experience already..
michaelsauter
commented
2 years ago Cool. I've made the small suggested change in #364.
I am aware of the following four ways to build for multiple architectures:
uname to figure out the architecture dynamically (I don't particularly like it because it can be brittle and will duplicate logic between Dockerfiles)Dockerfile per architecture (I don't particularly like it because it increases maintenance effort)henrjk
commented
2 years ago https://github.com/docker-library/official-images#multiple-architectures has some suggestions regarding conditional in the RUN commands to target various userspace architectures, to quote dpkg --print-architecture or apk -print-arch to detect the userspace architecture. Only use uname for architecture detection when more accurate tools cannot be installed. It also links to https://github.com/docker-library/golang/blob/b879b60a7d94128c8fb5aea763cf31772495511d/1.16/buster/Dockerfile#L24-L68
henrjk
commented
2 years ago Also this could sometimes be of relevant: https://github.com/sixeyed/diamol/tree/master/ch16/lab
henrjk
commented
2 years ago Article Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide - Docker Blog has a section about predefined variables such as BUILDPLATFORM and TARGETPLATFORM.
henrjk
commented
2 years ago Tried out a fix for ods-deploy as shown in commit 1638d40 but getting this on the M1 mac:
→ ./build-and-push-images.sh --image helm
Building image localhost:5000/ods/ods-helm...
[+] Building 2.5s (21/21) FINISHED
=> [internal] load build definition from Dockerfile.helm 0.0s
=> => transferring dockerfile: 2.06kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal:8.4 0.6s
=> [internal] load metadata for registry.access.redhat.com/ubi8/go-toolset:1.16.12 0.6s
=> [builder 1/9] FROM registry.access.redhat.com/ubi8/go-toolset:1.16.12@sha256:287d341349331a19866a2862a55e0cc5c412f21a86ba80629f59579ef6a38c03 0.0s
=> [stage-1 1/7] FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4@sha256:c536d4c63253318fdfc1db499f8f4bb0881db7fbd6f3d1554b4d54c812f85cc7 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 5.93kB 0.0s
=> CACHED [stage-1 2/7] RUN microdnf install skopeo-1.4* git-2.27* tar-1.30* && microdnf clean all 0.0s
=> CACHED [builder 2/9] RUN mkdir -p /tmp/helm && cd /tmp && curl -LO https://get.helm.sh/helm-v3.5.2-linux-arm64.tar.gz && tar -zxvf helm-v3.5.2-linux-arm64.tar.gz -C /tmp/helm && mv /tmp/h 0.0s
=> CACHED [builder 3/9] COPY go.* . 0.0s
=> CACHED [builder 4/9] COPY cmd cmd 0.0s
=> CACHED [builder 5/9] COPY internal internal 0.0s
=> CACHED [builder 6/9] COPY pkg pkg 0.0s
=> CACHED [builder 7/9] RUN cd cmd/deploy-with-helm && CGO_ENABLED=0 go build -o /usr/local/bin/deploy-with-helm 0.0s
=> CACHED [builder 8/9] RUN go install go.mozilla.org/sops/v3/cmd/sops@v3.7.1 && sops --version 0.0s
=> CACHED [builder 9/9] RUN go install filippo.io/age/cmd/...@v1.0.0 && age --version 0.0s
=> CACHED [stage-1 3/7] COPY --from=builder /usr/local/bin/deploy-with-helm /usr/local/bin/deploy-with-helm 0.0s
=> CACHED [stage-1 4/7] COPY --from=builder /usr/local/bin/helm /usr/local/bin/helm 0.0s
=> CACHED [stage-1 5/7] COPY --from=builder /usr/local/bin/sops /usr/local/bin/sops 0.0s
=> CACHED [stage-1 6/7] COPY --from=builder /usr/local/bin/age /usr/local/bin/age 0.0s
=> ERROR [stage-1 7/7] RUN mkdir -p $HELM_PLUGINS && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/databus23/helm-diff --version v${HELM_PLUGIN_DIFF_VERSION} && HELM_DATA_ 1.8s
------
> [stage-1 7/7] RUN mkdir -p $HELM_PLUGINS && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/databus23/helm-diff --version v${HELM_PLUGIN_DIFF_VERSION} && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/jkroepke/helm-secrets --version v${HELM_PLUGIN_SECRETS_VERSION} && ls -lah $HELM_PLUGINS && sops --version && age --version:
#21 1.722 No prebuild binary for linux-arm64.
#21 1.722 Failed to install helm-diff
#21 1.722 For support, go to https://github.com/databus23/helm-diff.
#21 1.722 Error: plugin install hook for "diff" exited with error
------
executor failed running [/bin/sh -c mkdir -p $HELM_PLUGINS && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/databus23/helm-diff --version v${HELM_PLUGIN_DIFF_VERSION} && HELM_DATA_HOME=${HELM_PLUGINS%/*} helm plugin install https://github.com/jkroepke/helm-secrets --version v${HELM_PLUGIN_SECRETS_VERSION} && ls -lah $HELM_PLUGINS && sops --version && age --version]: exit code: 1
For helm-diff there is already a PR at https://github.com/databus23/helm-diff/pull/267
michaelsauter
commented
2 years ago @henrjk Nice. Actually, this is resolved via a different PR in master? https://github.com/databus23/helm-diff/pull/294
UPDATE: Was too quick, 294 is actually fixing something else ...
henrjk
commented
2 years ago The build issue in the typescript tooling is because Chrome is not available on arm64, except for macos while our image needs it for linux. It would be good if FE/cypress experts see whether we could switch to Chromium which would/should be available on linux/arm64 as well.
Details are in the log reported in https://github.com/opendevstack/ods-pipeline/issues/360#issuecomment-1002955210
henrjk
commented
2 years ago @henninggross @netzartist @m-apsolon any thoughts on whether using Chromium is viable?
This issue could be relevant.
For me make build-and-push-images fails. Below is an excerpt
The above is from the second run. The full log from the first run is at build-and-push-images.log
Here is some info about the machine
I believe docker build builds for the os platform by default (arm64) and succeeds for ods-buildah, ods-finish, ods-go-toolset, ods-gradle-toolset but fails for ods-helm