search

alpine-docker / k8s

All-In-One Kubernetes tools (kubectl, helm, iam-authenticator, eksctl, etc)
MIT License
200 stars 89 forks source link

Multi arch build attempt 2 #52

Closed chenghu2 closed 1 year ago

chenghu2 commented 1 year ago

reattempt of #51

chenghu2 commented 1 year ago

@ozbillwang sorry I didn't read your issue comment carefully, I have updated the circleci config based on https://github.com/alpine-docker/multi-arch-docker-build-operator/blob/23c8042c6ce97891cd4341fd11db215ac29c0a1d/.circleci/config.yml#L5-L10 I also updated the build script based on your previous work of https://github.com/alpine-docker/helm/pull/37 Please let me know if you have more feedbacks, I'm happy to address.

ozbillwang commented 1 year ago

nice job, let me review

ozbillwang commented 1 year ago

Need these lines to set arch name in Dockerfile

https://github.com/alpine-docker/helm/blob/master/Dockerfile#L11-L18

RUN case `uname -m` in \
        x86_64) ARCH=amd64; ;; \
        armv7l) ARCH=arm; ;; \
        aarch64) ARCH=arm64; ;; \
        ppc64le) ARCH=ppc64le; ;; \
        s390x) ARCH=s390x; ;; \
        *) echo "un-supported arch, exit ..."; exit 1; ;; \
    esac && \
chenghu2 commented 1 year ago

@ozbillwang thanks for checking, I was actually testing the new platforms I added. I think for this image, we can only use --platform=linux/amd64,linux/arm64 because

 > [linux/s390x  6/17] RUN helm plugin install https://github.com/databus23/helm-diff && rm -rf /tmp/helm-*:
#0 2.445 mkdir: can't create directory '': No such file or directory
#0 2.484 No prebuild binary for linux-s390x.
#0 2.485 Failed to install helm-diff
#0 2.486        For support, go to https://github.com/databus23/helm-diff.
#0 2.489 Error: plugin install hook for "diff" exited with error

The same error happens for linux/arm/v7,linux/arm64/v8,linux/arm/v6,linux/ppc64le as well. Curious what's your thought on this?

ozbillwang commented 1 year ago

Good points, this will be a block to support other platforms.

Let's clean other platforms except linux/amd64,linux/arm64

ozbillwang commented 1 year ago

I am testing your Dockerfile as well, here is the patch.

I do not fully test it yet, the change includes to export a variable ARCH to /envfile and re-use by other commands

The reason is, in Dockerfile, new generated varaibles can't be re-used in next RUN commands.

Refer this for solution: https://stackoverflow.com/a/68290395/3671801

diff --git a/Dockerfile b/Dockerfile
index f947a64..134e8dc 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,6 +1,6 @@
 FROM alpine

-ARG TARGETARCH
+ARG ARCH

 # Ignore to update versions here
 # docker build --no-cache --build-arg KUBECTL_VERSION=${tag} --build-arg HELM_VERSION=${helm} --build-arg KUSTOMIZE_VERSION=${kustomize_version} -t ${image}:${tag} .
@@ -11,13 +11,23 @@ ARG KUBESEAL_VERSION=0.18.1

 # Install helm (latest release)
 # ENV BASE_URL="https://storage.googleapis.com/kubernetes-helm"
-ENV BASE_URL="https://get.helm.sh"
-ENV TAR_FILE="helm-v${HELM_VERSION}-linux-${TARGETARCH}.tar.gz"
-RUN apk add --update --no-cache curl ca-certificates bash git && \
-    curl -sL ${BASE_URL}/${TAR_FILE} | tar -xvz && \
-    mv linux-${TARGETARCH}/helm /usr/bin/helm && \
+RUN case `uname -m` in \
+        x86_64) ARCH=amd64; ;; \
+        armv7l) ARCH=arm; ;; \
+        aarch64) ARCH=arm64; ;; \
+        ppc64le) ARCH=ppc64le; ;; \
+        s390x) ARCH=s390x; ;; \
+        *) echo "un-supported arch, exit ..."; exit 1; ;; \
+    esac && \
+    echo "export ARCH=$ARCH" > /envfile && \
+    cat /envfile
+
+RUN . /envfile && echo $ARCH && \
+    apk add --update --no-cache curl ca-certificates bash git && \
+    curl -sL https://get.helm.sh/helm-v${HELM_VERSION}-linux-${ARCH}.tar.gz | tar -xvz && \
+    mv linux-${ARCH}/helm /usr/bin/helm && \
     chmod +x /usr/bin/helm && \
-    rm -rf linux-${TARGETARCH}
+    rm -rf linux-${ARCH}

 # add helm-diff
 RUN helm plugin install https://github.com/databus23/helm-diff && rm -rf /tmp/helm-*
@@ -31,19 +41,22 @@ RUN helm plugin install https://github.com/chartmuseum/helm-push && \
     /root/.local/share/helm/plugins/helm-push/testdata \
     /root/.cache/helm/plugins/https-github.com-chartmuseum-helm-push/testdata

-# Install kubectl (same version of aws esk)
-RUN curl -sLO https://storage.googleapis.com/kubernetes-release/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl && \
+# Install kubectl
+RUN . /envfile && echo $ARCH && \
+    curl -sLO https://storage.googleapis.com/kubernetes-release/release/v${KUBECTL_VERSION}/bin/linux/${ARCH}/kubectl && \
     mv kubectl /usr/bin/kubectl && \
     chmod +x /usr/bin/kubectl

 # Install kustomize (latest release)
-RUN curl -sLO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_${TARGETARCH}.tar.gz && \
-    tar xvzf kustomize_${KUSTOMIZE_VERSION}_linux_${TARGETARCH}.tar.gz && \
+RUN . /envfile && echo $ARCH && \
+    curl -sLO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_${ARCH}.tar.gz && \
+    tar xvzf kustomize_${KUSTOMIZE_VERSION}_linux_${ARCH}.tar.gz && \
     mv kustomize /usr/bin/kustomize && \
     chmod +x /usr/bin/kustomize

 # Install eksctl (latest version)
-RUN curl -sL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_${TARGETARCH}.tar.gz" | tar xz -C /tmp && \
+RUN . /envfile && echo $ARCH && \
+    curl -sL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_${ARCH}.tar.gz" | tar xz -C /tmp && \
     mv /tmp/eksctl /usr/bin && \
     chmod +x /usr/bin/eksctl

@@ -59,15 +72,17 @@ RUN apk add --update --no-cache jq yq

 # https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html
 # Install aws-iam-authenticator (latest version)
-RUN authenticator=$(curl -fs https://api.github.com/repos/kubernetes-sigs/aws-iam-authenticator/releases/latest | jq --raw-output '.name' | sed 's/^v//') && \
-    curl -fL https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v${authenticator}/aws-iam-authenticator_${authenticator}_linux_${TARGETARCH} -o /usr/bin/aws-iam-authenticator && \
+RUN . /envfile && echo $ARCH && \
+    authenticator=$(curl -fs https://api.github.com/repos/kubernetes-sigs/aws-iam-authenticator/releases/latest | jq --raw-output '.name' | sed 's/^v//') && \
+    curl -fL https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v${authenticator}/aws-iam-authenticator_${authenticator}_linux_${ARCH} -o /usr/bin/aws-iam-authenticator && \
     chmod +x /usr/bin/aws-iam-authenticator

 # Install for envsubst
 RUN apk add --update --no-cache gettext

 # Install kubeseal
-RUN curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-${TARGETARCH}.tar.gz -o - | tar xz -C /usr/bin/ && \
+RUN . /envfile && echo $ARCH && \
+    curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-${ARCH}.tar.gz -o - | tar xz -C /usr/bin/ && \
     chmod +x /usr/bin/kubeseal

 WORKDIR /apps

Save it locally, such as patch-file, you can patch it in your environment and run the build again. 

git patch patch-file
chenghu2 commented 1 year ago

@ozbillwang I tested your patch, but the env variable is not working for me 🤔 . In my latest commit, I droppred other platforms except linux/amd64,linux/arm64 and it builds locally for me, I also added a note in the readme.

ozbillwang commented 1 year ago

I updated the patch, should be fine now. can you test again.

diff --git a/Dockerfile b/Dockerfile
index f947a64..134e8dc 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,6 +1,6 @@
 FROM alpine

-ARG TARGETARCH
+ARG ARCH

 # Ignore to update versions here
 # docker build --no-cache --build-arg KUBECTL_VERSION=${tag} --build-arg HELM_VERSION=${helm} --build-arg KUSTOMIZE_VERSION=${kustomize_version} -t ${image}:${tag} .
@@ -11,13 +11,23 @@ ARG KUBESEAL_VERSION=0.18.1

 # Install helm (latest release)
 # ENV BASE_URL="https://storage.googleapis.com/kubernetes-helm"
-ENV BASE_URL="https://get.helm.sh"
-ENV TAR_FILE="helm-v${HELM_VERSION}-linux-${TARGETARCH}.tar.gz"
-RUN apk add --update --no-cache curl ca-certificates bash git && \
-    curl -sL ${BASE_URL}/${TAR_FILE} | tar -xvz && \
-    mv linux-${TARGETARCH}/helm /usr/bin/helm && \
+RUN case `uname -m` in \
+        x86_64) ARCH=amd64; ;; \
+        armv7l) ARCH=arm; ;; \
+        aarch64) ARCH=arm64; ;; \
+        ppc64le) ARCH=ppc64le; ;; \
+        s390x) ARCH=s390x; ;; \
+        *) echo "un-supported arch, exit ..."; exit 1; ;; \
+    esac && \
+    echo "export ARCH=$ARCH" > /envfile && \
+    cat /envfile
+
+RUN . /envfile && echo $ARCH && \
+    apk add --update --no-cache curl ca-certificates bash git && \
+    curl -sL https://get.helm.sh/helm-v${HELM_VERSION}-linux-${ARCH}.tar.gz | tar -xvz && \
+    mv linux-${ARCH}/helm /usr/bin/helm && \
     chmod +x /usr/bin/helm && \
-    rm -rf linux-${TARGETARCH}
+    rm -rf linux-${ARCH}

 # add helm-diff
 RUN helm plugin install https://github.com/databus23/helm-diff && rm -rf /tmp/helm-*
@@ -31,19 +41,22 @@ RUN helm plugin install https://github.com/chartmuseum/helm-push && \
     /root/.local/share/helm/plugins/helm-push/testdata \
     /root/.cache/helm/plugins/https-github.com-chartmuseum-helm-push/testdata

-# Install kubectl (same version of aws esk)
-RUN curl -sLO https://storage.googleapis.com/kubernetes-release/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl && \
+# Install kubectl
+RUN . /envfile && echo $ARCH && \
+    curl -sLO https://storage.googleapis.com/kubernetes-release/release/v${KUBECTL_VERSION}/bin/linux/${ARCH}/kubectl && \
     mv kubectl /usr/bin/kubectl && \
     chmod +x /usr/bin/kubectl

 # Install kustomize (latest release)
-RUN curl -sLO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_${TARGETARCH}.tar.gz && \
-    tar xvzf kustomize_${KUSTOMIZE_VERSION}_linux_${TARGETARCH}.tar.gz && \
+RUN . /envfile && echo $ARCH && \
+    curl -sLO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_${ARCH}.tar.gz && \
+    tar xvzf kustomize_${KUSTOMIZE_VERSION}_linux_${ARCH}.tar.gz && \
     mv kustomize /usr/bin/kustomize && \
     chmod +x /usr/bin/kustomize

 # Install eksctl (latest version)
-RUN curl -sL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_${TARGETARCH}.tar.gz" | tar xz -C /tmp && \
+RUN . /envfile && echo $ARCH && \
+    curl -sL "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_${ARCH}.tar.gz" | tar xz -C /tmp && \
     mv /tmp/eksctl /usr/bin && \
     chmod +x /usr/bin/eksctl

@@ -59,15 +72,17 @@ RUN apk add --update --no-cache jq yq

 # https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html
 # Install aws-iam-authenticator (latest version)
-RUN authenticator=$(curl -fs https://api.github.com/repos/kubernetes-sigs/aws-iam-authenticator/releases/latest | jq --raw-output '.name' | sed 's/^v//') && \
-    curl -fL https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v${authenticator}/aws-iam-authenticator_${authenticator}_linux_${TARGETARCH} -o /usr/bin/aws-iam-authenticator && \
+RUN . /envfile && echo $ARCH && \
+    authenticator=$(curl -fs https://api.github.com/repos/kubernetes-sigs/aws-iam-authenticator/releases/latest | jq --raw-output '.name' | sed 's/^v//') && \
+    curl -fL https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v${authenticator}/aws-iam-authenticator_${authenticator}_linux_${ARCH} -o /usr/bin/aws-iam-authenticator && \
     chmod +x /usr/bin/aws-iam-authenticator

 # Install for envsubst
 RUN apk add --update --no-cache gettext

 # Install kubeseal
-RUN curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-${TARGETARCH}.tar.gz -o - | tar xz -C /usr/bin/ && \
+RUN . /envfile && echo $ARCH && \
+    curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-${ARCH}.tar.gz -o - | tar xz -C /usr/bin/ && \
     chmod +x /usr/bin/kubeseal

 WORKDIR /apps
chenghu2 commented 1 year ago

@ozbillwang Thanks, I just committed the patch.

chenghu2 commented 1 year ago

@ozbillwang thanks for fixing the CI for me! However, I don't see the docker build actually running at the master CI based on the CI log

ozbillwang commented 1 year ago

the change is not successful, I revert it back https://github.com/alpine-docker/k8s/pull/53