ARM64 image is not working

Summary

Image for arm64 architecture is not working.

Steps to reproduce

Add Mattermost Helm repo :

helm repo add mattermost https://helm.mattermost.com
helm repo update

Deploy the operator on arm64 architecture with Helm :

helm upgrade \
  --install \
  --create-namespace \
  --namespace mattermost \
  --set-string mattermostOperator.image.tag="master" \
  mattermost-operator mattermost/mattermost-operator

Notes: The image tagged master is available for both amd64 and arm64 (cf. https://hub.docker.com/layers/mattermost/mattermost-operator/master/images/sha256-46f4463f4fa537f0721b289b2430df1870c816ffdda554f75b699fcccfc04f27).

Version used :

Mattermost operator image tagged master
Mattermost not deployed at the time

Environment :

Hosts
- Raspberry Pi 4 model B - 8GB
- Kernel version 6.1.21-v8+
- OS Debian GNU/Linux 11 (bullseye)
Softwares
- K3S version v1.27.3+k3s1

Expected behavior

Because docker image is now build for arm64 architecture the mattermost-operator container should be able to start in that kind of hosts as it does for amd64.

Observed behavior (that appears unintentional)

kubectl -n mattermost get pod                                          
NAME                                   READY   STATUS             RESTARTS       AGE
mattermost-operator-57649f4c95-pl4wv   0/1     CrashLoopBackOff   15 (67s ago)   53m

kubectl -n mattermost describe pod/mattermost-operator-57649f4c95-pl4wv                               

Name:             mattermost-operator-57649f4c95-pl4wv
Namespace:        mattermost
Priority:         0
Service Account:  mattermost-operator
Node:             pi3/192.168.0.103
Start Time:       Sat, 29 Jul 2023 15:43:30 +0200
Labels:           app.kubernetes.io/instance=mattermost-operator
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=mattermost-operator
                  helm.sh/chart=mattermost-operator-0.3.5
                  name=mattermost-operator
                  pod-template-hash=57649f4c95
Annotations:      <none>
Status:           Running
IP:               10.42.0.20
IPs:
  IP:           10.42.0.20
Controlled By:  ReplicaSet/mattermost-operator-57649f4c95
Containers:
  mattermost-operator:
    Container ID:  containerd://9cbcdba0f16f8463f3a87117f02055577010f20668986c591806d02b13fc885b
    Image:         docker.io/mattermost/mattermost-operator:master
    Image ID:      docker.io/mattermost/mattermost-operator@sha256:038faf39900e5558cd865c73c5e6cf26fff30d447373756d815d1a87b9b83951
    Port:          8383/TCP
    Host Port:     0/TCP
    Command:
      /mattermost-operator
    Args:
      --enable-leader-election
      --metrics-addr=0.0.0.0:8383
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 29 Jul 2023 16:35:27 +0200
      Finished:     Sat, 29 Jul 2023 16:35:27 +0200
    Ready:          False
    Restart Count:  15
    Environment:
      MAX_RECONCILING_INSTALLATIONS:  20
      MAX_RECONCILE_CONCURRENCY:      10
      REQUEUE_ON_LIMIT_DELAY:         20s
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jdv5r (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-jdv5r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  55m                  default-scheduler  Successfully assigned mattermost/mattermost-operator-57649f4c95-pl4wv to pi3
  Normal   Pulled     55m                  kubelet            Successfully pulled image "docker.io/mattermost/mattermost-operator:master" in 872.745291ms (872.774346ms including waiting)
  Normal   Pulled     55m                  kubelet            Successfully pulled image "docker.io/mattermost/mattermost-operator:master" in 910.09848ms (910.126016ms including waiting)
  Normal   Pulled     55m                  kubelet            Successfully pulled image "docker.io/mattermost/mattermost-operator:master" in 891.489988ms (891.518284ms including waiting)
  Normal   Created    54m (x4 over 55m)    kubelet            Created container mattermost-operator
  Normal   Started    54m (x4 over 55m)    kubelet            Started container mattermost-operator
  Normal   Pulled     54m                  kubelet            Successfully pulled image "docker.io/mattermost/mattermost-operator:master" in 901.788315ms (902.006977ms including waiting)
  Normal   Pulling    54m (x5 over 55m)    kubelet            Pulling image "docker.io/mattermost/mattermost-operator:master"
  Normal   Pulled     54m                  kubelet            Successfully pulled image "docker.io/mattermost/mattermost-operator:master" in 872.718637ms (872.742265ms including waiting)
  Warning  BackOff    38s (x253 over 55m)  kubelet            Back-off restarting failed container mattermost-operator in pod mattermost-operator-57649f4c95-pl4wv_mattermost(b15d1eba-6e4a-4d95-8db5-c98f29ec1d98)

Possible fixes

Maybe add a step to setup QEMU emulator in .github/workflows/ci.yml and .github/workflows/cd.yml could fix the issue :

# ...
      - name: ci/setup-buildx
        uses: docker/setup-buildx-action@15c905b16b06416d2086efa066dd8e3a35cc7f98 # v2.4.0
        with:
          version: v0.7.1

      # Add this
      - name: Set up QEMU (for multi platform build)
        uses: docker/setup-qemu-action@2b82ce82d56a2a04d2637cd93a637ae1b359c0a7 # v2.2.0

      - name: ci/docker-login
        uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a # v2.1.0
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
# ...

Multi platform build docs :

https://docs.docker.com/build/ci/github-actions/multi-platform/

https://github.com/docker/setup-qemu-action

Notes: An example of a multi arch build I'm using in a repository at work (cf. https://github.com/cloud-pi-native/console/blob/main/.github/workflows/release.yml).

After building the mattermost-operator image locally with --platform=linux/arm64, it still doesn't seem to work.

After taking a closer look at the build system, I discovered that the binary contained in the docker image was not built for the correct architecture (cf. https://github.com/mattermost/mattermost-operator/blob/master/Makefile#L122).

I made a PR with the corrections available here.

I built the image with these corrections and pushed it into my personal registry, with the command :

OPERATOR_IMAGE_NAME=registry.ohmlab.fr/library/mattermost-operator OPERATOR_IMAGE_TAG=new-master make buildx-image

Note: The image is available with the command :
 docker pull registry.ohmlab.fr/library/mattermost-operator:new-master

Next, I deployed the operator via Helm with the previously corrected image using the command :

helm upgrade \
  --install \
  --create-namespace \
  --namespace mattermost \
  --set-string mattermostOperator.image.repository="registry.ohmlab.fr/library/mattermost-operator" \
  --set-string mattermostOperator.image.tag="new-master" \
  mattermost-operator mattermost/mattermost-operator

And it seems to work:

kubectl -n mattermost get pod
NAME                                   READY   STATUS    RESTARTS   AGE
mattermost-operator-56bcbf47bf-hq6jb   1/1     Running   0          58s

kubectl -n mattermost describe pod/mattermost-operator-56bcbf47bf-hq6jb 
Name:             mattermost-operator-56bcbf47bf-hq6jb
Namespace:        mattermost
Priority:         0
Service Account:  mattermost-operator
Node:             pi3/192.168.0.103
Start Time:       Thu, 03 Aug 2023 01:23:36 +0200
Labels:           app.kubernetes.io/instance=mattermost-operator
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=mattermost-operator
                  helm.sh/chart=mattermost-operator-0.3.5
                  name=mattermost-operator
                  pod-template-hash=56bcbf47bf
Annotations:      <none>
Status:           Running
IP:               10.42.0.21
IPs:
  IP:           10.42.0.21
Controlled By:  ReplicaSet/mattermost-operator-56bcbf47bf
Containers:
  mattermost-operator:
    Container ID:  containerd://d8dc3afaf5164060ee031dce747b99c43ee54b52db76a098e93bb64f3ac23cc7
    Image:         registry.ohmlab.fr/library/mattermost-operator:new-master
    Image ID:      registry.ohmlab.fr/library/mattermost-operator@sha256:c88c42b9945d0b1c555a165405db88c97c6a165e2461993376a2f0ab4c8b95b7
    Port:          8383/TCP
    Host Port:     0/TCP
    Command:
      /mattermost-operator
    Args:
      --enable-leader-election
      --metrics-addr=0.0.0.0:8383
    State:          Running
      Started:      Thu, 03 Aug 2023 01:23:43 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      MAX_RECONCILING_INSTALLATIONS:  20
      MAX_RECONCILE_CONCURRENCY:      10
      REQUEUE_ON_LIMIT_DELAY:         20s
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5mk8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-j5mk8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  64s   default-scheduler  Successfully assigned mattermost/mattermost-operator-56bcbf47bf-hq6jb to pi3
  Normal  Pulling    64s   kubelet            Pulling image "registry.ohmlab.fr/library/mattermost-operator:new-master"
  Normal  Pulled     58s   kubelet            Successfully pulled image "registry.ohmlab.fr/library/mattermost-operator:new-master" in 5.767722162s (5.767754643s including waiting)
  Normal  Created    58s   kubelet            Created container mattermost-operator
  Normal  Started    58s   kubelet            Started container mattermost-operator

mattermost / mattermost-operator