Handler arm build does not work on v0.77.0

[Gentoli]

What happened: In the arm image, manager in handler calls nmstatectl, but nmstatectl the bundled binary is not executable on arm.

What you expected to happen: handler pods runs.

How to reproduce it (as minimally and precisely as possible):

• Install v0.77.0 (on cluster with arm nodes)

• Remove kubernetes.io/arch: amd64 node selector by overriding NMState CR with:

  apiVersion: nmstate.io/v1
  kind: NMState
  metadata:
  name: nmstate
  spec:
  nodeSelector:
    kubernetes.io/os: linux

• handler pods on arm node will crash

Anything else we need to know?:

• looks like os commands (e.g. sh, bash) from the image is also not arm
• latest image with working os commands is v0.72.0. manager works on this version with (RELATED_IMAGE_HANDLER_IMAGE=quay.io/nmstate/kubernetes-nmstate-handler:v0.72.0 set on the operator)

Environment:

• NodeNetworkState on affected nodes (use kubectl get nodenetworkstate <node_name> -o yaml): N/A
• Problematic NodeNetworkConfigurationPolicy: N/A
• kubernetes-nmstate image (use kubectl get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'): quay.io/nmstate/kubernetes-nmstate-handler:v0.77.0
• NetworkManager version (use nmcli --version): nmcli tool, version 1.40.0-5.0.1.el8_7
• Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"54b89e5be0a03b4b1e215ef9b48487a74fae5a9e", GitTreeState:"clean", BuildDate:"2022-11-16T06:24:32Z", GoVersion:"go1.19.3 A1727 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
• OS (e.g. from /etc/os-release): oracle linux 8
• Others:

Logs:

{"level":"info","ts":"2023-04-19T08:56:52.376Z","logger":"setup","msg":"Try to take exclusive lock on file: /var/k8s_nmstate/handler_lock"}
{"level":"info","ts":"2023-04-19T08:56:52.376Z","logger":"setup","msg":"Successfully took nmstate exclusive lock"}
{"level":"error","ts":"2023-04-19T08:56:52.410Z","logger":"setup","msg":"failed checking nmstatectl health","error":"failed to execute nmstatectl show: 'fork/exec /usr/bin/nmstatectl: exec format error' '' ''","stacktrace":"main.checkNmstateIsWorking\n\t/workdir/cmd/handler/main.go:245\nmain.mainHandler\n\t/workdir/cmd/handler/main.go:150\nmain.main\n\t/workdir/cmd/handler/main.go:82\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250"}

[qinqon]

Clearly we are building the x86_64 versions when we do "ARCH=arm64"

Running transaction
  Preparing        :                                                        1/1 
  Installing       : nmstate-2.2.10-3.el9.x86_64                            1/1 
  Running scriptlet: nmstate-2.2.10-3.el9.x86_64                            1/1 
  Verifying        : nmstate-2.2.10-3.el9.x86_64                            1/1

[Gentoli]

@qinqon I think the base image is using the wrong architecture, that's why the package manager installs the amd64 version

[qinqon]

@Gentoli found the issue https://github.com/nmstate/kubernetes-nmstate/pull/1188

[qinqon]

@Gentoli can you verify v0.78.0 works fine at arm64 now ?

[Gentoli]

@qinqon Thanks for checking. Looks like v0.78.0 works 🎉

~ % kubectl exec nmstate-handler-hrgqz -n nmstate -it -- sh
sh-5.1# nmstatectl 
error: 'nmstatectl' requires a subcommand but one was not provided

USAGE:
    nmstatectl [OPTIONS] <SUBCOMMAND>

For more information try --help
sh-5.1# uname -a
Linux oke-c46r437jxda-n6xzpyovv4a-sk5oftixwhq-0 5.15.0-6.80.3.1.el8uek.aarch64 #2 SMP Tue Jan 10 12:26:56 PST 2023 aarch64 aarch64 aarch64 GNU/Linux
sh-5.1# 
exit
~ % kubectl get pod nmstate-handler-hrgqz -n nmstate -o yaml | grep image
    image: quay.io/nmstate/kubernetes-nmstate-handler:v0.78.0
    imagePullPolicy: Always
    image: quay.io/nmstate/kubernetes-nmstate-handler:v0.78.0
    imageID: quay.io/nmstate/kubernetes-nmstate-handler@sha256:92437bdead30e63a90a843d3430eceb39ab15fc9a338368ce1ba9ba42b9139cc