K3s Support

[adamhurm]

I just noticed that you released an update and added arm/arm64 support That's awesome, thank you so much.

I attempted deploying on my arm64 k3s cluster and had most of the pods running, but I unfortunately still ran into CrashLoopBackOff state for two pods. The two pods both had an exec format error message.

[madhuakula]

Thanks a million for testing ARM images and sharing results @adamhurm. I don't have pi or something to test ARM, thanks for testing. It looks like a container image issue because the error I can see is Golang's different compiled (AMD64) binary.

Let me fix this in Dockerfiles and release the new images for ARM 😊

Thanks a ton once again for the testing and feedback 🙏

[madhuakula]

Thanks to both (@phpsystems and @adamhurm) of you, I hope these bugs are fixed in #48. Please do let me know if you still face any issues 🙏

[adamhurm]

Hi @madhuakula, I am unfortunately still getting the exec format error for build-code and system-monitor.

I attempted to manually build the build-code image since it has a prebuilt app.tar.gz package inside. My process was a bit complicated because I am running k3s without docker. I installed docker separately to build the image and import it using ctr.

$ cd infrastructure/build-code
$ tar xzvf app.tar.gz
$ cd app

# unable to run the binary bundled by default
$ ./app
-bash: ./app: cannot execute binary file: Exec format error

# building locally
$ GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build

# now I can run the ./app binary
$ ./app
 ┌───────────────────────────────────────────────────┐ 
 │                    Fiber v2.1.4                   │ 
 │               http://127.0.0.1:3000               │ 
 │                                                   │ 
 │ Handlers ............. 5  Threads ............. 4 │ 
 │ Prefork ....... Disabled  PID ........... 1459465 │ 
 └───────────────────────────────────────────────────┘ 
^C

# repackaging
$ cd ..
$ tar czvf app.tar.gz app

# building image and exporting
$ docker build -t madhuakula/k8s-goat-build-code .
$ docker save -o build-code.tar madhuakula/k8s-goat-build-code

# making sure the old images are pruned
$ crictl rmi --prune
Deleted: docker.io/madhuakula/k8s-goat-build-code:latest

# importing new, locally-built image
$ k3s ctr images import build-code.tar
$ crictl images ls
IMAGE                                        TAG                 IMAGE ID            SIZE
docker.io/madhuakula/k8s-goat-build-code     latest              1d7c8ea517d21       132MB

# set imagePullPolicy to Never so I can test my local build
$ grep imagePullPolicy scenarios/build-code/deployment.yaml -C 2
      - name: build-code
        image: madhuakula/k8s-goat-build-code
        imagePullPolicy: Never
        resources:
          limits:

$ kubectl apply -f scenarios/build-code/deployment.yaml
$ kubectl get pods
NAME                                         READY   STATUS              RESTARTS      AGE
build-code-deployment-64d576d8cc-8gsqw       0/1     Error               2 (30s ago)   39s
$ kubectl logs build-code-deployment-64d576d8cc-8gsqw
standard_init_linux.go:228: exec user process caused: exec format error

[adamhurm]

Update: I think the recent Dockerfile changes fixed the system-monitor issue. I can now get system-monitor to Running status, but build-code still fails with exec format error

[madhuakula]

Oops!

Gotcha. I think I am building the Golang binary with amd64 and putting in an app.tar.gz file.

Let me fix this tomorrow. Good catch 👍

[adamhurm]

Update: I just learned that you can't really run local containers with k3s in containerd mode, which I wasted a bit too much time on. 😅 I created a Docker Hub account and pushed my local image build there. It worked perfectly on the first try. 🙃

---

This is what I posted earlier when I was troubleshooting:

Curiously, my locally compiled binary works when I run the image with docker and containerd. It will run until I kill the process. However, it's failing when I run kubectl.

pi@node0:~/kubernetes-goat $ kubectl apply -f scenarios/build-code/deployment.yaml
deployment.apps/build-code-deployment created
service/build-code-service created
pi@node0:~/kubernetes-goat $ kubectl get pods
NAME                                     READY   STATUS   RESTARTS      AGE
build-code-deployment-5f66d7bfff-cbp46   0/1     Error    5 (99s ago)   3m42s
pi@node0:~/kubernetes-goat $ kubectl logs build-code-deployment-5f66d7bfff-cbp46
standard_init_linux.go:228: exec user process caused: exec format error
pi@node0:~/kubernetes-goat $ sudo docker run madhuakula/k8s-goat-build-code

 ┌───────────────────────────────────────────────────┐
 │                    Fiber v2.1.4                   │
 │               http://127.0.0.1:3000               │
 │                                                   │
 │ Handlers ............. 5  Threads ............. 4 │
 │ Prefork ....... Disabled  PID ................. 1 │
 └───────────────────────────────────────────────────┘

^C
pi@node0:~/kubernetes-goat $ sudo k3s ctr run docker.io/madhuakula/k8s-goat-build-code:latest v1

 ┌───────────────────────────────────────────────────┐
 │                    Fiber v2.1.4                   │
 │               http://127.0.0.1:3000               │
 │                                                   │
 │ Handlers ............. 5  Threads ............. 4 │
 │ Prefork ....... Disabled  PID ................. 1 │
 └───────────────────────────────────────────────────┘

^C
pi@node0:~/kubernetes-goat $

I'm not really sure what's going on here. I'll try to research some more and come back.

[adamhurm]

One final update for anyone that comes across this Issue: @madhuakula has updated the k8s-goat-build-code image on Docker Hub and the project now has fully-functional arm64 support!

[madhuakula]

Thank you @adamhurm for reporting this bug :)

This is the fix for the reference https://github.com/madhuakula/kubernetes-goat/commit/37cb580ec7e8f915d9d735a391d9d324721fdb71

[phpsystems]

A note on resources: I ran out of them (memory) on 2x Raspberry pi 2Ws workernodes.

Good excuse to fix the other 2....

[adamhurm]

@phpsystems Good to know. I didn’t run into any resources constraints but my cluster has 4x Raspberry Pi 4 Model B (4GB RAM).

If you need me to do any comparison benchmarks I also have a single Pi 2W that I could try using.