Add tests for Onprem k8s providers

[daemon1024]

We should test KubeArmor periodically and on release on the onprem k8s solutions like kind, minikube, microk8s and k0s.

The tests should run a smoke test to verify if basic kubearmor functionality works or not.

[MeenuyD]

Hello @daemon1024 I would like to work on this issue can please guide me on how to get started

[daemon1024]

@MeenuyD Thanks a lot for the the interest.

For starters, you can try setting up a minikube/microk8s environment on a Ubuntu Machine.

Follow https://github.com/kubearmor/KubeArmor/blob/main/getting-started/deployment_guide.md to play around with KubeArmor.

Once you are confident with KubeArmor you can try executing the Smoke Tests based on https://github.com/kubearmor/KubeArmor/tree/main/tests#how-to-execute-testsuites

Now the task of this issue is to automate these all. So we will create a github workflow for the same. Something like https://github.com/kubearmor/KubeArmor/blob/main/.github/workflows/ci-test-ginkgo.yml but the event trigger would be a cronjob and release.

[swastik959]

@daemon1024 can I work on this issue of it is not assigned

[daemon1024]

We have

• [ ] k0s ( Will be supported after #1399 )
• [x] k3s ( Already being used in PRs )
• [ ] Minikube
• [ ] MicroK8s
• [ ] KinD

So you can work on adding one of them? @swastik959 WDYT? Thanks for the interest. Please do comment here what you would be working on.

It would be great if the workflow in general is abstracted in such a way that testing up other providers won't be difficult.

[swastik959]

@daemon1024 I can work on minikube and kinD since I am familiar with them

[MeenuyD]

Hello @swastik959 I am working on the issue

[daemon1024]

Just a disclaimer, Expect KinD to not work since it depends on the underlying Host. So we can start with Minikube. @MeenuyD and @swastik959 You can collaborate on Slack if needed. Microk8s should be straight forward as well, so @MeenuyD y don't you pick that up. Thanks.

[rootxrishabh]

Hey @daemon1024, can I pick up microk8s?

[rootxrishabh]

Kubearmor is failing on minikube, kubearmor controller and relay server are working fine. I tested using docker as the engine.

[anurag-rajawat]

Hi @rootxrishabh, can you please share the logs?

[anurag-rajawat]

@rootxrishabh how did you install KubeArmor using karmor or KubeArmor operator?

[rootxrishabh]

Hey @anurag-rajawat, I installed using karmor.

[anurag-rajawat]

I tried to reproduce it on my system but could not reproduce it.

$ kubectl get no -o wide
NAME       STATUS   ROLES           AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION          CONTAINER-RUNTIME
minikube   Ready    control-plane   7m41s   v1.27.4   192.168.49.2   <none>        Ubuntu 22.04.2 LTS   6.2.9-300.fc38.x86_64   docker://24.0.4

$ karmor probe

Found KubeArmor running in Kubernetes

Daemonset :
        kubearmor       Desired: 1      Ready: 1        Available: 1
Deployments :
        kubearmor-controller    Desired: 1      Ready: 1        Available: 1
        kubearmor-relay         Desired: 1      Ready: 1        Available: 1
Containers :
        kubearmor-controller-7cb5467b99-xq795   Running: 2      Image Version: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
        kubearmor-relay-5ccb6b6ffb-gh55l        Running: 1      Image Version: kubearmor/kubearmor-relay-server:latest
        kubearmor-wc6kt                         Running: 1      Image Version: kubearmor/kubearmor:stable
Node 1 :
        OS Image:                       Ubuntu 22.04.2 LTS
        Kernel Version:                 6.2.9-300.fc38.x86_64
        Kubelet Version:                v1.27.4
        Container Runtime:              docker://24.0.4
        Active LSM:                     BPFLSM
        Host Security:                  false
        Container Security:             true
        Container Default Posture:      audit(File)             audit(Capabilities)     audit(Network)
        Host Default Posture:           audit(File)             audit(Capabilities)     audit(Network)
        Host Visibility:                none
Armored Up pods :
+-----------+-----------------+------------+------+--------+
| NAMESPACE | DEFAULT POSTURE | VISIBILITY | NAME | POLICY |
+-----------+-----------------+------------+------+--------+
+-----------+-----------------+------------+------+--------+

[anurag-rajawat]

Could you please check the logs of the init container? It is failing in your case.

kubectl -n kube-system logs -f <kubearmor_pod> -c init

[rootxrishabh]

Here the output from the init container

[rootxrishabh]

Seems like a missing dependency or a PATH problem.

[anurag-rajawat]

BTF support was first introduced in Linux kernel 5.1.x, so you will need to install a kernel version at least that new.

Please install the kernel headers and check for BTF support.

$ sudo apt install linux-headers-$(uname -r)

or you can also upgrade your system to install the latest kernel.

Check whether CONFIG_DEBUG_INFO_BTF=y exists and is set to y.

$ cat /boot/config-$(uname -r) | grep  -e "BTF"

[sheharyaar]

Is this completed ? if no, then is there anyone working on this ?

[AllenOps08]

@daemon1024 Anyone working on this?

[navin772]

@anurag-rajawat @rootxrishabh does KubeArmor policy enforcement works in AppArmor mode in minikube? I tried to install via both karmor and helm, it gives the warning: ⚠️ KubeArmor is running in Audit mode, only Observability will be available and Policy Enforcement won't be available.

Minikube works when BPFLSM is enabled. K3s on the same machine works (enforcement) with both AppArmor and BPFLSM.

[navin772]

@daemon1024 Since crio is not officially supported by microk8s as per https://github.com/canonical/microk8s/issues/3549 , should I create a CI test for containerd only, I ran the tests and they pass for containerd.

Also, what should be the frequency of the cronjob to test periodically?