Automating Performance Benchmark

[Shreyas220]

Feature Request

Description

The aim is to assess Kubearmor's performance under real-world conditions and understand its impact on the workload as well as how Kubearmor itself performs.

This project aims to develop an automated benchmarking Tool for KubeArmor. The focus will be to enhance the efficiency and repeatability of performance evaluations, which are currently being done manually. The benchmarking system will streamline the process across various scenarios, ensuring that KubeArmor’s performance is consistently and rigorously evaluated after each release. Benchmarking should be independent of any CIs, assuming we have a cluster and kubearmor we should be able to run benchmark anywhere

Please go through the Benchmarking Guide and our Benchmarking Data

Scope:

• [ ] Environment Setup

• [ ] Benchmark Execution: Automate the execution of benchmarks under various scenarios (With BPF LSM and AppArmor).

  • without Kubearmor: To establish baseline
  • with Kubearmor (without policy)
  • with Kubearmor (with process policy)
  • with Kubearmor (with process, network policy)
  • with Kubearmor (with process, network, file policy)

• [ ] Metrics Collection: Collect key performance metrics such as CPU and memory usage of KubeArmor, relay server, and the performance (CPU, memory, and throughput) of the workload under test. We collecting through KSPs but need manual intervention to average out the correct range of data

• [ ] Reporting: Develop a system to report the benchmarking results automatically. This could involve sending notifications to a Slack channel or github comment or any other channel with the latest performance metrics after each release.

• [ ] (optional) Workload Selection: Identify the appropriate workload that effectively represents real-world usage scenarios for KubeArmor. (using google microservice demo but are open to recommendation )

[tesla59]

Hi, i have a few questions regarding the proposal

• Is the benchmarking system designed to run on GitHub Actions or any other CI pipeline?
• Assuming it is supposed to be run as CI step on every release, should the creation of k8s cluster also be automated? (Maybe we can create multi-cluster node using k3d which creates k3s cluster node as containers)
• Are we also taking in account for scalability testing?
• We can also include benchmarking using the native Go test tool to include benchmarking of functions and modules within the codebase

[Shreyas220]

Hey @tesla59

• Is the benchmarking system designed to run on GitHub Actions or any other CI pipeline?

Benchmarking should be independent of any CIs, assuming we have a cluster and kubearmor we should be able to run benchmark anywhere.

• Assuming it is supposed to be run as CI step on every release, should the creation of k8s cluster also be automated? (Maybe we can create multi-cluster node using k3d which creates k3s cluster node as containers)

No, as mentioned above automating creation of k8s env is not needed , we are settting up k8s env in our workflow, similarly we can use k3d to setup multi node cluster

• Are we also taking in account for scalability testing?

Yes, This is would be an important metric to see whether kubearmor is able to handle increased load

This can be divided in two parts

• increased number of events in different scenarios for Kubearmor
• increasing the number of nodes(kubearmor is present on each node) to stress test kubearmor relay

• We can also include benchmarking using the native Go test tool to include benchmarking of functions and modules within the codebase

Currently not in scope of the project, but i think we can expand the scope of the project to include this if time permits

[Sayanjones]

Hi @Shreyas220, I am interested to work on this project. Can we discuss this further?

[Stan370]

Hi @Shreyas220, I have been studying this issue. Also, there are a few questions when I am writing the proposal. Are there any preferred tools or technologies (Prometheus Query, Elasticsearch) that the team is considering for implementing the benchmarking automation and metrics collection? Could you please provide some guidance on the proposal?

[Shreyas220]

Hey @Stan370 and @Sayanjones Thanks for the interest

Are there any preferred tools or technologies (Prometheus Query, Elasticsearch) that the team is considering for implementing the benchmarking automation and metrics collection

we don't have a strict preference, but we currently use KSM and Prometheus to gather and summarize metrics.

The aim is to assess Kubearmor's performance under real-world conditions and understand its impact on the workload as well as how Kubearmor itself performs.

Questions to keep in mind

1. How will event generation be handled?

• We're interested in generating events(making request to the workload) that measure throughput, and we currently use Locust for this purpose.

2. How will we collect metrics?

• We need to track CPU and memory usage for both Kubearmor and the Kubearmor relay, as well as for the workload and the throughput of requests made by the event generator(we should collect these metrics hen requests are being made to the workload).We currently use Locust

3. Please consider the different scenarios when collecting metrics, and mention how you will apply them to the environment , such as:

Kubearmor when using

1. BPF LSM
2. Apparmor

Also when Kubearmor functioning

• without any policies
• Kubearmor (with process policy)
• Kubearmor (with process, network policy)
• Kubearmor (with process, network, file policy)

Also scenarios where Kubearmor's event visibility is set to none, and we only receive alerts for policy violations.

And more if needed

we need to understand how we can apply/remove policies and change annotations in a Kubernetes environment. Therefore, you would need to know how to deploy Kubearmor in these various scenarios. (Dont worry there are documentation)

---

As a thought, we are considering the idea of deploying a pod that could manage all these tasks. It would be responsible for generating events for the workload, collecting metrics, switching between different scenarios benchmarking them to ultimately provide us with the results. We plan to use the tools already present, so you don't need to create new ones(unless you want to :joy: ). Instead, please propose a method to automate these tasks using the existing tools(preferably how it's done in benchmarking guide)

We would appreciate any suggestions you might have.