Choose benchmark criteria

[ls-1801]

Sources:

https://www.brendangregg.com/blog/2016-01-27/unikernel-profiling-from-dom0.html

[ls-1801]

The Evaluation used within the MirageOS Paper:

Microbenchmarks to evaluate:

• Boot-time
  • Comparison against minimal Linux Kernel (Unclear which one) + Linux Kernel with Apache Webserver
  • Measured Time from starting the Unikernel via XEN until receiving a UDP Packet from the Unikernel
  • Modified Version of XEN that would allow for parallel boots, as the Boot was mostly limited by XEN
• Threading
  • Time to construct a single Thread
  • Many Threads sleeping for 0.5 to 1.5 s
  • Thread-Times accuracy. How far off is the wakeup time from the desired time
• Network and Storage
  • Flooded with Pings (10⁶), stress-test for 72 hours
  • Measure Latency
  • Compare to standard Linux VM
• TCP Stack
  • Full Connection Cycle
  • TCP Benchmark: Fast Retransmit / Recovery / New Reno / Window Scaling
  • using iperf

Applications:

• DNS Server
• OpenFlow Controller (Networking Switch)
• Dynamic Web Server

Code and Binary Size

[fschmllr]

@ls-1801 here is something related that I found.

Artifacts, including experiments and graphs, for the paper: "Unikraft: Fast, Specialized Unikernels the Easy Way" (EuroSys'21 - Best Paper Award).

https://github.com/unikraft/eurosys21-artifacts

[ls-1801]

Measurements from the Unikraft Eurosys21 Experiments

• Dependency Graph Comparision
  • (Linux Kernel)
  • Nginx
  • Hello World
• Syscalls by popular Applications
• Image size of Unikraft Applications
  • Nginx Helloword Sqlite Redis (LTO + DCE)
• Boot Time
  • Already showed that using an optimized XEN 2 ms boot time are possible for no-op apps
  • Different VMMs, Solo5, FireCracker, QEMU (Micro VM), QEMU (1NIC), QEMU (KVM?)
  • Time from First Guest Instruction -> main() is called
  • Network Card increased boot time
  • Static vs. Dynamic Page Tables
• Application Performance Testing
  • HTTP Reply Performance Testing (wrk)
  • Redis Performance testing (redis-benchmark)
    • with varying memory allocators
  • SQL Lite
    • Memory Allocation Performance
    • newlib vs. musl
• Filesystem Latency
• Syscall Time (With and without Mitigation) on KVM

[ls-1801]

brendangregg Blog Post

Flamegraphs

• Focuses on MirageOS
• Flame graph for Userspace Application could improve performance while
  • Running a Testsuite
  • Development
• Differences between Userspace and running on a Hypervisor:
  • Single Address space
  • No context switches
  • Hypercalls rather than Syscalls
  • Different Kernel Code --> Some performance issues are only visible when running virtualized

Stackwalking under Xen

• DWARF not possible?
• Fallback to FP (has to be enabled by the compiler)
• Symbol Table from the mirageos objdump
• xenctx can be used to dump registers and stacktrace

PoC

• Create many Snapshot and build the Flamegraph
• Don't use the shell, instead modify xenctx
• calls xc_domain_pause(), which increases the latency of the unikernel

[ls-1801]

Goals

• The Operator should boot as fast as possible
• The Operator should not be overwhelmed by incoming tuples, so it should process tuples as fast as possible
• (The Operator should compile as fast as possible)

  Non-Goals

• Test other Applications in a Unikernel scenario
• Test Disk I/O
• Threading, although it might be interesting if the processing is bound by computation
• Memory Allocation, if possible

What do we need to Measure:

Boot-Time

We have to measure the boot-time and identify parameters that would influence the boot time. A few that other projects have already discovered:

• VMM
• Code size
• Level of security (Mitigation)
• Static vs. Dynamic Page
• Hardware (Setup of NIC, etc.)
• Unikernel Framework

Tuple Processing Performance

Measure the performance of the Unikernel Processing capabilities

• Are we limited by I/O or Compute?
• Is the Unikernel the bottleneck or the VMM or the Hardware?
• Pin-Point where the bottleneck lies using performance evaluation tools/visualization (flame graphs)

Combination of Both

Does Boot Time equal the time it takes for the Operator to achieve its maximum processing capabilities?

Approach

Boot-Time

We would like to know the time it takes for the Query Processor to decide it needs an Operator until the time the Operator processes tuples at maximum speed In a scenario where the Nebula Stream System wants to use Adhoc compilation of Operators, it might also be beneficial to investigate the impact of compilation time

• MirageOS tests the boot time of the Unikernel by sending a UDP Package to the Hypervisor
• Unikraft measures the time from the first Unikernel Instruction to calling the main() method

The MirageOS approach appears to be the easiest. It could technically be implemented on the spot. The Unikraft approach is not trivial.

Tuple Processing Performance

A Python benchmark could be used to flood the Unikernel with tuples. We need a mechanism to detect at which point the Unikernel is overwhelmed. A Map Operator could be useful to ensure that no tuples are dropped. The approach demonstrated in the brendangregg blog post could be used to identify hot areas of the Unikernel Execution. As noted by the Author, this will highly impact the performance of the Unikernel.