[Bug]: sysbench shows devbox is slower than docker, podman, apptainer, proot, and native in CPU test

[00sapo]

Current Behavior (bug) Slow sysbench when compared with docker, podman, and apptainer

Expected Behavior (fix) Faster, as native OS

Additional context I'm conducting some benchmarks about personal environments for scientific computing. I can't understand why devbox is slower than containerized solutions like podman, docker, and apptainer.

Here is the code of my benchmarks: https://github.com/00sapo/Container-benchmarks

The main issue is about the sysbench's CPU test. The results:

Tool	CPU (events/sec)
native	23.12
proot	22.27
podman	23.16
docker	23.17
apptainer	23.17
devbox	20.89

For short, here is how I setup devbox:

1. devbox init
2. devbox add sysbench
3. devbox run ./benchmark.sh > results/devbox.txt

The relevant content of benchmark.sh is: sysbench cpu --cpu-max-prime=200000 --time=$TIME run

The generated devbox,json:

{
  "packages": [
    "sysbench"
  ],
  "shell": {
    "init_hook": [
      "echo 'Welcome to devbox!' > /dev/null"
    ],
    "scripts": {
      "test": [
        "echo \"Error: no test specified\" && exit 1"
      ]
    }
  }
}

The generated devbox.lock:

{
  "lockfile_version": "1",
  "packages": {
    "sysbench": {
      "resolved": "github:NixOS/nixpkgs/f80ac848e3d6f0c12c52758c0f25c10c97ca3b62#sysbench",
      "source": "nixpkg"
    }
  }
}

[Lagoja]

Hey @00sapo, thanks for sharing this benchmark! We have some work planned that we think should reduce the CPU time/delay with devbox.

As a quick question -- do you know why the CPU is only getting 20 events/second? When I run this on an M1 Macbook I'm getting events per second: 10156458.33. Given that you're running the benchmark on an i9 CPU, I wouldn't expect that big of a disparity.

[00sapo]

Hi, thanks for the reply. I'm looking forward to see the improvements. I was also wondering if the compilation options of nix's sysbench are the same as for Fedora/Arch. I don't think sysbench takes into account initial/final delays...

I'm neither sure what the algorithm of sysbench is (not well documented, would need to check the code). Anyway, that number ("events per seconds") reduced a lot when I increased the maximum prime number and the maximum time. My results were run with 300 as maximum time (=5 minutes) and 200000 as maximum prime number.

[Lagoja]

Hey @00sapo, we just published a pre-release that should improve the performance of binaries installed by Devbox. Can you try the benchmark again using Devbox 0.8.0-dev?

To test the 0.8 prerelease, export the DEVBOX_USE_VERSION environment variable in your shell

export DEVBOX_USE_VERSION=0.8.0-dev

To revert back to the last release, unset this variable.

[00sapo]

I tried from scratch using DEVBOX_USE_VERSION=0.8.0-dev in all the steps but got

events per second:    21.00

Trying with pure nix-env:

events per second:    21.53

🫤

[00sapo]

For comparison, the native benchmark remains the same exact number (23.12) even on multiple runs. This is because that number is an average, I guess. So, since that metric is stable, the improvement of 0.11 is also stable I think, meaning 0.8.0-dev is slightly faster than the stable release, but there must be something more impactful on the performance of this test somewhere.

Fedora compilation script Arch compilation script Nixpkgs compilation script

---

edit

After having asked for help on the Nix channels, it looks like the issue resides in OS-level building scripts that disable compiler optimizations to guarantee reproducible packages. For instance, python is remarkably slower in NixOS and consequently in devbox.

With this, you should find a workaround or remove the statement on the homepage saying Don't sacrifice speed