Open 00sapo opened 10 months ago
Hey @00sapo, thanks for sharing this benchmark! We have some work planned that we think should reduce the CPU time/delay with devbox.
As a quick question -- do you know why the CPU is only getting 20 events/second? When I run this on an M1 Macbook I'm getting events per second: 10156458.33
. Given that you're running the benchmark on an i9 CPU, I wouldn't expect that big of a disparity.
Hi, thanks for the reply. I'm looking forward to see the improvements. I was also wondering if the compilation options of nix's sysbench are the same as for Fedora/Arch. I don't think sysbench takes into account initial/final delays...
I'm neither sure what the algorithm of sysbench is (not well documented, would need to check the code). Anyway, that number ("events per seconds") reduced a lot when I increased the maximum prime number and the maximum time. My results were run with 300 as maximum time (=5 minutes) and 200000 as maximum prime number.
Hey @00sapo, we just published a pre-release that should improve the performance of binaries installed by Devbox. Can you try the benchmark again using Devbox 0.8.0-dev?
To test the 0.8 prerelease, export the DEVBOX_USE_VERSION environment variable in your shell
export DEVBOX_USE_VERSION=0.8.0-dev
To revert back to the last release, unset this variable.
I tried from scratch using DEVBOX_USE_VERSION=0.8.0-dev
in all the steps but got
events per second: 21.00
Trying with pure nix-env:
events per second: 21.53
🫤
For comparison, the native benchmark remains the same exact number (23.12) even on multiple runs. This is because that number is an average, I guess. So, since that metric is stable, the improvement of 0.11 is also stable I think, meaning 0.8.0-dev is slightly faster than the stable release, but there must be something more impactful on the performance of this test somewhere.
Fedora compilation script Arch compilation script Nixpkgs compilation script
edit
After having asked for help on the Nix channels, it looks like the issue resides in OS-level building scripts that disable compiler optimizations to guarantee reproducible packages. For instance, python is remarkably slower in NixOS and consequently in devbox.
With this, you should find a workaround or remove the statement on the homepage saying Don't sacrifice speed
Current Behavior (bug) Slow sysbench when compared with docker, podman, and apptainer
Expected Behavior (fix) Faster, as native OS
Additional context I'm conducting some benchmarks about personal environments for scientific computing. I can't understand why devbox is slower than containerized solutions like podman, docker, and apptainer.
Here is the code of my benchmarks: https://github.com/00sapo/Container-benchmarks
The main issue is about the sysbench's CPU test. The results:
For short, here is how I setup devbox:
devbox init
devbox add sysbench
devbox run ./benchmark.sh > results/devbox.txt
The relevant content of
benchmark.sh
is:sysbench cpu --cpu-max-prime=200000 --time=$TIME run
The generated
devbox,json
:The generated
devbox.lock
: