search

astro / microvm.nix

NixOS MicroVMs
https://astro.github.io/microvm.nix/
MIT License
1.38k stars 101 forks source link

nixos-modules: add closure size + startup time optimization #146

Closed lf- closed 1 year ago

lf- commented 1 year ago

This is based on my work on MapleCTF to run microvm.nix inside a docker container (incidentally, an awesome microvm.nix use case) and have the size not cause substantial issues.

Overall this saves about 700MB of closure size of a naive no-op VM configuration at practically the sole cost of eating a qemu compile.

co/microvm.nix » nix path-info -sSh ./result1
/nix/store/ligbkxkl1hnz2pvj8d9dfic991zfc0s0-microvm-qemu-nixos     1.7K  756.3M

co/microvm.nix » nix path-info -sSh ./result-without
/nix/store/h218db586pc627ai96j5cq2gssbvaxk8-microvm-qemu-nixos     1.7K    1.4G

Without:

[root@nixos:~]# systemd-analyze time
Startup finished in 1.402s (kernel) + 8.830s (userspace) = 10.233s
multi-user.target reached after 8.830s in userspace.

With:

[root@nixos:~]# systemd-analyze time
Startup finished in 295ms (kernel) + 2.066s (initrd) + 2.471s (userspace) = 4.834s
multi-user.target reached after 2.466s in userspace.

This looks impressive, and perhaps it is, but there is a lot of speed to still get out. The dominant factors in each system startup are:

Old:

New:

This group (in total, 3.1s ish in initrd; I have no idea why they are this bad!):

Then:

TL;DR: overall a huge improvement but it could have another 50% shaved off, and I really have no idea why all that hardware init takes 3 seconds!

lf- commented 1 year ago

You can get the systemd-analyze plot outputs here: https://gist.github.com/lf-/961f8f6be2de4f08f33a0cae505fd4c8

lf- commented 1 year ago

(systemd bug this works around: https://github.com/systemd/systemd/issues/29388)

lf- commented 1 year ago

oh, the nonsense device times are actually just, the entire initrd basically. https://github.com/systemd/systemd/issues/29010

lf- commented 1 year ago

Surplus commit has been dropped.