search

astro / microvm.nix

NixOS MicroVMs
https://astro.github.io/microvm.nix/
MIT License
1.4k stars 102 forks source link

Derivations built with writableStoreOverlay and reboots #134

Open Eisfunke opened 1 year ago

Eisfunke commented 1 year ago

I've added a writableStoreOverlayto a microvm according to the instructions here: https://astro.github.io/microvm.nix/shares.html

This does work, I can now build/download derivations inside the vm, e.g. with nix run nixpkgs#hello. However, when the vm is rebooted, and try the same command again, the derivations are downloaded again.

As far as I understand, that happens because the Nix daemon doesn't know about those derivations anymore, because its database isn't persisted acroos the reboot. So it won't know about previously added derivations, even though the paths still exist in the filesystem.

As I understand the code, this problem is tackled for the paths needed for the vm's system config via nix-store --load-db during boot, see here.

Is my understanding correct and this is expected? If so, I think this should be explained in the docs (I can prepare a PR for that), because I intuitively expected writableStoreOverlay to be usable across reboots.

Also, is there a proper way to get a writable store in a microvm that survives reboots and preferably enables garbage collection? It seems microvms always either have a read-only squashfs store with optionally the writable overlay, or if you turn storeOnDisk off wants to use the host store directly, and it seems like a very bad idea to try and write stuff there from inside the vm. I also tried mounting /nix or /nix/var as a share additionally to using the writable overlay, but that just seems to corrupt after rebooting.

(My, admittedly convoluted, usecase: I'm trying to run a GitLab Runner that can build Nix derivations inside of a microvm for additional isolation and so that I don't have to have a Docker daemon running on my host system. I'd prefer to be able to keep already built derivations cached across reboots. My config for reference: https://git.eisfunke.com/config/nixos/-/blob/main/nixos/server/git.nix)

Thank you for this project! Sorry for the long issue, I'd be thankful for any pointers in the right direction.

astro commented 1 year ago

Your reckoning is 100% right. I agree, I should add that behaviour to the docs until the problem is solved. It would be nice to keep the derivations across reboots, so feel encouraged to research further!

ghost commented 1 year ago

Is any workaround available?

Eisfunke commented 1 year ago

What I'm currently trying is using a self-hosted binary cache: https://github.com/zhaofengli/attic

You can push any derivations built in the VM to the store and won't have to re-build them. As you can host the cache on the same machine or network it should be pretty quick to download from there. Could be an alternative depending on your usecase, if you don't strictly need the derivations to stay in the VM's store. I'd even argue it's cleaner because the VM needs less state that way.