search

astro / microvm.nix

NixOS MicroVMs
https://astro.github.io/microvm.nix/
MIT License
1.24k stars 93 forks source link

`sops-nix` remounts secrets on every `nixos-rebuild switch`, breaking virtiofs mounts in VMs #239

Closed romner-set closed 3 months ago

romner-set commented 4 months ago

Title. When mounting sops-nix secrets in VMs, rebuilding the system config remounts them on the host which makes the mounted dir appear empty on the guest. It wouldn't be an issue if it only happened on manual changes, but auto-updates also trigger this...

I don't think it's an inherent issue in microvm.nix, I'm just looking for solutions and I think this is the best place to ask. So far I came up with:

  1. just restarting all the VMs after each auto-upgrade, which is not that great because Unbound would have to rebuild its cache each time and uptime would suffer a bit
  2. encrypting the secrets separately for each VM and managing them inside their own config, would likely solve all issues but it would also mean each VM would need its own encryption key, sops configuration, etc. which is very unpractical for any larger amount of VMs
  3. adding an option to microvm.nix for readonly mounts, something like reading the host dir once on VM startup and then somehow copying that to the VM without it going into its store. Not perfect because changes to the host dir wouldn't propagate, but that can pretty easily be solved with sops-nix's restartUnits option
  4. making sops-nix not remount anything if the secrets aren't changed, not sure how practical this is as it seems to use a generations-like system where each rebuild makes a whole new dir in /run/secrets.d/

3 or 4 would likely be the best solutions, though I'm not sure how hard either would be to implement. Any other ideas?

astro commented 3 months ago

To poke at the underlying issue, could you experiment with virtiofsd flags --no-announce-submounts, --cache, --no-readdirplus, --inode-file-handles? The last one is already implemented as option microvm.virtiofsd.inodeFileHandles in the host module.

You can build with a local microvm.nix checkout using nix ... --override-input microvm git+file:///...

romner-set commented 3 months ago

Apologies for not responding earlier, but I've just tried a few things (--no-announce-submounts --no-readdirplus, --cache=always, --inode-file-handles=mandatory and combinations thereof) and none of those made any discernible difference.

I did however find sops-nix's keepGenerations option, and setting that to 0 (i.e. stopping sops-nix from ever removing old generations in /run/secrets.d/) seems to have fixed the issue completely, even though the shares still point to the /run/secrets symlink. It does cause a minor memory leak of sorts since each nixos-rebuild switch adds a new (in my case) ~100kB dir into RAM, but it's quite manageable.

I think the problem may be that virtiofsd reads the /run/secrets symlink and gets pointed to e.g. /run/secrets.d/3, which is what it then actually shares with the guest, and once sops-nix makes /run/secrets point to /run/secrets.d/4 and deletes /3, virtiofsd doesn't re-read the symlink and still tries to share /run/secrets.d/3.

astro commented 3 months ago

Thank you for posting a workaround! I've added a blurb to the FAQ in 917b08d. Feel free to open a PR if you think that additional details should be mentioned.