Cannot update a microvm reliably and can't nixos-rebuild swithc

[wokalski]

Hey! Thanks for the project. I am using declaratively deployed vms.

The first deployment is flawless but then I'm encountering 2 issues with so called day 2 operations.

Can't update the VM reliably

After I change my VM config and run nix run .#nixosConfigurations.adv1-0--cplane.config.microvm.deploy.rebuild -- root@146.59.118.241 root@146.59.118.241, the configuration is not effectively switched. I do remember about microvm -Ru but it doesnt' do anything. In order to reliably change the VM I have to rm -rf /var/lib/microvms/ and restart the systemd unit.

Can't run nixos-rebuild switch

So I want to treat my declaratively deployed VM like a regular NixOS host, ideally i don't want to use the microvm commands to change the config. So I port forwarded my VM for tests, ran nixos-rebuild switch from my laptop and to my surprise I saw:

(root@127.0.0.1) Password: 
copying 0 paths...
Shared connection to 127.0.0.1 closed.
Shared connection to 127.0.0.1 closed.
Failed to install bootloader
Shared connection to 127.0.0.1 closed.
warning: error(s) occurred while switching to the new configuration

I am not sure but I believe that my problems stem from the fact that I did tihs:

            volumes = [{
              image = guest.vm.rootVolume;
              mountPoint = "/";
              autoCreate = false;
            }] ++ (map (volume: {
              image = volume;
              mountPoint = null;
              autoCreate = false;
            }) (guest.vm.extraVolumes or [ ]));

I.e. I have a volume at / so that the entire vm is persistent. again i did this bc i wanted to have the VM behave like a regular host but it might have screwed things up. Maybe there are better ways, if yes, please do let me know 😄. maybe i should've just mounted /var for example.

[wokalski]

I am still curious to understand what was going on but possible would be smarter to achieve my goals re persistence with impermanence https://github.com/astro/microvm.nix/issues/52

[wokalski]

Ok, i partially understood #1. Since I use imperative updates, I cannot actually use microvm -u. I don't have an updateFlake and i manage the machine with ssh. So after it gets deployed i have to use the imperative interface exclusively.

And when it comes to the second part, I get this weird thing on ssh switch:

Warning: do not know how to make this configuration bootable; please enable a boot loader.

[astro]

MicroVMs are not to be treated like regular hosts. They don't have a boot loader, they use the direct kernel+initrd loading feature of the VMMs. nixos-rebuild will not work inside the VM but the VM systems are built by the host. This is by design as a VM could be compromised.

Do you want declarative (part of the host config) or imperative (microvm command, rebuild ssh deployment) VMs?

If you use rebuild, make sure that the two host parameters are actually distinct: the first is the NixOS host, the latter is the MicroVM.

Having a volume or a share mounted at / is supposed to work.

[wokalski]

it's almost all clear but one thing did not work for me: i want to declaratively provision and imperatively manage the VMs and i believe i could not effectively update the VM i mounted / but i might be wrong. I will check it soon. I now have a better grasp.

I wanted to have nixos-rebuild under disposal to be able to treat the VM like a regular host but i might as well not, it just changes my setup a bit. Do you anticipate any problems with running installOnHost/sshSwitch from inside of the VM that's being updated :D? I am using kubernetes and i will use a "special container" that acts as an update orchestrator and i want it to manage the config of the VM that the container runs on.