systemd-nspawn can not start nixos-on-nixos

[Andrei-Pozolotin]

Issue description

attempt to start nixos-on-nixos via systemd-nspawn fails with error in nspawn.c:

Directory /nix/store/xc7npmnz5ncvw795p9z1xsy4bk3m7spl-nixos-system-nixos-18.03.132304.45cd6090d9a doesn't look like an OS root directory (os-release file is missing). Refusing.

despite os-release apparently present, although as symlink:

ls -las /root/tester/etc/os-release 
0 lrwxrwxrwx 1 root root 58 Jan  1  1970 /root/tester/etc/os-release -> /nix/store/f5nj4avyxmrc4cf36aab17a474vdb36r-etc-os-release

Steps to reproduce

1. given basic /root/config.nix:

   { config, pkgs, ... }:
   {
   imports =
   [ # Include the results of the hardware scan.
   #      ./hardware-configuration.nix
   ];
   boot.loader.systemd-boot.enable = true;
   boot.loader.efi.canTouchEfiVariables = true;
   fileSystems."/".device = "/dev/invalid" ;
   system.stateVersion = "18.09";
   }

2. produce system image:

   nix build -f '<nixpkgs/nixos>' system -I nixos-config=/root/config.nix -o /root/tester

3. now try to instantiate:

   systemd-nspawn --boot --directory=/root/tester

Technical details

nixos:

 - system: `"x86_64-linux"`
 - host os: `Linux 4.14.32, NixOS, 18.03.131792.becbe4dbe16 (Impala)`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.0`
 - channels(root): `"nixos-18.03"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs`

systemd:

systemd 237
+PAM +AUDIT -SELINUX +IMA +APPARMOR +SMACK -SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN2 -IDN -PCRE2 default-hierarchy=hybrid

[Andrei-Pozolotin]

another issue is that systemd-nspawn --boot expects 3 hard coded init locations: https://github.com/systemd/systemd/blob/master/src/nspawn/nspawn.c#L2468

exec_target = "/usr/lib/systemd/systemd, /lib/systemd/systemd, /sbin/init";

none of which are provided by current

nix build -f '<nixpkgs/nixos>' system

[matthewbauer]

Note that we have a systemd fork without those paths:

https://github.com/NixOS/systemd/blob/nixos-v230/src/nspawn/nspawn.c

[markuskowa]

The problem can be solved in 18.03 by setting boot.loader.initScript.enable = true. That takes care of the /sbin/initlink.

[flokli]

Being able to systemd-nspawn random system closures from the nix-store seems to be a cool idea :-)

My nixos-system-nixos-* has a ./init present - I wonder why we do have boot.loader.initScript.enable at all, and don't just move ./init to ./sbin/init…

I'm not really sure if it's wise to teach our systemd-nspawn to look at ./init too if we can just move the init to a standard location.

cc @fpletz @Mic92 @arianvp

[arianvp]

--boot is just a convenience wrapper that expects init in a specific location and sets Parameters= accordingly

instead you can use the Parameters-= flag in systemd-nspawn directly and manually specify the location of init. The container will then boot without any changes

see man systemd.nspawn

       Parameters=
           Takes a space-separated list of arguments. This is either a command
           line, beginning with the binary name to execute, or – if Boot= is
           enabled – the list of arguments to pass to the init process. This
           setting corresponds to the command line parameters passed on the
           systemd-nspawn command line.

So instead of

systemd-nspawn --boot --directory=/root/tester

do

systemd-nspawn  --directory=/root/tester -- /run/current-system/sw/init

and the system should just boot.

See https://github.com/arianvp/nixos-stuff/blob/master/modules/containers-v2.nix#L65-L74 which is a full implementation of this idea by generation systemd.nspawn files

[flokli]

@arianvp true, so this is mostly UX / badly documented. Could you add a small line in the docs, or wiki, so it's better accessible?

@Andrei-Pozolotin does that solve your problem?

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[arianvp]

Note stale; just awaiting #74316 to be merged

[stale[bot]]

I marked this as stale due to inactivity. → More info

[arianvp]

Same status.

[arianvp]

@pasqui23 why did you close this? It's not stale and still an issue

[pasqui23]

Must have clicked by accident.Sorry

[stale[bot]]

I marked this as stale due to inactivity. → More info

[flokli]

I'm not sure https://github.com/NixOS/nixpkgs/pull/74316 is a solution for this.

systemd-nspawn can be invoked on any --image, not necessarily those part of the NixOS (host) system configuration.

We should probably move/symlink init into a location that the systemd-nspawn binary is capable to find and start.

It's probably a more generic fix (enabling booting NixOS images from other distributions) than just patching it in our systemd derivation.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[tv42]

By the same logic as above, https://github.com/NixOS/nixpkgs/pull/74316 is still not merged.