coreutils not on scope at install-grub.pl

[Mikilio]

Describe the bug

Following bug occurs in install-grub.pl when executing the following command

$ sudo --preserve-env=PATH,NIX_PATH `which nixos-install` --root /mnt --flake /mnt/etc/nixos#homestation

Coreutils functions are not in scope (finishes install when executing in nix develop nixpkgs#coreutils but result doesn't boot).

Steps To Reproduce

Steps to reproduce the behavior:

1. run my script and choose an empty drive on your system
2. run

   sudo git clone https://github.com/Mikilio/dotfiles.git /mnt/etc/nixos

3. run

   $ sudo --preserve-env=PATH,NIX_PATH `which nixos-install` --root /mnt --flake /mnt/etc/nixos#homestation

Expected behavior

It should just finish installation and new disk should be able to boot.

Examples

Output:

installing the boot loader...
setting up /etc...
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
updating GRUB 2 menu...
/nix/var/nix/profiles/system/sw/bin/bash: line 10: rmdir: command not found

Additional context

Installing from Non-NixOS system (Fedora) The exact commit of my config when this bug occured is 4b14be37e623395232eee63504ca42050385135a

Notify maintainers

@xaverdh @rnhmjoj

Metadata

error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)

       at «string»:1:25:

            1| {...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (nix-info) ]; } ""
             |                         ^
(use '--show-trace' to show detailed location information)

[xaverdh]

I can't find any direct reference to this in your config, but probably one of the external flakes that you use, sets boot.loader.grub.extraInstallCommands and uses rmdir instead of ${lib.getBin coreutils}/bin/rmdir?

PS: With a recent enough nix, you can set nix.nixPath = [ "nixpkgs=flake:nixpkgs" ] to get a working NIX_PATH for the old tooling

[xaverdh]

Just to clarify: install-grub.pl is not to blame here, since it is written in perl, and does not explicitly shell out to rmdir. Instead the error originates from the shell script that executes install-grub.pl I think.

[xaverdh]

Since its all scripts in the store, can you try and check which piece of code exactly fails (starting with which nixos-install and chasing references)?

[Mikilio]

First I'd like to apologize for my late response.

I remember trying to track the error and found that install-grub.pl is the only place where it happens.

I will look again when I come back to my deployment script. Back then I worked around the error by using systemd-boot.

When I come back to this error I am not sure if I will be able to reproduce this error because my fedora installation does not exist anymore.

[ISibboI]

I am having the same problem when following this guide to install nixos on a dedicated server: https://github.com/nix-community/nixos-install-scripts/blob/master/hosters/hetzner-dedicated/hetzner-dedicated-wipe-and-install-nixos.sh

I have modified the script to include pkgs.coreutils in the installation environment, but it did not help.

My `configuration.nix`

``` { config, pkgs, ... }: { imports = [ # Include the results of the hardware scan. ./hardware-configuration.nix ]; # Use GRUB2 as the boot loader. # We don't use systemd-boot because Hetzner uses BIOS legacy boot. boot.loader.systemd-boot.enable = false; boot.loader.grub = { enable = true; efiSupport = false; devices = [ "/dev/sda" "/dev/sdb" ]; }; networking.hostName = "hetzner"; # The mdadm RAID1s were created with 'mdadm --create ... --homehost=hetzner', # but the hostname for each machine may be different, and mdadm's HOMEHOST # setting defaults to '' (using the system hostname). # This results mdadm considering such disks as "foreign" as opposed to # "local", and showing them as e.g. '/dev/md/hetzner:root0' # instead of '/dev/md/root0'. # This is mdadm's protection against accidentally putting a RAID disk # into the wrong machine and corrupting data by accidental sync, see # https://bugzilla.redhat.com/show_bug.cgi?id=606481#c14 and onward. # We do not worry about plugging disks into the wrong machine because # we will never exchange disks between machines, so we tell mdadm to # ignore the homehost entirely. environment.etc."mdadm.conf".text = '' HOMEHOST ''; # The RAIDs are assembled in stage1, so we need to make the config # available there. boot.initrd.services.swraid.mdadmConf = config.environment.etc."mdadm.conf".text; # Network (Hetzner uses static IP assignments, and we don't use DHCP here) networking.useDHCP = false; networking.interfaces."enp0s31f6".ipv4.addresses = [ { address = ""; # FIXME: The prefix length is commonly, but not always, 24. # You should check what the prefix length is for your server # by inspecting the netmask in the "IPs" tab of the Hetzner UI. # For example, a netmask of 255.255.255.0 means prefix length 24 # (24 leading 1s), and 255.255.255.192 means prefix length 26 # (26 leading 1s). prefixLength = 26; } ]; networking.interfaces."enp0s31f6".ipv6.addresses = [ { address = ""; prefixLength = 64; } ]; networking.defaultGateway = ""; networking.defaultGateway6 = { address = "fe80::1"; interface = "enp0s31f6"; }; networking.nameservers = [ "8.8.8.8" ]; # Initial empty root password for easy login: users.users.root.initialHashedPassword = ""; services.openssh.settings.PermitRootLogin = "prohibit-password"; users.users.root.openssh.authorizedKeys.keys = [ # FIXME Replace this by your SSH pubkey! "ssh-rsa " ]; services.openssh.enable = true; # FIXME # This value determines the NixOS release with which your system is to be # compatible, in order to avoid breaking some software such as database # servers. You should change this only after NixOS release notes say you # should. system.stateVersion = "23.05"; # Did you read the comment? } ```

Running nixos-install results in the following:

$ PATH="$PATH" `which nixos-install` --no-root-passwd --root /mnt --max-jobs 40
++ which nixos-install
+ PATH=/root/.nix-profile/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ /root/.nix-profile/bin/nixos-install --no-root-passwd --root /mnt --max-jobs 40
building the configuration in /mnt/etc/nixos/configuration.nix...
/nix/store/c17y1bb883fyxxgjp1v0f3f9sijnh9a9-nixos-system-hetzner-23.05.2209.ac1acba43b2
installing the boot loader...
setting up /etc...
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/sda...
Installing for i386-pc platform.
Installation finished. No error reported.
installing the GRUB 2 boot loader on /dev/sdb...
Installing for i386-pc platform.
Installation finished. No error reported.
/nix/var/nix/profiles/system/sw/bin/bash: line 10: rmdir: command not found

Edit 1: I tried executing the nixos-install command with -vvvvvvvvvv, but there is no more information on this error. If I instead add --keep-going, it anyways stops after the missing rmdir command, and e.g. the ssh key of root is not added correctly.

Edit 2: Tracing the error:

• which nixos-install resolves to /nix/store/ka7qp6wsjzss72x0zrswh7rp42f4rv97-nixos-install/bin/nixos-install, which in line 199 calls nixos-enter with a command that includes rmdir:

  NIXOS_INSTALL_BOOTLOADER=1 nixos-enter --root "$mountPoint" -c "$(cat <<'EOF'
   # Create a bind mount for each of the mount points inside the target file
   # system. This preserves the validity of their absolute paths after changing
   # the root with `nixos-enter`.
   # Without this the bootloader installation may fail due to options that
   # contain paths referenced during evaluation, like initrd.secrets.
   # when not root, re-execute the script in an unshared namespace
   mount --rbind --mkdir / "$mountPoint"
   mount --make-rslave "$mountPoint"
   /run/current-system/bin/switch-to-configuration boot
   umount -R "$mountPoint" && rmdir "$mountPoint"

• which nixos-enter resolves to /nix/store/y74pw332ib4z5m7acfv7pxvlaw3j111p-nixos-enter/bin/nixos-enter. In there I am lost though, since I don't find the log message setting up /etc... in there. However, grepping gave me this:

  $ grep -rF 'setting up /etc...' /nix/store
  /nix/store/5zc2agvlkh5ckgjfcpfy0amzrrsczfgk-nixos/modules/system/etc/etc.nix:        echo "setting up /etc..."
  /nix/store/i5g8iw709ky0nmlhqlphcwbvlwjmpg7v-nixpkgs-23.05/nixpkgs/nixos/modules/system/etc/etc.nix:        echo    "setting up /etc..."
  /nix/store/mz5byrgkj4gkwcp0qcixjkwb3ffiqz3y-nixpkgs/nixpkgs/nixos/modules/system/etc/etc.nix:        echo "setting up    /etc..."

  Here I am lost where to continue. I don't know much about nix, but if more information is needed, I am happy to provide it.

[ISibboI]

@xaverdh @rnhmjoj is there anything more I can provide to help resolve this? I would also be interested in fixing it myself via a pull request, but I would need some guidance, since this would be my first contribution to NixOS.

[Mikilio]

@xaverdh @rnhmjoj is there anything more I can provide to help resolve this? I would also be interested in fixing it myself via a pull request, but I would need some guidance, since this would be my first contribution to NixOS.

I think a good point to start would be to follow the instructions originally directed at me.

Run the scripts manually (or mark all with set -x) to see which one it actually fails.

A good start already is to have a relatively simple setup to reproduce the error, like you do.

[ISibboI]

Thanks for the guidance! I have now changed line 209 in /nix/store/ka7qp6wsjzss72x0zrswh7rp42f4rv97-nixos-install/bin/nixos-install from umount -R "$mountPoint" && rmdir "$mountPoint" to umount -R "$mountPoint" && echo "now doing rmdir in nixos-install" && rmdir "$mountPoint" && echo "rmdir in nixos-install was successful"

Now my log ends with:

now doing rmdir in nixos-install
/nix/var/nix/profiles/system/sw/bin/bash: line 10: rmdir: command not found

So clearly, the offending rmdir is the one in nixos-install at line 209.

How do I get coreutils in scope for that one?

[xaverdh]

You can add pkgs.coreutils here

[Mikilio]

You can add pkgs.coreutils here

I feel like I remember wanting to do this before and you insisted on using full paths like so /nix/var/nix/profiles/system/sw/bin/rmdir now I don't know what the reason was back then but maybe this applies here as well?

[rnhmjoj]

The script is indeed missing a coreutils dependency. I'm not sure why it would fail on that line, though. There are references to dirname, touch, mkdir, install, rm (all in coreutils) before that line... It has probably something to do with nixos-enter.

[ISibboI]

Adding pkgs.coreutils at the point propsed by xaverdh seems to do nothing. It still fails while complaining that rmdir does not exist. But I am very confused, because in my fork, the line with the rmdir contains this: (rmdir "$mountPoint" 2>/dev/null || true). Does || not apply to a command not being found?

[xaverdh]

/nix/var/nix/profiles/system/sw/bin/bash: line 10: rmdir: command not found ah that probably means that we are trying to find rmdir in that location inside the chroot mount namespace, not in PATH. So the modifications to PATH do not propagate there.

[xaverdh]

The script is indeed missing a coreutils dependency. I'm not sure why it would fail on that line, though. There are references to dirname, touch, mkdir, install, rm (all in coreutils) before that line... It has probably something to do with nixos-enter.

The $system/sw/bin/bash -c part is executed in chroot while the other stuff is not. That is why the other commands don't fail I think

[xaverdh]

So I can think of two possible solutions:

• hard code "${lib.getBin coreutils}/bin/rmdir"
• set PATH=${...} at the start of the commands passed to -c edit: Both of this will not work, see below

[rnhmjoj]

But why would coreutils not be in the PATH of the chroot?

[xaverdh]

You are right, it does take the indirection through PATH, but that points to /run/current-system/sw/bin. I misinterpreted that part. Still /run/current-system/sw/bin will be resolved in the changed root, so will not find the coreutils of the ambient system.

[xaverdh]

(I thought the issue was that chroot clears PATH, but that is not happening, I just checked)

[Mikilio]

Maybe it is not that the path is cleared, but rather that the path is not correct anymore in the new root.

[xaverdh]

Maybe it is not that the path is cleared, but rather that the path is not correct anymore in the new root.

Yes, that is what I meant

[rncwind]

I'm also having this issue with the same install script for a hetzner dedi. Was a workaround found for this, as this method of install is broken, and so is nixos-generators kexec-bundle making it very difficult to actually install nixos on hetzner right now.

[xaverdh]

So coreutils-full is part of requiredPackages and should therefore be part of the new system. So I don't quite see why it would fail, since the path should be the same (/run/current-system/sw/bin/rmdir) as in the ambient system. Can you check if the install succeeds when you replace rmdir by /run/current-system/sw/bin/rmdir?

[rncwind]

Yeah, that seems to have worked. Very odd how it's not picking it up from the system we are trying to install from. Thanks for the help, I hope this narrows it down a bit!

[xaverdh]

Wild guess would be that some form of caching by the shell happens, that makes it look for the resolved path from the ambient system. Anyway if that does happen to fix it, care to open a pr?

[Animeshz]

So is there a temporary workaround for this at the moment, if anybody was able to bypass the error?

PS: Nvm I sudo edited the script and hardcoded the path /nix/var/nix/profiles/system/sw/bin/rmdir there in nixos-install:209. I know not good to sudo edit cryptographically hashed nix path, but anyway, it works till the moment things are installed.