NixOS installer can't find /dev/root during Stage 1

[pmarreck]

Everything worked fine up until this point.

Additionally, I couldn't successfully build the ISO locally (it errored; I also tried with --pure, same error), so I had to download one of the releases.

The output looks like this (note that this is a text copy from a photo I took on my iPhone, so character recognition errors may exist):

<<< NixOS Stage 1 >>> 

loading module loop...
loading module overlay...
loading module dm_mod...
running udev...
Starting version 251.3
starting device mapper and LVM...
mounting tmpfs on /...
waiting for device /dev/root to appear........................
Timed out waiting for device /dev/root, trying to mount anyway.
mounting /dev/root on /iso...
[   24.286920] /dev/root: Can't open blockdev
[   24.287235] /dev/root: Can't open blockdev
[   24.287527] /dev/root: Can't open blockdev
[   24.287841] /dev/root: Can't open blockdev
[   24.288213] /dev/root: Can't open blockdev
mount: mounting /dev/root on /mnt-root/iso failed: No such file or directory

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on '/mnt-root' and then start stage 2. Press one
of the following keys:

   i) to launch an interactive shell
   f) to start an interactive shell having pid 1 (needed if you want to start stage 2's init manually)
   r) to reboot immediately
   *) to ignore the error and continue

[pmarreck]

additionally, when I start an interactive shell, I see an error: can't access tty: job control turned off. I have no Nix commands available! (Nor Git, zfs, etc.... all the stuff that nixos normally provides) So I'm dead in the water, from what I can tell.

[tpwrules]

Can you share more about the system you tried to build on and the system you tried to run on? Did you upgrade to a recent nixpkgs in the pin file and hit #16 ?

Stage 1 is just the initramfs whose only purpose in life is to mount the rootfs, so yes, all your usual tools will be missing. Use ls and dmesg to see where the root device is, then mount to mount it. I have had issues with some flash drives on some ARM systems where they take a long time to be recognized and you get this error.

[pmarreck]

I didn't create any network connections so it would be impossible to upgrade anything. I only used the prebuilt ISO.

I'm on a Macbook Pro M1 Max 16in 2021 with 64gb RAM.

[tpwrules]

I'm still not sure what you have done here to try to build the ISO. Unfortunately it cannot be cross-compiled from Nix on Darwin. You need a Linux system, which can be a VM if you want to build the image on your Mac. I recommend UTM and an aarch64 NixOS guest. It is free, easy to install, and quite fast. Of course, this system or VM needs internet access so Nix can download the source code and so forth during the build. You will also need network access in the installer as described in my manual.

As for the issue about /dev/root, there are two possibilities I have seen. First, make sure the ISO got transferred to your flash drive correctly as described in my manual. dd is the only correct way to do this. The ISO must be transferred to the drive block device itself, not a partition on the drive. You can use dd on macOS for this with the prebuilt ISO.

Second, the flash drive might be quirky. Try a different one. If you can't, you can probably get the installer to start with the following steps:

1. Attempt to start the installer normally
2. When the boot fails and you are prompted, hit i to start a shell
3. Unplug your flash drive, plug it into a different port, then wait 30 seconds
4. Run the command mount -t iso9660 /dev/root /mnt-root/iso
5. Exit the shell by running exit to continue the boot process

For the record, my machine is the same as yours.

[pmarreck]

I'm still not sure what you have done here to try to build the ISO. Unfortunately it cannot be cross-compiled from Nix on Darwin.

I did try Nix on Darwin first, although wouldn't that NOT be a cross-compile, given the native architecture is aarch64 on the M1 Macs? Or does Nix on MacOS use Intel binaries? (Wow, how am I just considering this? I just always assumed it was aarch64...)

Regardless, I have a separate NixOS install on a PC (in fact, that is where I am currently typing from, and how I caught the "Nix bug" which resulted in the desire to pursue this in the first place!) and can try to cross-compile it from there. You're right in that I did originally try to compile it directly on Nix overlaid on MacOS. (I've also played with UTM and am familiar with virtualization solutions such as the VMWare insider preview that runs AArch64 OS'es.)

You will also need network access in the installer as described in my manual.

OK, I'll drag a network cable over. ;)

First, make sure the ISO got transferred to your flash drive correctly as described in my manual. dd is the only correct way to do this.

That's what I did. In fact, here is the exact command I used, copied right out of my history:

sudo dd if=nixos-22.11pre-m1-release-2022-08-14.iso of=/dev/rdisk2 status=progress bs=1m

Notable details that might matter are that I used rdisk (raw disk access) instead of disk (buffered disk access), and rdisk2 instead of rdisk2s1.

This is the prebuilt ISO I downloaded from the releases page.

I will try to troubleshoot using your steps at the end, there.

[tpwrules]

I did try Nix on Darwin first, although wouldn't that NOT be a cross-compile, given the native architecture is aarch64 on the M1 Macs? Or does Nix on MacOS use Intel binaries? (Wow, how am I just considering this? I just always assumed it was aarch64...)

Nix on macOS can use Rosetta to run Intel macOS binaries if you like. But what you compile for depends on not just the processor architecture, but also OS and runtime (like libc type). Compiling on macOS for Linux is a cross-compile even on the same machine because the way the program communicates with the OS is different. I and some others have tried, but it's not feasible for a full OS.

OK, I'll drag a network cable over. ;)

You can use WiFi if you like but it's a bit of typing.

That's what I did. In fact, here is the exact command I used, copied right out of my history:

That looks correct, just double checking.

[tpwrules]

Hopefully you managed to get everything going. Feel free to re-open if not.

[obrmao]

Hello @pmarreck @tpwrules any suggestions for further troubleshooting?

I have tried the image from the release page, along with cross-compiling the image on amd64 nixOS machine with 2 different flashdrives using dd if=nixos-22.11pre-m1-release-2022-11-24.iso of=/dev/sdb status=progress, retrying with bs=1m as well.

Each time I get the exact error message as described in the issue.

Happy to gather more data and looking forward to further suggestions!

[tpwrules]

What machine are you on? When did you run the Asahi installer? I have a suspicion there is something wrong with USB in the latest kernel and boot stuff.

[obrmao]

Hey @tpwrules ,

Asahi Installer was run yesterday (v0.5), after fully wiping the machine (2020 MacBook Air M1).

Some further investigation revealed that the root of the problem is that m1n1 does recognize and boot the USB, however it is not recognized by the kernel when installer is running - looking through dmesg logs there are no usb entries, other than loading the driver and the symlink from /dev/root to /dev/disk/by-label/ is broken, since we are missing the entry. lsusb also shows an empty list, reloading the usb and related modules with modprobe and re-inserting the flash drives does not have any effect, with fdisk -l and blkid only showing the internal nvme drives in the ash shell.

The only entry that does show up inconsistently after the failed boot in dmesg is ISOFS: unsupported/invalid hardware sector size 4096 but looking through the kernel source it might be a false positive.

I ended up removing NixOs partitions with diskutil, installed asahi minimal and will install NixOS from there today (since nixos-infect will not work for archlinux host). Happy to wipe it instead and go with usb one more time if further troubleshooting would help here.

[tpwrules]

Yeah, that is the fundamental issue that the kernel cannot re-mount the USB drive.

How do you plan to install from asahi minimal? If you use the U-Boot from that, does the USB drive boot properly? Or are you going to install Nix and then use nixos-install from Arch?

It would be great if you could pull m1n1/boot.bin off the ESP and post it here. You can mount the ESP in macOS using diskutil list and diskutil mount.

[tpwrules]

It looks like this is almost certainly my fault. A module needed for USB to work is now missing from the initrd. I didn't test properly on my machine so I thought everything worked. I am searching and working with the Asahi folks to identify it and will make a new release when it's fixed.

[tpwrules]

Fixed faster than I thought, though the new release is still building. Please file a new issue if you still run into problems.

[obrmao]

Hello @tpwrules ,

That one worked. Thanks!

[Smaug123]

Second, the flash drive might be quirky. Try a different one. If you can't, you can probably get the installer to start with the following steps:

1. Attempt to start the installer normally
2. When the boot fails and you are prompted, hit i to start a shell
3. Unplug your flash drive, plug it into a different port, then wait 30 seconds
4. Run the command mount -t iso9660 /dev/root /mnt-root/iso
5. Exit the shell by running exit to continue the boot process

This procedure has been consistently necessary for me, by the way - a one-line link to this comment from the installation guide might be handy for those of us with slightly borked USB sticks ("If /dev/root fails to mount during stage 1, see this comment"). I'd raise a PR except I'm on my phone.

[pmarreck]

Does this install ARM NixOS or x64 NixOS on Apple hardware? (And is x64 arch pretty much recommended for broad compatibility and not having to rebuild the world on every kernel update?)