Stuck trying to boot

[justinmoon]

I'm on an M1 MacBook and I followed all the instructions in the YouTube video & instructions in this repo. I ran make iso/nixos.iso to build an ISO. Tried with VMWare Fusion downloaded from here and brew install --cask vmware-fusion-tech-preview.

I get to this screen and select the top option

But I get stuck here

Any ideas how to fix this?

[futurepaul]

Same here

[mitchellh]

I just noticed this as well. This looks like a kernel issue (as in, broadly related to the kernel, kernel config, etc.).

Booting works with Kernel 5.15.26, but fails for me .31.

I'm unsure what to do so I reported a Fusion TP issue: https://communities.vmware.com/t5/Fusion-for-Apple-Silicon-Tech/Failure-to-boot-Linux-Kernel-5-15-31-works-with-5-15-26/m-p/2901884#M1245

Meanwhile, I pushed a revert commit to ensure we stay on .26 for now. I'll look into pinning that later but hopefully we can get this resolved.

[mitchellh]

I've been told this might be a kernel bug, and the fix is already in tree (the kernel tree) but not released.

[mitchellh]

Waiting for this commit to get released: https://github.com/torvalds/linux/commit/27599aacbaefcbf2af7b06b0029459bbf682000d (we think its that commit)

[gnattu]

Unfortunately it is not that commit. I tried the 5.18-rc1 and no luck.

[mitchellh]

:(

https://communities.vmware.com/t5/Fusion-for-Apple-Silicon-Tech/Linux-5-17-EFI-stub-does-not-boot-hangs-at-loading-kernel/m-p/2900884#M1189

https://communities.vmware.com/t5/Fusion-for-Apple-Silicon-Tech/Failure-to-boot-Linux-Kernel-5-15-31-works-with-5-15-26/m-p/2901884#M1245

[mitchellh]

It looks like other hypervisors are having similar issues so it looks like we'll have to wait for VMware to update, is my guess, and stick to older kenels until then. https://gitlab.com/qemu-project/qemu/-/issues/899

[gnattu]

I posted a very tricky workaround here

This patchset removes ID_AA64ISAR2_EL1from 5.17 kernel which prevents the VM to boot. Not a good approach at all, as we are removing a kernel feature to support a broken hypervisor.

I personally use custom built kernel for a long time because VMware related drivers are not enabled by default on Arch Linux ARM's kernel config, so to apply an extra patch-set is not a big deal for me, but this would be a big problem for a lot who relies on prebuilt kernels.

[nateberkopec]

The image built by iso/build.sh still exhibits this issue 🤔 How do we pin the kernel version there?

[mitchellh]

The image built by iso/build.sh still exhibits this issue 🤔 How do we pin the kernel version there?

I don't know, but this is a good question. I rarely rebootstrap from an ISO so I don't regularly hit this. There should be a way to pin a specific nixpkgs for the ISO build. I'm also looking into the YOLO solution that @gnattu posted, its very easy to apply kernel patches with Nix, but the downside is compiling a kernel is no small task. Early on, I had to compile a patched kernel for M1+Fusion support (like, right when M1 came out) and even on M1 devices it takes like 20 to 30 minutes.

[marko911]

Any update on this ? Anyone able to get an iso with an older kernel version ?

[justinmoon]

I ended up getting it working with an older kernal, but don't remember the details.

[marko911]

I ended up getting it working with an older kernal, but don't remember the details.

Would love to know how you actually specify kernel when building iso or finding iso with specific kernel ?

[justinmoon]

I ended up getting it working with an older kernal, but don't remember the details.

Would love to know how you actually specify kernel when building iso or finding iso with specific kernel ?

I used nixos 21.05 in VMWare Fusion Tech Preview. Not sure about more than that.

[dmartzol]

I was not able to install 21.05 either on M1 using VMWare Fusion Tech Preview.

Any update on this ? Anyone able to get an iso with an older kernel version ?

@marko911, you can find ISOs here: https://releases.nixos.org/?prefix=nixos/

[marko911]

@dmartzol I was able to install 21.05 but now running the bootstrap script fails with

building the system configuration...
error: attribute 'shellDryRun' missing

       at /nix/store/d5j5vgsfvwkaqmc1qhwaay5mp51yaqkm-source/modules/programs/bash.nix:12:9:

           11|     checkPhase = ''
           12|       ${pkgs.stdenv.shellDryRun} "$target"
             |         ^
           13|     '';

[dmartzol]

@dmartzol I was able to install 21.05 but now running the bootstrap script fails with

building the system configuration...
error: attribute 'shellDryRun' missing

       at /nix/store/d5j5vgsfvwkaqmc1qhwaay5mp51yaqkm-source/modules/programs/bash.nix:12:9:

           11|     checkPhase = ''
           12|       ${pkgs.stdenv.shellDryRun} "$target"
             |         ^
           13|     '';

Yes, sorry when I said I was not able to install I was referring exactly to this same error

[marko911]

I think the issue is that all the config files are using 22.05 as the version and so the missing attribute error throws because that attribute doesn't exist in the 21.05 version which is using old kernel ? My best guess. I'm new to this ecosystem

[nbareil]

For the record, the VMWare issue was fixed in 22H2 (released on July 28th this year), it is compatible with the kernel v5.18.6 (and 5.19.0 as well) available in NixOS 22.05.

🍾 🥳

[marko911]

@nbareil Yes, I just re-did it and it does work. But I still get the same error as above

(root@172.16.218.128) Password: 
building the system configuration...
error: attribute 'shellDryRun' missing

       at /nix/store/d5j5vgsfvwkaqmc1qhwaay5mp51yaqkm-source/modules/programs/bash.nix:12:9:

           11|     checkPhase = ''
           12|       ${pkgs.stdenv.shellDryRun} "$target"
             |         ^
           13|     '';

[nbareil]

error: attribute 'shellDryRun' missing

Isn't it https://github.com/nix-community/home-manager/issues/2722 ?

[marko911]

error: attribute 'shellDryRun' missing

Isn't it nix-community/home-manager#2722 ?

No , because the versions for me aren't different. they're both 22.05

[marko911]

For anyone wondering, currently if you pull this repo and use aarch64 with 22.05 nixos , you must change lines 50 and 51 in flake.nix from

inherit overlays home-manager;
nixpkgs = inputs.nixpkgs-old-kernel;

to

inherit overlays nixpkgs home-manager;

[nateberkopec]

Confirmed that master works on a new install/VM. I think this can be closed.

[justinmoon]

Confirmed that master works on a new install/VM. I think this can be closed.

What exactly did you do? I just this ISO with VMWare Fusion Tech Preview on an M1 and it never gets past this. I'm not even getting to apply any of the stuff in this repo ...

[mitchellh]

The original issue is indeed fixed with VMW TP 22H2.

[nateberkopec]

@justinmoon 🤷‍♂️ I'm making my own ISO as the README suggests.