Raspberry Pi 4: No HDMI console output after upgrade from 21.11 ceaca998029

aij commented 2 years ago

Describe the bug

If I do a nixos-rebuild switch && reboot or boot from the latest nixos-sd-image-22.05.1241.5ec2af9f483-aarch64-linux.img.zst, I no longer get a login prompt/shell and the monitor goes into power saving mode.

Steps To Reproduce

Steps to reproduce the behavior:

On a RPI4 with keyboard and monitor , boot with nixos-sd-image-21.11.335677.ceaca998029-aarch64-linux.img.zst
nixos-generate-config && nixos-rebuild switch && reboot (System comes back up fine)
nix-channel --update && nixos-rebuild switch (Note that console font size changes but otherwise things seem ok)
reboot
Watch the bootloader and kernel output scroll by, then the screen goes black, and instead of a login prompt, it says No Signal

Expected behavior

I expect to see a login prompt after boot, or at least console output showing an error. When booting from the latest installer I expect to see a bash shell prompt.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

This is a minimal config with no xserver enabled. If I enable openssh, it seemed to take a really long time before accepting connections, but eventually it did and I was able to get these errors from dmesg

[   14.500222] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   14.500248] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   21.220149] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[   21.220967] Console: switching to colour frame buffer device 480x135
[   31.460142] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[   41.700144] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CONNECTOR:32:HDMI-A-1] flip_done timed out
[   51.940146] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:60:plane-3] flip_done timed out
[   62.180145] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[   62.256774] vc4-drm gpu: [drm] fb0: vc4drmfb frame buffer device
[   72.676153] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[   82.916122] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CONNECTOR:32:HDMI-A-1] flip_done timed out
[   93.154805] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:60:plane-3] flip_done timed out
[  103.394097] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[  113.633731] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[  123.873547] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CONNECTOR:32:HDMI-A-1] flip_done timed out
[  134.113426] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:60:plane-3] flip_done timed out
[  144.353376] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[  154.592836] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[  164.832220] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CONNECTOR:32:HDMI-A-1] flip_done timed out
[  175.071729] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:60:plane-3] flip_done timed out
[  185.311353] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[  195.807049] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out
[  206.046825] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CONNECTOR:32:HDMI-A-1] flip_done timed out
[  216.286609] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:60:plane-3] flip_done timed out
[  226.526417] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:64:crtc-3] flip_done timed out

nvd diff shows these changes between ceaca998029 and 891016b5cf1 (sorry this was after including my standard configuration [on both sides], so includes some irrelevant packages):

[nix-shell:~]# nvd diff /nix/var/nix/profiles/system-{3,4}-link
<<< /nix/var/nix/profiles/system-3-link
>>> /nix/var/nix/profiles/system-4-link
Version changes:
[U*]  #01  bind                9.16.16-host, 9.16.16-lib, 9.16.16-man -> 9.16.27-host, 9.16.27-lib, 9.16.27-man
[U*]  #02  cpupower            5.10.94 -> 5.10.122
[U.]  #03  cyrus-sasl          2.1.27 -> 2.1.28
[U.]  #04  expat               2.4.3 -> 2.4.7
[U.]  #05  getent-glibc        2.33-78 -> 2.33-123
[U*]  #06  git                 2.33.1, 2.33.1-doc -> 2.33.3, 2.33.3-doc
[U*]  #07  glibc               2.33-78, 2.33-78-bin, 2.33-78-dev -> 2.33-123, 2.33-123-bin, 2.33-123-dev
[U*]  #08  glibc-locales       2.33-78 -> 2.33-123
[U*]  #09  gzip                1.11, 1.11-info, 1.11-man -> 1.12, 1.12-info, 1.12-man
[U.]  #10  initrd-linux        5.10.94 -> 5.10.122
[U*]  #11  libressl            3.4.1, 3.4.1-man, 3.4.1-nc -> 3.4.3, 3.4.3-man, 3.4.3-nc
[U.]  #12  linux               5.10.94, 5.10.94-dev, 5.10.94-modules-shrunk -> 5.10.122, 5.10.122-dev, 5.10.122-modules-shrunk
[U.]  #13  mount-util-linux    2.37.3 -> 2.37.4
[U.]  #14  netdata             1.32.1 -> 1.33.1
[U.]  #15  nixos-system-nixos  21.11.335677.ceaca998029 -> 21.11.337909.891016b5cf1
[U.]  #16  nss                 3.73 -> 3.76.1
[U.]  #17  nss-cacert          3.74, 3.74-p11kit -> 3.77, 3.77-p11kit
[U*]  #18  openssl             1.1.1m x2, 1.1.1m-bin, 1.1.1m-dev, 1.1.1m-doc, 1.1.1m-man -> 1.1.1o x2, 1.1.1o-bin, 1.1.1o-dev, 1.1.1o-doc, 1.1.1o-man
[U.]  #19  python3             3.9.6, 3.9.6-env -> 3.9.13, 3.9.13-env
[U.]  #20  python3-minimal     3.9.6 -> 3.9.13
[U*]  #21  strace              5.16 -> 5.17
[U*]  #22  tailscale           1.20.1 -> 1.22.2
[U.]  #23  tzdata              2021c -> 2022a
[U*]  #24  util-linux          2.37.3, 2.37.3-bin x2, 2.37.3-dev, 2.37.3-lib x2, 2.37.3-man -> 2.37.4, 2.37.4-bin x2, 2.37.4-dev, 2.37.4-lib x2, 2.37.4-man
[U*]  #25  vim                 8.2.4186 -> 8.2.4975
[U*]  #26  wget                1.21.2 -> 1.21.3
[U.]  #27  zlib                1.2.11, 1.2.11-dev -> 1.2.12, 1.2.12-dev
[U*]  #28  zstd                1.5.0, 1.5.0-bin, 1.5.0-dev, 1.5.0-man -> 1.5.2, 1.5.2-bin, 1.5.2-dev, 1.5.2-man
Added packages:
[A.]  #1  initrd-kmod-blacklist-ubuntu  <none>
[A.]  #2  mdadm.conf                    <none>
[A.]  #3  protobuf                      3.19.0
Closure size: 716 -> 719 (707 paths added, 704 paths removed, delta +3).

[nix-shell:~]#

Notify maintainers

Not sure. @blitz @samueldr maybe know something?

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[root@nixos:~]# nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 5.10.122, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.16`
 - channels(root): `"nixos-21.11.337909.891016b5cf1"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

aij commented 2 years ago

I tried switching my monitor from HDMI-0 to HDMI-1 and found the missing output.

When rebooting with HDMI-1, I see the bootloader output, but then nothing from the kernel until the login prompt appears. I'm guessing the kernel output is still being sent to HDMI-0 until it switches to a framebuffer mode.

So, I can change the outputs as a workaround, but it would still be nice to understand what changed and how to fix it properly.

blitz commented 2 years ago

Besides bisecting the changes, I also don't have a good suggestion here. :(

RyanGibb commented 2 years ago

Using HDMI-1 instead of HDMI-0 is also working for me.

The default Raspberry Pi OS doesn't need this workaround.

Some raspberry pi resources [0, 1] suggest setting hdmi_force_hotplug = 1. Perhaps that's worth a try?

[0] https://forums.raspberrypi.com/viewtopic.php?t=34061 [1] https://forums.raspberrypi.com/viewtopic.php?f=28&t=58151

ryanbooker commented 1 year ago

Has anyone found a reliable work around for this? None of the mentioned ideas are working for me, and I can't even use the last working image, it's not compatible with Pi4B I just received.

I see console output for a while, then the monitor turns off and I see double green flashes for a while. But nothing further happens.

RyanGibb commented 1 year ago

@ryanbrooker perhaps try https://github.com/NixOS/nixpkgs/issues/191095#issuecomment-1320982678?

ryanbooker commented 1 year ago

Thanks for the tip @RyanGibb. I'm trying to follow this guide at: https://nixos.org/guides/installing-nixos-on-a-raspberry-pi.html

I'm not sure where I would update that nixos configuration after creating the image on the SD card as described in the guide?

RyanGibb commented 1 year ago

I see! I wasn't aware of this guide. It's a shame that the most recent working build your directed to doesn't seem to work. I had a dig around and it looks like it's a little over 2 years old: https://hydra.nixos.org/build/134721359. (I'm not sure whether this is intentional or not).

Can I ask how familiar you are with NixOS? Are you using it on any other machines?

ryanbooker commented 1 year ago

This is my maiden voyage with NixOS. I use nix for packages on my Macs, but I've never used NixOS before. The Pi is working with Raspbian. But I bought it for tinkering. :)

Oh, and I've tried the latest build from Hydra (about to try the one mentioned in the actual guide). In case it's not clear above, I've also tried the "working" build suggested and it is not compatible with the current Pi4Bs.

RyanGibb commented 1 year ago

I'm not sure where I would update that nixos configuration after creating the image on the SD card as described in the guide?

So normally you would be able to modify the Nixos configuration on the SD image on the Pi itself, but in this case the lack of display output is preventing that. You could try and SSH in, but that would require a SSH server up and running as well as a network connection -- which might require WiFi authentication if you don't have an easily available ethernet connection -- which also requires a configuration change. It's the chicken and the egg!

Fortunately we can pre-compile a custom SD image instead of having to modify it on the Pi directly.

Basically what the default SD image is building is defined here: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/sd-card/sd-image.nix

You can build this yourself with:

$ git clone git@github.com:NixOS/nixpkgs.git
$ cd nixpkgs
$ nix-build nixos -I nixos-config=nixos/modules/installer/sd-card/sd-image-aarch64.nix -A config.system.build.sdImage

So if you modify nixos/modules/installer/sd-card/sd-image-aarch64.nix to add boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi4; I (hope) this image would work.

This is my maiden voyage with NixOS. I use nix for packages on my Macs, but I've never used NixOS before. The Pi is working with Raspbian. But I bought it for tinkering. :)

Glad to hear it it :-)

The reason I ask about your familiarity with nixos is it's also possible to define such an image in your own nixos configuration. My example, using Nix flakes, is here with this fix. I also have an SSH server, authenticated keys, and WiFi connections authenticated through this. But given that this is your first system I would just try the above initially.

ryanbooker commented 1 year ago

Brilliant. Thanks so much for the information. I'll let you know the outcome. :)

RyanGibb commented 1 year ago

You're welcome, and good luck!

ryanbooker commented 1 year ago

It seems that I need to do the build on a linux machine? I found something called darwin.builder that suggests it can be a build runner for linux builds on macOS, but I can't get it to work.

RyanGibb commented 1 year ago

Ah yes, that could be an issue. I've gotten architecture cross compilation working but not OS cross compilation. This is the relevant manual sections: https://nixos.org/manual/nixpkgs/stable/#chap-cross

Can you share the error messages? And is your mac aarch64 or x86_64?

If you have access to a linux box, it might be easier to build the image on that.

ryanbooker commented 1 year ago

Thanks for the link. My Mac is aarch64. I found this, which sounds like it is for this purpose? https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder

ryanbooker commented 1 year ago

This is the error I see, even with darwin.builder running—which appears to be working, as direnv sees it as an option when building local environments.

% nix-build nixos -I nixos-config=nixos/modules/installer/sd-card/sd-image-aarch64.nix -A config.system.build.sdImage
error: assertion '(stdenv).isLinux' failed

       at /Users/ryanbooker/Developer/foss/nixpkgs/pkgs/os-specific/linux/kernel/generic.nix:72:1:

           71|
           72| assert stdenv.isLinux;
             | ^
           73|
(use '--show-trace' to show detailed location information)

RyanGibb commented 1 year ago

Thanks for the link. My Mac is aarch64. I found this, which sounds like it is for this purpose? https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder

That looks like exactly what we need! It looks like it runs a linux VM as a (local) remote builder.

This is the error I see, even with darwin.builder running—which appears to be working, as direnv sees it as an option when building local environments.

% nix-build nixos -I nixos-config=nixos/modules/installer/sd-card/sd-image-aarch64.nix -A config.system.build.sdImage
error: assertion '(stdenv).isLinux' failed

       at /Users/ryanbooker/Developer/foss/nixpkgs/pkgs/os-specific/linux/kernel/generic.nix:72:1:

           71|
           72| assert stdenv.isLinux;
             | ^
           73|
(use '--show-trace' to show detailed location information)

Hmm, it looks like your not using the Linux VM as a remote builder for some reason... When you say direnv sees it as an option for building local environments, what do you mean exactly?

RyanGibb commented 1 year ago

Looks like darwin.builder was merged recently: https://github.com/NixOS/nixpkgs/pull/206951

ryanbooker commented 1 year ago

I have direnv enabled via home-manager on my Mac. When it triggers a build for an environment in a directory that has a shell.nix a log message is output indicating that there is a builder ready to do aarch64-linux builds. I think that really means nix-shell sees the builder…

The log message(s) look like this:

1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([aarch64-linux], 4, [], [])

So that makes me think nix knows about the builder.

RyanGibb commented 1 year ago

It couldn't be because it needs to run as sudo could it? I ran into that as described here: https://github.com/NixOS/nix/issues/1572#issuecomment-1307986633.

Also, it's worth testing the connectivity as described here with:

nix store ping --store 'ssh-ng://builder@localhost aarch64-linux /etc/nix/builder_ed25519 4 - - - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo='

Otherwise I'm not too sure how to debug this with the old CLI. I know with the new CLI that you can use nix build --max-jobs 0 to force a build on the remote only (max-jobs referring to local jobs). Maybe try the following flake:

$ mkdir sd-image-flake
$ cd sd-image-flake
$ echo '{
  outputs = { self, nixpkgs, ... }: {

    nixosConfigurations.sd-image =
      nixpkgs.lib.nixosSystem {
        system = null;
        pkgs = null;
        modules =
          [
            "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
            ({ config, pkgs, ... }: {
              networking.hostName = "sd-image";
              nixpkgs.hostPlatform.system = "aarch64-linux";
              system.stateVersion = "22.11";
              boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;
            })
          ];
        };
  };
}' > flake.nix
$ nix build .#nixosConfigurations.sd-image.config.system.build.toplevel --max-jobs 0 --builders 'ssh-ng://builder@localhost aarch64-linux /etc/nix/builder_ed25519 4 - - - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo='

And add -vvvv to nix build for verbose debugging if this doesn't work.

RyanGibb commented 1 year ago

Also, I should have said this sooner, but have you tried the other HDMI port and/or another (low resolution) monitor?

ryanbooker commented 1 year ago

Also, I should have said this sooner, but have you tried the other HDMI port and/or another (low resolution) monitor?

Looking into the details of your other comment (thank you again), but yeah I've tried the only two monitor options I have and both HDMI ports.

ryanbooker commented 1 year ago

Ok. This doesn't work… but I think this passing the builder description for --store:

nix store ping --store 'ssh-ng://builder@localhost aarch64-linux /etc/nix/builder_ed25519 4 - - - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo='

However, this does work:

nix store ping --store 'ssh-ng://builder@localhost?ssh-key=/etc/nix/builder_ed25519'

I have listed my user in /etc/nix/nix.conf in extra-trusted-users = foo as described in the darwin.builder instructions.

ryanbooker commented 1 year ago

Turns out, this almost works:

nix-build nixos -I nixos-config=nixos/modules/installer/sd-card/sd-image-aarch64.nix -A config.system.build.sdImage --system aarch64-linux

It starts building, and then:

building '/nix/store/bxqfqa3yxp8w4kyfac07aqq9nw126lzf-config.txt.drv'...
building '/nix/store/ifci4y1rakx0a1db2fp9k1iyy2qg8vq1-nixos-manual-combined.drv' on 'ssh-ng://builder@localhost'...
building '/nix/store/p67b4n2912hz23rqsgvx5r8pn5gnrdnf-options-docbook.xml.drv' on 'ssh-ng://builder@localhost'...
/nix/store/fz80arp0cldc6lprk1z11g3d62vpmj78-bash-5.2-p15/bin/bash: /nix/store/fz80arp0cldc6lprk1z11g3d62vpmj78-bash-5.2-p15/bin/bash: cannot execute binary file
copying 0 paths...
building '/nix/store/ifci4y1rakx0a1db2fp9k1iyy2qg8vq1-nixos-manual-combined.drv'...
error: builder for '/nix/store/bxqfqa3yxp8w4kyfac07aqq9nw126lzf-config.txt.drv' failed with exit code 126;
       last 1 log lines:
       > /nix/store/fz80arp0cldc6lprk1z11g3d62vpmj78-bash-5.2-p15/bin/bash: /nix/store/fz80arp0cldc6lprk1z11g3d62vpmj78-bash-5.2-p15/bin/bash: cannot execute binary file
       For full logs, run 'nix log /nix/store/bxqfqa3yxp8w4kyfac07aqq9nw126lzf-config.txt.drv'.
error: 1 dependencies of derivation '/nix/store/np5r3159q0acgl1c85gzpy02mji1calk-nixos-sd-image-23.05pre-git-aarch64-linux.img.drv' failed to build

nix log contains:

/nix/store/fz80arp0cldc6lprk1z11g3d62vpmj78-bash-5.2-p15/bin/bash: /nix/store/fz80arp0cldc6lprk1z11g3d62vpmj78-bash-5.2-p15/bin/bash: cannot execute binary file

I think it may be time to install nix on the raspberry pi's standard install, and then build the nixos image there.

RyanGibb commented 1 year ago

Okay, so, I don't think --system aarch64-linux will help you here. That's just for the new CLI: https://nixos.org/manual/nix/stable/command-ref/new-cli/nix.html?highlight=--system#options. The equivalent for nix-build I think is --argstr system aarch64-linux. I think you're not actually building on the remote as a result of this.

Also, I noticed that 'ssh-ng://builder@localhost?ssh-key=/etc/nix/builder_ed25519' doesn't include the system of the linux VM. I think it should be 'ssh-ng://builder@localhost?ssh-key=/etc/nix/builder_ed25519 aarch64-linux'.

NB there shouldn't be any cross compilation (from Nix's perspective here) as our virtualization buildPlatform (nomenclature from https://nixos.org/manual/nixpkgs/stable/#chap-cross) matches our hostPlatform of aarch64-linux.

If the above doesn't work out I'd encourage you to try the flake option above (the new CLI has a better UX imo).

I'm also happy to build this image for you and share the binary which might be a lot easier!

ryanbooker commented 1 year ago

Thank you again for your help. Part of this is to aid my understanding/learning. So I'm happy to keep trying, though I appreciate the offer.

Re the nix store command, definitely only this works:

nix store ping --store 'ssh-ng://builder@localhost?ssh-key=/etc/nix/builder_ed25519'

If I include the arch it thinks it's part of the key path and fails.

However, on the nix build command, it is now finding the full builder definition that I have in /etc/nix/nix.conf, which does include the arch, etc.

Now we're getting close. I think it's all working as far as the darwin.builder goes. But there is an actual build error now (I've verified the same error is occurring on the Pi and on the builder:

error: build of '/nix/store/600mg4p4h448iyj28788cds6sjsncvwc-linux-5.15.84-1.20230106-modules-shrunk.drv' on 'ssh-ng://builder@localhost' failed: builder for '/nix/store/600mg4p4h448iyj28788cds6sjsncvwc-linux-5.15.84-1.20230106-modules-shrunk.drv' failed with exit code 1;
       last 10 log lines:
       >   copying dependency: /nix/store/4wj95ngckp0zg3ihyw5x62b0af39xkaj-linux-5.15.84-1.20230106-modules/lib/modules/5.15.84/kernel/drivers/message/fusion/mptscsih.ko.xz
       >   copying dependency: /nix/store/4wj95ngckp0zg3ihyw5x62b0af39xkaj-linux-5.15.84-1.20230106-modules/lib/modules/5.15.84/kernel/drivers/message/fusion/mptspi.ko.xz
       > root module: vmxnet3
       >   copying dependency: /nix/store/4wj95ngckp0zg3ihyw5x62b0af39xkaj-linux-5.15.84-1.20230106-modules/lib/modules/5.15.84/kernel/drivers/net/vmxnet3/vmxnet3.ko.xz
       > root module: vsock
       >   copying dependency: /nix/store/4wj95ngckp0zg3ihyw5x62b0af39xkaj-linux-5.15.84-1.20230106-modules/lib/modules/5.15.84/kernel/net/vmw_vsock/vsock.ko.xz
       > root module: simplefb
       >   builtin dependency: simplefb
       > root module: sun4i-drm
       > modprobe: FATAL: Module sun4i-drm not found in directory /nix/store/4wj95ngckp0zg3ihyw5x62b0af39xkaj-linux-5.15.84-1.20230106-modules/lib/modules/5.15.84
       For full logs, run 'nix log /nix/store/600mg4p4h448iyj28788cds6sjsncvwc-linux-5.15.84-1.20230106-modules-shrunk.drv'.
error: builder for '/nix/store/600mg4p4h448iyj28788cds6sjsncvwc-linux-5.15.84-1.20230106-modules-shrunk.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/fdqr3dv8c14i8gxf8wwrycv9xgbf6pc6-stage-1-init.sh.drv' failed to build
error: 1 dependencies of derivation '/nix/store/92hcy2ji71cp9liw8qah12b7479v6kf2-initrd-linux-5.15.84-1.20230106.drv' failed to build
error: 1 dependencies of derivation '/nix/store/s82blwyr204yxr28c1jcqjddcrfmnvlq-nixos-system-nixos-23.05pre-git.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vbjighz5hv6hjqpsjwbyasai8y0fg73w-ext4-fs.img.zst.drv' failed to build
error: 1 dependencies of derivation '/nix/store/np5r3159q0acgl1c85gzpy02mji1calk-nixos-sd-image-23.05pre-git-aarch64-linux.img.drv' failed to build

Which appears to be this issue (link to preposed solution that I haven't tried yet): https://github.com/NixOS/nixpkgs/issues/154163#issuecomment-1350599022

ryanbooker commented 1 year ago

Whoa. I added the overlay in that link and bam. It compiled!

RyanGibb commented 1 year ago

Re the nix store command, definitely only this works:
nix store ping --store 'ssh-ng://builder@localhost?ssh-key=/etc/nix/builder_ed25519'
If I include the arch it thinks it's part of the key path and fails.

Strange...

Whoa. I added the overlay in that link and bam. It compiled!

Magic! Let me know if it boots on the Pi okay.

ryanbooker commented 1 year ago

It boots, but the manual and the installation guide suggest I should have been automatically logged in. However, there is a login prompt and I have no idea what the login details are!

I tried nixos with no password. But that's not it. So close!

RyanGibb commented 1 year ago

Ah, I think you need users.users.root.initialHashedPassword = "";!

This is defined here (and the nixos user here).

That's my bad, it looks like what we really wanted was https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/sd-card/sd-image-aarch64-installer.nix, which imports https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/sd-card/sd-image-aarch64.nix.

ryanbooker commented 1 year ago

Yep. I worked it out. Just copying the new image across to my SD card now…

I'm in. Awesome. Thank you so much or your guidance. I learned a lot, including more places to look for info and answers. Cheers. 🎉

RyanGibb commented 1 year ago

You're welcome! Glad I could help :)

rgov commented 1 year ago

@ryanbooker Would you consider writing up a gist on the full sequence to set up a builder on Mac and create a working bootable NixOS image for RPi 4? It's a little hard to piece together from this thread. Thank you.

XenGi commented 1 year ago

I would be very interested in the write up as well. I spent way to much time on this little thing already. Unbelievable that high resolution monitors (1080p+) are still such a big problem with it. I'm trying to get my rpi4 up and running with a 1440p one here.

n8henrie commented 1 year ago

One other alternative is using GitHub Actions to build the image. This is example is for a Pi 3 but builds an image and uploads it as a release artifact: https://github.com/n8henrie/nixos-btrfs-pi

Similar idea but builds an x86_64 NixOS rescue image that I can download from anywhere and throw on a USB: https://github.com/n8henrie/nixos-rescue

NixOS / nixpkgs