Raspberry Pi 5 support

[cor]

Has anyone been able to run NixOS on the newly released Raspberry Pi 5? It's not included yet in the wiki's compatibility matrix.

[samueldr]

"NixOS" proper won't see support for the Raspberry Pi 5.

Just like all other ARM platforms, NixOS support depends on the platform's support in upstream mainline projects.

We will not repeat the mistake we did with the Raspberry Pi 4 and adding bespoke support for a proprietary ecosystem. It is too much work to support for the resources available to NixOS.

Note that this reply applies to any $BOARD, and not limited to ARM things.

So the roadmap to "NixOS booting trivially on Raspberry Pi 5" is:

• upstream support in U-Boot
• upstream support in Linux

Then we can add its U-Boot build to the token pre-baked firmware partition (assuming we still ship it by then) and it should work [just as well as mainline supports it].

---

An additional side-note: With the super-powers conferred to the end-users of NixOS, nothing stops someone from providing useful configurations, overlays, etc to make it work in the meantime as a workaround. Though as implied, it would be user-supported.

[06kellyjac]

I think we're all on the same page that it'd be great to have official support but as @samueldr says a lot of work has gone into supporting the Pi (and the oddities of other SOCs). It'd also probably be less painful to do some of this bespoke work if there was some kind of commitment or timeline from the manufacturers to getting this stuff upstreamed in the end.

https://github.com/NixOS/nixos-hardware/commits/master/raspberry-pi/4 https://github.com/NixOS/nixpkgs/commits/master/pkgs/os-specific/linux/firmware/raspberrypi https://github.com/NixOS/nixpkgs/commits/master/pkgs/os-specific/linux/raspberrypi-eeprom https://github.com/NixOS/nixpkgs/commits/master/pkgs/os-specific/linux/kernel/linux-rpi.nix https://github.com/NixOS/nixpkgs/commits/master/nixos/modules/system/boot/loader/raspberrypi

A nix-community project might be more appropriate this time around so there's a centralized spot to collaborate but it's still distinctly marked unofficial. I'm not sure if it should be general SOCs, split by architectures, Pis, or just Pi5 focused though :shrug:

vision5/riscv support has been going alright without a more central project due to a couple of vocal and active community members + a bit of sponsorship support

[MayNiklas]

I wasn't able to get the Pi 5 to boot with NixOS. The Linux kernel received some commits, that should help in the future.

The boot loader shows me the following:

Device-tree file "bcm2712-rpi-5-b.dtb" not found.

The installed operating system (OS) does not indicate support for Raspberry Pi 5
Update the OS or set os_check=0 in config.txt to skip this check.

Wasn't able to get it to boot - no matter what I've tried.

[softerdk]

My pi5 needs Nixos :-)

[WizardUli]

I wasn't able to get the Pi 5 to boot with NixOS. The Linux kernel received some commits, that should help in the future.

The boot loader shows me the following:

Device-tree file "bcm2712-rpi-5-b.dtb" not found.

The installed operating system (OS) does not indicate support for Raspberry Pi 5
Update the OS or set os_check=0 in config.txt to skip this check.

Wasn't able to get it to boot - no matter what I've tried.

You need to use patched kernel sources from Raspberry. You also need to use device tree files from Raspberry and merge them using opensource but non-standard dtmerge (unless Pi5 bootloader changed significantly which is also a possibility).

I wholeheartedly support the decision not to deal with these oddities officially and support only boards with all drivers & patches & DTSs in the upstream kernel & having standard EFI!

But it is NOT that hard to get a new embedded system running in Nix. Not at all. I actually found Nix through embedded systems when I was fed up after many years with Buildroot and similar build systems and Nix (but only with (those unstable) Flakes because of very practical reasons!) when used for embedded systems is like a fairy-tale in comparison.

I do plan to get Pi5 running along with a simplified disk image generation (lots of unneeded magic stuff surrounding the current make-ext4-fs.nix and all its wrappers galore) some-when next year.

[ElvishJerricco]

There are actually patches for u-boot for the rpi5, though they're far from complete. I was able to use that to get a very haphazard build of nixos's sd image booting, but it's pretty rough.

[leo60228]

I got things sort of working without U-Boot: My code is at https://github.com/leo60228/nixpkgs/tree/rpi5, but this isn't really in a state where I'd recommend using it. It's probably best to wait until U-Boot support is in a usable state.

[leo60228]

NixOS seems to "just work" with https://github.com/worproject/rpi5-uefi, though device support is currently very limited (Ethernet and fan control are the most painful omissions for headless/server use). For some applications, it might already be enough, though.

Compared to U-Boot, this has the major benefit of being able to just use the standard aarch64 installer with a mainline kernel. If device tree support gets added to EDK2, it'd hopefully be possible to switch to the vendor kernel post-installation for more complete hardware support, without needing a special Pi-specific installer image.

Using the vendor kernel with ACPI might be superior in some senses, but this seems fragile, and I really doubt it'd be considered a supported usecase by the driver developers.

[leo60228]

There are a number of reasons this is a bad idea, but installing NixOS using EDK2 and then adding this to the configuration seems to mostly work:

{
  boot.kernelPackages = (import <nixpkgs-rpi5> {}).linuxPackages_rpi5;
  boot.kernelParams = [ "dtb=\\bcm2712-rpi-5-b.dtb" ];
}

[leo60228]

rpi5-uefi 0.2 adds device tree support, which seems to work perfectly when using the vendor kernel with UEFI! I'll try to make a flake with the Pi 5 kernel and start writing a wiki page.

[leo60228]

I've created a preliminary wiki page with basic installation instructions: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_5

[tkerber]

The above wiki entry is very helpful for getting the PI to boot, but I've had no success booting to X11 or Wayland. While the wiki does list "Graphics drivers are not yet available in NixOS.", this surprises me, given that mesa 23.3.0 has since made it into nixpkgs, which I believe has the drivers for the pi's GPU.

If anyone gets graphics working, I'd be curious how :)

[nbdd0121]

@leo60228 I tested rpi5-uefi and you flake, and it seems to be working perfectly! Thanks a lot!

I think the only addition in your flake compared to the upstream linux-rpi vendor kernel is the added defconfig line? If so, would you mind submitting a PR so we can just use linuxKernel.kernels.linux_rpi5 (and hopefully be able to use the binary cache)?

[leo60228]

I think the only addition in your flake compared to the upstream linux-rpi vendor kernel is the added defconfig line? If so, would you mind submitting a PR so we can just use linuxKernel.kernels.linux_rpi5 (and hopefully be able to use the binary cache)?

I made that flake before the recent PR updating linux-rpi in Nixpkgs, but this is more or less correct. I was waiting on that PR to be merged before submitting one adding linux_rpi5, I'll try to do that once I'm home tonight if I remember.

[ElvishJerricco]

Realistically, we should be able to build one rpi kernel for rpi 3, 4, and 5. That's how raspbian works. We can even do the same with u-boot, using u-boot's rpi_arm64_defconfig.

[nbdd0121]

Apparently according to https://www.raspberrypi.com/documentation/computers/linux_kernel.html, rpi5 uses a different config so it uses 16K page instead of 4K.

[ElvishJerricco]

@nbdd0121 hm, seems like it's a tradeoff of performance vs compatibility. For performance, use the rpi5 specific kernel, for compatibility, use the rpi4 one.

[RyanGibb]

I've hacked around and got the installation medium booting from the EFI. Does anyone know if it's possible to install NixOS to the same medium (SD card) as the EFI? The EDK2 docs are elusive.

[arti5an]

I used the wiki page @leo60228 created as somewhat of a guide (thanks - you're a star), got my Pi 5 to boot the UEFI from SD, imaged the aarch64 minimal installer to a USB drive and booted that. From there I was indeed able to install to the SD, with NixOS's boot images sharing the same partition as the UEFI loader. Whether that's a good or bad idea I don't know. Not sure what would happen if you somehow booted the installer from the same SD however. I suspect having the partition(s) mounted might hinder those efforts. :shrug:

A few notes from my experience which may be useful:

1. I have the official cooling heatsink/fan, and after booting the UEFI image it locked on full speed. The CPU also runs at full speed with the standard kernel, so that first install and kernel build get a little noisy.

2. This one's easily avoided, but may be what allowed me to use the same partition for boot... I neglected to repartition my SD to GPT (I started with the official Pi image and partitions on there). I simply reformatted the existing partitions to save a step (or so I thought). I later found out it was an MBR partition table when I tried to use systemd-boot via configuration.nix. I resorted to disabling that and using grub as it was a little too late in the day to start again. Rookie mistake!

3. Finally, probably through lack of experience at this level and limited free time, I haven't managed to get a desktop environment to boot once it's installed and the vendor kernel applied. I suspect that probably just needs some config.txt modifications, so if I crack that particular case I'll share my findings and/or submit a PR. I plan to investigate how this was pulled off (and works so beautifully I might add) on my Pi 4. I doubt I'll get there before someone with superior Nix-fu to my own however. :smile:

That aforementioned partition table issue I do plan to rectify in due course, but currently that kernel build from @leo60228 is unfortunately an overnight job for my Pi, so I don't have the free time to keep retrying at the moment. If I do get around to trying that then I'll report my findings, good or bad, unless someone beats me to it.

[RyanGibb]

That aforementioned partition table issue I do plan to rectify in due course, but currently that kernel build from @leo60228 is unfortunately an overnight job for my Pi

Try:

$ sudo gdisk /dev/<path>

Then w to confirm converting the mbr table to a gpt table.

[RyanGibb]

I'm another who has got it working with rpi5-uefi and @leo60228's flake!

The debian UEFI docs where useful in understanding you can extract the bootloader to the same partition as the EFI implementation.

For posterity, here are some step-by-step instructions.

Format your SD-card, create filesystems, and extract the Raspberry Pi UEFI EDK2 implementation to the boot entry:

sd-card=/dev/sda
sudo parted $sd-card mklabel gpt
sudo parted $sd-card -- mkpart ESP fat32 1MB 512MB
sudo parted $sd-card -- mkpart primary 512MiB 100%
sudo mkfs.fat -F 32 -n BOOT ${sd-card}1
sudo mkfs.ext4 -L NIXOS ${sd-card}2
curl https://github.com/worproject/rpi5-uefi/releases/download/v0.2/RPi5_UEFI_Release_v0.2.zip --output ./RPi5_UEFI_Release_v0.2.zip
sudo mount ${sd-card}1 /mnt/
sudo unzip ./RPi5_UEFI_Release_v0.2.zip -d /mnt/

Flash NixOS arm64 minimial installer to USB drive:

usb-drive=/dev/sdb
curl https://channels.nixos.org/nixos-23.11/latest-nixos-minimal-aarch64-linux.iso --output ./latest-nixos-minimal-aarch64-linux.iso
sudo dd bs=4M if=./latest-nixos-minimal-aarch64-linux.iso of=${usb-drive} conv=fsync oflag=direct status=progress

On the Pi now, boot into the installer and network it over WiFi (as Ethernet doesn't work).

sudo systemctl start wpa_supplicant
wpa_cli
> add_network
0
> set_network 0 ssid "myhomenetwork"
OK
> set_network 0 psk "mypassword"
OK
> set_network 0 key_mgmt WPA-PSK
OK
> enable_network 0
OK

Mount the SD-card:

sd-card=/dev/mmcblk0p
sudo mount ${sd-card}p2 /mnt
sudo mkdir /mnt/boot
sudo mount ${sd-card}p1 /mnt/boot

Generate and edit your config with nixos-generate-config --root /mnt.

You'll want to set:

  boot.loader.grub.device = "nodev";
  boot.loader.grub.efiSupport = true;

To install the bootloader to /EFI/ on the boot partition.

And to add the vendor kernel with e.g.:

  boot.kernelPackages = (import (builtins.fetchTarball https://gitlab.com/vriska/nix-rpi5/-/archive/main.tar.gz)).legacyPackages.aarch64-linux.linuxPackages_rpi5;

Then you can install NixOS with

sudo nixos-install

Then to boot the vendor kernel, in the UEFI settings switch from ACPI to Device Tree in

Device Manager
-> Raspberry Pi Configuration
-> ACPI / Device Tree
-> System Table Mode

I also removed force_turbo=1 from /boot/config.txt as suggested.

NB I couldn't get NixOS to boot automatically even adding the /EFI/ file with:

Boot Maintenance Manager
-> Boot Options
-> Add Boot Option
-> BOOT[VenHw(<uuid>)/SD(0x0)/HD(1,GPT,<uuid>)]
-> <EFI>
-> <NixOS-boot>
-> grubaa64.efi

So I need to manually boot from file:

Boot Maintenance Manager
-> Boot Options
-> Boot From File
-> BOOT[VenHw(<uuid>)/SD(0x0)/HD(1,GPT,<uuid>)]
-> <EFI>
-> <NixOS-boot>
-> grubaa64.efi

Perhaps this is related to using a GPT partition table.

I also tried to a cross-compiled installation straight to the SD card from an x86_64 machine which failed due to grub's perl dependency not building.

sudo nixos-install --root /mnt/sd/ --flake ~/projects/nixos-rat#rat-install
[sudo] password for ryan:
warning: Git tree '/home/ryan/projects/nixos-rat' is dirty
building the flake in git+file:///home/ryan/projects/nixos-rat...
warning: Git tree '/home/ryan/projects/nixos-rat' is dirty
error: build of '/nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv' on 'ssh-ng://rtg24@daintree.cl.cam.ac.uk' failed: builder for '/nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv' failed with exit code 2;
       last 10 log lines:
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::PurePerl.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::Intro.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::DocumentLocator.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::PurePerl::Reader.3
       > Installing /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/share/man/man3/XML::SAX::ParserFactory.3
       > Appending installation info to /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/lib/perl5/5.38.0/aarch64-linux/perllocal.pod
       > Can't locate XML/SAX.pm in @INC (you may need to install the XML::SAX module) (@INC entries checked: /nix/store/886v5dis8c552ahqdp169svcynh0rfxr-perl-aarch64-unknown-linux-gnu-5.38.0-mini/lib/perl5/cross_perl/5.38.0 /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/5.38.0 /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/5.38.0/aarch64-linux /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/site_perl/5.38.0 /nix/store/3iafhgzzbv7n2bx1vj2ffb812mqn54s5-perl-aarch64-unknown-linux-gnu-5.38.0/lib/perl5/site_perl/5.38.0/aarch64-linux /nix/store/5rafz6z3p427rlpyz575xg089igz4fzx-perl5.38.0-XML-NamespaceSupport-1.12-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0 /nix/store/5rafz6z3p427rlpyz575xg089igz4fzx-perl5.38.0-XML-NamespaceSupport-1.12-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0/aarch64-linux /nix/store/qhdfvfqy4vy7p4w4kxr3fh0mb9xmvswv-perl5.38.0-XML-SAX-Base-1.09-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0 /nix/store/qhdfvfqy4vy7p4w4kxr3fh0mb9xmvswv-perl5.38.0-XML-SAX-Base-1.09-aarch64-unknown-linux-gnu/lib/perl5/site_perl/5.38.0/aarch64-linux /nix/store/p5fzs2zh9faxdnqpzxfmcv17kai0waal-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu/lib/perl5/site_perl .).
       > BEGIN failed--compilation aborted.
       > make: *** [Makefile:743: install_sax_pureperl] Error 2
       For full logs, run 'nix log /nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv'.
error: builder for '/nix/store/vn4d9m24vfx4xirvc7xa5i8qzyi43mai-perl5.38.0-XML-SAX-1.02-aarch64-unknown-linux-gnu.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/kpz47x1vwg4k9byid1fb7rb4wa1rf3ck-perl-aarch64-unknown-linux-gnu-5.38.0-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/mba97l6yh5f7rx9xrc55mf5csvg51m3s-install-grub.sh.drv' failed to build
error: 1 dependencies of derivation '/nix/store/c5d5z8hrgh8x7w0nfklyvvhf48jc8fzb-nixos-system-rat-23.11.20240115.b8dd8be.drv' failed to build

Perhaps cross-compiling with QEMU would work.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/install-nixos-on-raspberry-pi-5/38833/2

[leo60228]

I couldn't get NixOS to boot automatically

I don't think EFI runtime variables are supported (yet?), so you should set boot.loader.efi.canTouchEfiVariables = false; (and boot.loader.grub.efiInstallAsRemovable = true; for GRUB, I think? I'm using systemd-boot).

Finally, probably through lack of experience at this level and limited free time, I haven't managed to get a desktop environment to boot once it's installed and the vendor kernel applied. I suspect that probably just needs some config.txt modifications, so if I crack that particular case I'll share my findings and/or submit a PR.

You need to add dtoverlay=vc4-kms-v3d-pi5 for the GPU driver to work (I've added this to the wiki page). From there, Wayland DEs should work. I wasn't able to get X11 to work, and from what I can tell upstream doesn't support it on this hardware. I had some issues with graphical corruption on KDE that didn't happen on GNOME, but I was going to wait for Plasma 6 before looking in to that further.

[cmacrae]

Thanks for all the useful conversation had here and resources put together! I was able to get NixOS installed with relative ease on my Pi 5 yesterday, which has now replaced my previous NixOS Pi 4 as a DHCP, DNS, WireGuard, and nginx server for my network and has been working great, much more performant :sparkles:

I figured I'd share a little bit of my experience here in case it helps others.
For the most part, I followed @RyanGibb's handy overview to get going. The only difference really is that I made sure to set the ESP property on the first partition of the SD card with parted:

parted /dev/mmcblk0 -s set 1 esp on

Then proceeded with the mounting. Generate the config. Edit the config to what should roughly be a minimal working system:

configuration.nix

```nix { config, lib, pkgs, ... }: { imports = [ ./hardware-configuration.nix ]; boot.loader.systemd-boot.enable = true; boot.loader.efi.canTouchEfiVariables = false; networking.hostName = "rpi5"; # Define your hostname. networking.wireless.iwd = { enable = true; settings.General.EnableNetworkConfiguration = true; }; time.timeZone = "Europe/London"; environment.systemPackages = with pkgs; [ vim curl git ]; # just for access after install to continue provisioning services.openssh.enable = true; services.openssh.settings.PermitRootLogin = "yes"; networking.firewall.enable = false; system.stateVersion = "23.11"; } ```

hardware-configuration.nix

```nix { config, lib, pkgs, modulesPath, ... }: { imports = [ (modulesPath + "/installer/scan/not-detected.nix") ]; boot.initrd.availableKernelModules = [ "usbhid" "usb_storage" ]; boot.initrd.kernelModules = [ ]; boot.kernelModules = [ ]; boot.kernelPackages = (import (builtins.fetchTarball https://gitlab.com/vriska/nix-rpi5/-/archive/main.tar.gz)).legacyPackages.aarch64-linux.linuxPackages_rpi5; boot.extraModulePackages = [ ]; fileSystems."/" = { device = "/dev/disk/by-uuid/ecb26648-686e-403c-a415-406ac554653d"; fsType = "ext4"; }; fileSystems."/boot" = { device = "/dev/disk/by-uuid/0CAE-25FE"; fsType = "vfat"; }; swapDevices = [ ]; networking.useDHCP = lib.mkDefault true; nixpkgs.hostPlatform = lib.mkDefault "aarch64-linux"; } ```

Made sure to set boot.loader.efi.canTouchEfiVariables = false; and set boot.kernelPackages to use @leo60228's kindly provided flake output.

Before continuing with the install, here's what others may find particularly useful: the Pi is going to take a loooooong time to compile the kernel. If, like me, you happen to be using NixOS on another, more ample aarch64 system (I'm running NixOS with Asahi on an M2 MacBook Air), you can just build your Pi's system closure on that and then copy it to the installer's Nix store:

1. sshd should be on already for the nixos-installer environment, but if it's not, start it

2. set a password for the root user (sudo -i to get a root shell)

3. On your other aarch64 system, make sure you replicate the minimal "bootstrap" config for your Pi. I just defined a nixosConfiguration for my Pi in my flake with the config I laid out above

4. Build the system closure for the Pi on your more ample system. For me, with rpi5 being the name of the nixosConfiguration in my flake, that looked like:

   nix build -L .#nixosConfigurations.rpi5.config.system.build.toplevel

   (if you have problems with impurity because of flakes, either pass --impure or include the kernel flake as an input and reference the package that way)

5. Wait for the build to complete, grab some tea or a coffee :coffee:

6. When the build is done, copy the results to the nixos-installer's Nix store:

   nix copy -s --to ssh://root@<IP of your running nixos-installer> ./result 

7. Once the copy completes, you can run nixos-install as usual - all the results have already been built, so it'll only need to do a few more bootstrapping steps and copying things around

8. Finish the install as usual (set the root password, etc.)

9. Reboot and follow the steps @RyanGibb illustrated to move off ACPI

10. Proceed to save settings, remove your installer USB so it's just the SD card left, use the boot manager to try booting that, it'll prompt you to "Reset". Answer 'Y' and soon you'll, hopefully :crossed_fingers:, be greeted by the systemd boot menu

I appreciate that's a bit of a :brain: dump, so please feel free to ask for any clarification on the above steps.

One other curiosity I had before I did the install was if ethernet worked once booted to NixOS with the vendor kernel and Device Tree/System Table mode. It wasn't clear to me based on comments/info available if it would. For anyone wondering: it does :)

[miangraham]

Big thanks for the kernel packages and guides.

the Pi is going to take a loooooong time to compile the kernel

For anyone else who's curious, I ran the experiment to put a number on it. Haven't gone through with the installation yet, but building from a Pi 5 4GB running debian on an nvme drive, the result was:

$ nix build -L .#nixosConfigurations.....config.system.build.toplevel
# ...
took 2h9m50s

It's long, but it's not waiting for hydra long. ;)

[i-can-not-program]

I have installed NixOS on my rpi5 but the gpu doesn't work, it just uses llvmpipe. i have set dtoverlay=vc4-kms-v3d-pi5 in /boot/config.txt and changed to device tree mode It would be appreciated if someone could provide a example config with the gpu working

flake.nix

```nix { inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; nix-rpi5.url = "gitlab:vriska/nix-rpi5"; nix-rpi5.inputs.nixpkgs.follows = "nixpkgs"; }; outputs = inputs: { nixosConfigurations.rpi = inputs.nixpkgs.lib.nixosSystem { modules = [ ./rpi.nix ./configuration.nix { _module.args = {inherit inputs;}; } ]; }; }; } ```

configuration.nix

```nix { config, pkgs, lib, ... }: { imports = []; networking = { networkmanager.enable = false; wireless.iwd = { enable = true; settings = { General.EnableNetworkConfiguration = true; }; }; resolvconf.enable = false; dhcpcd.enable = false; }; environment.etc."resolv.conf".text = "nameserver 1.1.1.1"; services = { desktopManager.plasma6.enable = true; displayManager.sddm.enable = true; displayManager.sddm.wayland.enable = true; }; sound.enable = true; hardware.pulseaudio.enable = false; security.rtkit.enable = true; services.pipewire = { enable = true; alsa.enable = true; alsa.support32Bit = true; pulse.enable = true; }; nix.settings.experimental-features = [ "nix-command" "flakes" ]; system.stateVersion = "24.05"; } ```

rpi.nix

```nix { config, lib, pkgs, modulesPath, inputs, ... }: { imports = [(modulesPath + "/installer/scan/not-detected.nix")]; boot.loader.systemd-boot.enable = true; networking.hostName = "rpi"; boot.initrd.availableKernelModules = [ "usbhid" ]; boot.initrd.KernelModules = []; boot.kernelModules = []; boot.extraModulePackages = []; boot.kernelPackages = inputs.nix-rpi5.legacyPackages.aarch64-linux.linuxPackages_rpi5; boot.kernelParams = [ "quiet" ]; fileSystems."/" = { device = "/dev/disk/by-label/nixos"; fsType = "ext4"; }; fileSystems."/boot" = { device = "/dev/disk/by-label/EFI"; fsType = "vfat"; options = [ "umask=077" ]; }; nixpkgs.hostPlatform = lib.mkDefault "aarch64-linux"; } ```

[malteneuss]

@RyanGibb Do you or anyone else know a way to create directly an sd image that could be dd onto the sd (or nvme ssd with https://github.com/worproject/rpi5-uefi v0.3)? I believe it should be possible with https://github.com/nix-community/nixos-generators/ but i don't understand that library well enough yet.

[RyanGibb]

@RyanGibb Do you or anyone else know a way to create directly an sd image that could be dd onto the sd (or nvme ssd with https://github.com/worproject/rpi5-uefi v0.3)? I believe it should be possible with https://github.com/nix-community/nixos-generators/ but i don't understand that library well enough yet.

Indeed it is. Nixpkgs contains a NixOS module to create an SD image. It looks like nixos-generators uses this under the hood. See an example here, though I don't think I ever got it working due to cross compilation issues https://github.com/NixOS/aarch64-build-box/pull/182

[WizardUli]

@RyanGibb Do you or anyone else know a way to create directly an sd image that could be dd onto the sd (or nvme ssd with https://github.com/worproject/rpi5-uefi v0.3)? I believe it should be possible with https://github.com/nix-community/nixos-generators/ but i don't understand that library well enough yet.

For RPi4 with UEFI firmware 1.36 (1.37 doesn't work) and you need to switch to a setting to use devicetree instead of ACPI in its setup. https://gitlab.com/WizardUli/nixsauce I'm using a separate SD card for the UEFI firmware so I don't have to change to devicetree from ACPI every time (I don't know how to do it programmatically yet) and dd the resulting image to an USB flash disk.

I guess it can be made to create RPi5 images quite easily.

Personal opinion: nixos-generators are needlessly complicated and unwieldy layer over an already needlessly complicated functions in nixpkgs.

[malteneuss]

Indeed it is. Nixpkgs contains a NixOS module to create an SD image. It looks like nixos-generators uses this under the hood. See an example here, though I don't think I ever got it working due to cross compilation issues NixOS/aarch64-build-box#182

Ah nice. Would it be possible for you to automate your steps of the sd card partition preparations into such an image? The way i understand this thread is that we have all the building blocks to let users build custom NixOS non-installer images that can be cloned onto an SD card and booted from. So last step is to make it as convenient as the official Raspberry Pi installer for debian images, probably along the lines of @WizardUli setup https://gitlab.com/WizardUli/nixsauce/-/blob/main/lib/rpi-image.nix?ref_type=heads. I can help with testing and extending the documentation at https://wiki.nixos.org/wiki/NixOS_on_ARM/Raspberry_Pi_5

Regarding the cross-compiliation: Couldn't we use bin-fmt to build a native image without cross-compiling (albeit slower)?

[RyanGibb]

Ah nice. Would it be possible for you to automate your steps of the sd card partition preparations into such an image?

So it shouldn't require any such preparation if you're just flashing an SD card image. My instructions were for using the generic aarch64 installer from a USB, booting into the Pi, and installing the UEFI bootloader and vendor specific kernel on the Pi itself.

The way i understand this thread is that we have all the building blocks to let users build custom NixOS non-installer images that can be cloned onto an SD card and booted from.

Yep!

So last step is to make it as convenient as the official Raspberry Pi installer for debian images. I can help with testing and extending the documentation at https://wiki.nixos.org/wiki/NixOS_on_ARM/Raspberry_Pi_5

That sounds like a good idea to me. It looks like @WizardUli 's module adds the UEFI implementation to the SD image. https://gitlab.com/WizardUli/nixsauce/-/blob/main/lib/rpi-image.nix (U-boot would be preferable if and when it's upstreamed though I don't know the status of that). It should be possible to adapt this to put the NixOS installation and bootloader on the same image.

Regarding the cross-compiliation: Couldn't we use bin-fmt to build a native image without cross-compiling (albeit slower)?

No idea... would be interested in hearing if it works though!

[malteneuss]

I guess it can be made to create RPi5 images quite easily.

Personal opinion: nixos-generators are needlessly complicated and unwieldy layer over an already needlessly complicated functions in nixpkgs.

@WizardUli I just saw that you updated your config to build a Pi 5 image. This looks indeed a lot more straight-forward and understandable that going the nixos-generators route. Thanks. Do you have link to a blog post that explains your basic steps? Why run VMs, why in two VM steps, what are those other apps your using? Why qow2? etc...

[malteneuss]

That sounds like a good idea to me. It looks like @WizardUli 's module adds the UEFI implementation to the SD image. https://gitlab.com/WizardUli/nixsauce/-/blob/main/lib/rpi-image.nix (U-boot would be preferable if and when it's upstreamed though I don't know the status of that). It should be possible to adapt this to put the NixOS installation and bootloader on the same image.

@RyanGibb Nice. I would like to summarize the current state already in the Wiki. Maybe you can help me answer a few questions: The original Raspberry Debian based SD card image has a custom boot loader on a separate partition mounted as /boot/firmware that loads the kernel (and rest of the system) but doesn't support UEFI. The current setup uses an "EDK2" boot loader that supports UEFI instead in that /boot partition. Here we could also place U-boot. Why do we need UEFI at all and can't we reuse the standard pi boot loader? What advantages has U-boot? Where does the bootloader systemd-boot suggested by @leo60228 in https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1912908878 fit in? Is that another boot-stage after this UEFI bootloader and does this systemd-boot live in the /nix partition then? Are all these boot loaders able to boot headless? I for example have no external monitor and couldn't click or type anything outside an ssh connection. With the custom pi boot loader all boot settings are easily editable via sudo rpi-eeprom-config --edit or sudo nano /boot/firmware/config.txt.

[RyanGibb]

@RyanGibb Nice. I would like to summarize the current state already in the Wiki. Maybe you can help me answer a few questions: The original Raspberry Debian based SD card image has a custom boot loader on a separate partition mounted as /boot/firmware that loads the kernel (and rest of the system) but doesn't support UEFI. The current setup uses an "EDK2" boot loader that supports UEFI instead in that /boot partition. Here we could also place U-boot. Why do we need UEFI at all and can't we reuse the standard pi boot loader? What advantages has U-boot? Where does the bootloader systemd-boot suggested by @leo60228 in #260754 (comment) fit in? Is that another boot-stage after this UEFI bootloader and does this systemd-boot live in the /nix partition then?

So I'm stepping outside my comfort zone a little bit here but I believe U-boot is a first and second stage bootloader targetting embedded devices. Whereas UEFI is a first-stage bootloader and grub/systemd-boot are second-stage bootloaders (which are installed to the /efi/ partition). I think the debian image uses bootcode.bin from https://github.com/raspberrypi/firmware but this doesn't seem to be open source.

Are all these boot loaders able to boot headless? I for example have no external monitor and couldn't click or type anything outside an ssh connection. With the custom pi boot loader all boot settings are easily editable via sudo rpi-eeprom-config --edit or sudo nano /boot/firmware/config.txt.

I don't have any issues booting U-boot headless. I think it can be configured somehow but I haven't looked into it.

[RyanGibb]

See for more on the raspberry pi bootloader https://github.com/NixOS/nixpkgs/pull/241534

[WizardUli]

I guess it can be made to create RPi5 images quite easily. Personal opinion: nixos-generators are needlessly complicated and unwieldy layer over an already needlessly complicated functions in nixpkgs.

@WizardUli I just saw that you updated your config to build a Pi 5 image. This looks indeed a lot more straight-forward and understandable that going the nixos-generators route. Thanks. Do you have link to a blog post that explains your basic steps? Why run VMs, why in two VM steps, what are those other apps your using? Why qow2? etc...

I have not updated my config to build a Pi 5 image. I said that it can be made (as in modified/updated) to build a Pi 5 image.

The installation is split into two steps because unfortunately the systemd-boot's (which is what I'm using at the moment) install script requires running on target's architecture. It could be rewritten to be generic and or I could also try using grub. Anyway, notice that:

• bootloader installation runs under rpiSys.pkgs.vmTools.runInLinuxVM where rpiSys is a derivation representing target's system so its .pkgs are aarch64-linux and the runInLinuxVM is emulated and hopelessly slow.
• I tried leaving the first step under (binfmt) emulation too, but nixos-install calls nix-env --set which for some reason totally exploded in RAM usage under emulation... like 64G RAM full and swapping heavily. So I run it under pkgs.vmTools.runInLinuxVM where pkgs are native pkgs passed in from top. (they deserve to be renamed to nativePkgs)

I've seen people doing nix store and closure copying magic manually without nixos-install (which is mostly just calling the nix-env --set) but most of them were doing some things after the first boot which as you can observe is not necessary at all.

I want to simplify it further by removing nixos-install and play with nix-env myself (which is still reasonably high level command) I also want to look at installing systemd-boot without emulation. Only after that there will be a blog.

I'm also not interested at all at using uboot (or for f'sake native raspberry bootloader which doesn't even support boot entries so whole NixOS concept of booting to a previous/different configuration goes out of window). For anyone trying my image builder I want to reiterate that you need change some settings at the first boot if you are using the UEFI in my image... https://github.com/pftf/RPi4/issues/253 Workaround is to use a separate SD card for that UEFI image (and you can then remove it from my image builder since you are going to be using the one on the separate SD card).

qcow2 is inconsequential. Anything compressed is OK. Since the VM runs are derivations and that image is their input/output cached in a nix store, you don't want it to be uncompressed. Same goes for the final image. It's zstd compressed in order not to take precious disk space.

[djrmarques]

Hello! I have a probably very ridiculous question, but I don't have the knowledge to figure out the answer by myself. Why is it not possible to create nix definition that takes the kernel and bootloader used in Raspbian, and just uses it to boot rpi5?

I mean this code here:

• https://github.com/raspberrypi/firmware
• https://github.com/raspberrypi/linux

Also thanks a lot for the very informative thread.

EDIT: So I see we are already using the firmware here. But we are not pulling the kernel right?

EDIT2: We are also getting the linux from RPI here.

[WizardUli]

Hello! I have a probably very ridiculous question, but I don't have the knowledge to figure out the answer by myself. Why is it not possible to create nix definition that takes the kernel and bootloader used in Raspbian, and just uses it to boot rpi5?

I mean this code here:

* https://github.com/raspberrypi/firmware

* https://github.com/raspberrypi/linux

Also thanks a lot for the very informative thread.

EDIT: So I see we are already using the firmware here. But we are not pulling the kernel right?

Of course it is possible. Why wouldn't it be possible?

[djrmarques]

Hello! I have a probably very ridiculous question, but I don't have the knowledge to figure out the answer by myself. Why is it not possible to create nix definition that takes the kernel and bootloader used in Raspbian, and just uses it to boot rpi5? I mean this code here:

* https://github.com/raspberrypi/firmware

* https://github.com/raspberrypi/linux

Also thanks a lot for the very informative thread. EDIT: So I see we are already using the firmware here. But we are not pulling the kernel right?

Of course it is possible. Why wouldn't it be possible?

But then, and this is probably coming from my ignorance, I don't understand what is the big problem in supporting rpi5? I mean, it's just a matter of specifying the kernel from Raspbian no?

Ugly pseudo code incomming:

boot.kernelPackages = pkgs.linuxPackagesGetKernelFromRaspbian;

I don't have an rpi5 to try this out. In fact, I am using this thread to make a decision on if I should buy an Rpi5 or not :D

[nbdd0121]

You've already got boot.kernelPackages = pkgs.linuxPackages_rpi4. RPi5 works with the rpi4 kernel in unstable branch (not 23.11 branch though).

[djrmarques]

You've already got boot.kernelPackages = pkgs.linuxPackages_rpi4. RPi5 works with the rpi4 kernel in unstable branch (not 23.11 branch though).

I see.. Hopefully it will also work on 24.05 right? But if we could all along pull the kernel from raspbian (which works on all rpis), what was the problem with supporting rpi5? Sorry for the torrent of noob questions.

[cmacrae]

I see.. Hopefully it will also work on 24.05 right? But if we could all along pull the kernel from raspbian (which works on all rpis), what was the problem with supporting rpi5? Sorry for the torrent of noob questions.

I'll try my best to provide some more context on why, generally speaking, your ideas for an external kernel may not be considered a conventional approach in Nix :)

Pulling in a pre-compiled kernel from an external source who's build inputs, environment, configuration, and instructions for use in a system that's (likely) trying its best to be declarative, reproducible, and hermetic is a bit of an anti-pattern. As others have suggested, I'm sure you could get it to work, no problem. But it's not really in keeping with Nix ideals.

Now, that's just for use in one's personal configuration. If I'm understanding correctly, you're also suggesting to potentially have that inherited in nixpkgs/NixOS? Perhaps I'm misreading it, but on top of the above reasons (which become far more "serious" when considering nixpkgs/NixOS): even if the good folks here maintaining NixOS were to decide to inherit the build process for the Raspbian kernel, maintaining platform specific kernels has proven an absolute nightmare in the past. Raspbian is a project focused solely on providing a distribution for one set of boards. NixOS is intended to be a general purpose distribution, capable of running on many systems.

Hopefully that clears some things up? :)

Essentially, there's a history in NixOS of a very-loosely similar path being walked and it was pretty awful from a maintainability perspective. Personally - and I expect this likely extends to others involved in this conversation - using custom stuff for your own personal config to get you running for now is fine, but ultimately I'm waiting for the Pi5 specific kernel patches to upstream into the mainline Linux kernel and wait for that to filter down into NixOS

[djrmarques]

Hi @cmacrae! Thanks a lot for the detailed response. It really helps beginners like me. I also understand now the point that everyone was mentioning: The mainline Linux Kernel does not yet support rpi5. This is what I was not understanding.

Once again, thanks :)

[malteneuss]

I researched a bit about the boot process on the Pi 5 in order to better understand what we are adapting here. I summarized my understanding at https://wiki.nixos.org/wiki/NixOS_on_ARM/Raspberry_Pi_5. Please correct and extend what is wrong or missing.

[celesrenata]

I have installed NixOS on my rpi5 but the gpu doesn't work, it just uses llvmpipe. i have set dtoverlay=vc4-kms-v3d-pi5 in /boot/config.txt and changed to device tree mode It would be appreciated if someone could provide a example config with the gpu working flake.nix configuration.nix rpi.nix

I was also fighting the GPU issue, figured it out today!

We're forgetting to include the raspberrypi overlay.

I threw some more instructions up here https://github.com/celesrenata/nix-flakes/tree/rpi5/README.md

Hopefully they help somebody!

[leo60228]

I knew I was forgetting a step when I added it to the wiki, but couldn't think of what it was.... oops.

[malteneuss]

I just found out that U-Boot in its v2024.04 release got basic pi 5 support (e.g. booting from SD but not USB,NVMe): https://patchwork.ozlabs.org/project/uboot/list/?series=391659&state=%2A&archive=both

Is anyone able to update (or link to the MR) nixpkgs to the new U-boot release? If so, together with https://github.com/NixOS/nixos-hardware/pull/927 (landing in NixOS-hardware soon) we would be quite close to get a convenient Pi 5 setup along the lines of:

# flake.nix
{
  # kernel suitable for pi 5
  inputs.nixos-hardware.url = "github:malteneuss/nixos-hardware/raspberry-pi-5";
  inputs.nixpkgs-unstable.url = "nixpkgs/nixos-unstable";

  outputs =
    inputs@{ self
    , nixos-hardware
    , nixpkgs-unstable
    , ...
    }:

   nixosConfigurations.pi5 = nixpkgs-unstable.lib.nixosSystem {
        modules = [
           my-configuration.nix
           ({ pkgs, modulesPath, ... }: {
              nixpkgs = {
                localSystem = {
                  system = "x86_64-linux";
                };
                crossSystem = { # <--------------cross-compile on Intel/AMD machine. Emulated native compilation with binfmt too slow.
                  system = "aarch64-linux";
                };
              # workaround for breaking linux kernel changes https://github.com/NixOS/nixpkgs/issues/154163#issuecomment-1350599022
               overlays = [   # <-------------------------------
                  (final: super: {
                     makeModulesClosure = x: super.makeModulesClosure (x // { allowMissing = true; });
                   })
               ];
             }; # nixpkgs end
             imports = [
                nixos-hardware.nixosModules.raspberry-pi-5
                "${toString modulesPath}/installer/sd-card/sd-image-aarch64.nix"
             ];
            })
        ];
      };
};

which could then be build into an SD card image using the regular

nix build .\#nixosConfigurations.pi5.config.system.build.sdImage

edit: the new u-boot is already in nixpkgs-unstable: https://github.com/NixOS/nixpkgs/blob/8f9b659d9fb9af9d39cef689e2c3da5dd661b23a/pkgs/misc/uboot/default.nix#L31

[leo60228]

Using crossSystem globally is problematic since it means packages can't be fetched from the binary cache. If I didn't have a fast ARM machine available, I'd probably set crossSystem just for the kernel, and build the image using the Pi as an aarch64-linux remote builder (with Nix installed on Raspberry Pi OS).

[malteneuss]

@leo60228 These are two good setup variations that should be documented somewhere, probably on the Wiki page (i would love a setup where i don't need to fiddle around and install Nix on the Pi to build Nix stuff; just generate a ready NixOS image, dd it onto an sd-card or NVMe and be ready to go^^). The global cross-compile wasn't so bad though on an average laptop; maybe even faster than a native build on the Pi.

Since u-boot in nixpkgs has basic pi 5 support, what can we do now to create a "final" sd card image? My impression is that using something a long the lines of

boot.loader.generic-extlinux-compatible.enable = true;
boot.loader.systemd-boot.enable = false;
boot.loader.efi.canTouchEfiVariables = false;

should be sufficient.

[nvmd]

@leo60228 These are two good setup variations that should be documented somewhere, probably on the Wiki page (i would love a setup where i don't need to fiddle around and install Nix on the Pi to build Nix stuff; just generate a ready NixOS image, dd it onto an sd-card of NVMe and be ready to go^^). The global cross-compile wasn't so bad though on an average laptop; maybe even faster than a native build on the Pi.

Since u-boot in nixpkgs has basic pi 5 support, what can we do now to create a "final" sd card image? My impression is that using something a long the lines of

boot.loader.generic-extlinux-compatible.enable = true;
boot.loader.systemd-boot.enable = false;
boot.loader.efi.canTouchEfiVariables = false;

should be sufficient.

@malteneuss You may have a look at this project: https://github.com/tstat/raspberry-pi-nix. What it does is essentially generate such sd-image bootable with u-boot or raspberry pi's boot process, also managing config.txt from nix. It is not ideal, cross-compilation is not covered, for example. Or from an ideology standpoint. But it gets the job done. U-boot will obviously need to be overridden for RPi5. Raspberry Pi's boot works right out of the box for the 5 if you just want to start with something. IMO for a single-use installation image, Raspberry Pi's boot is acceptable, as you don't have more than a single nixos generation anyway.

Here is for https://github.com/tstat/raspberry-pi-nix/blob/master/sd-image/default.nix if you just want to know how sd-image with uboot is generated.