Raspberry Pi 4 support

[majewsky]

Raspberry Pi just announced immediate availability of the Model 4. It will probably be some time until the aarch64 maintainers can get their hands on these, but I figured this can go into the backlog already.

[cmacrae]

@diamondburned I didn't do anything specific, to be honest 🤷‍♂
Here's an example of a working "vanilla" NixOps deployment for the RPi4: https://git.io/JvfOp

[diamondburned]

I just got some issues about my Pi being aarch64 and my computer being x86.

[cmacrae]

@diamondburned You can use a combination of nix.buildMachines and nixpkgs.localSystem.system = "aarch64-linux" to solve this.

You can set your Pi up to be a remote builder for aarch64-linux builds with the following configuration on the NixOps machine you're deploying from. Something like this:

nix.buildMachines = [
  {
    hostName = "10.0.0.2"; # IP or hostname of your Pi here
    sshUser = "root";
    sshKey = "${builtins.getEnv("HOME")}/.ssh/id_rsa";
    systems = [ "aarch64-linux" ];
    maxJobs = 4;
  }
];

You'll first need to SSH as the root user on your NixOps host to your Pi with the private key to accept the fingerprint (or disable hostkey checking in your root user's ~/.ssh/config):

sudo ssh -i ~/.ssh/id_rsa root@10.0.0.2

Then, as shown in that example config I provided above, set nixpkgs.localSystem.system = "aarch64-linux" for the Pi deployment.

~Supposedly this should be enough... however, I also have to pass --option system aarch64-linux on the command line when deploying to force this :(~ Just tried without --option system aarch64-linux and it worked 🎉 Not sure what's changed on my system, as I used to have to do that for it to work, but looks good now 🤷‍♂

[vikanezrimaya]

I've circumvented this by using an option to emulate aarch64 automatically via qemu and binfmt. Most packages are in binary caches anyway...

[cmacrae]

Nice @kisik21 :ok_hand: I'd been planning on doing the same, just not sure how that'd look in Darwin (my NixOps host is a MacBook). Certainly should be achievable though!

[wizeman]

Has anyone had any luck booting a mainline 5.5 kernel (which supposedly supports the rpi4)?

I built it fine and then tried to boot it (without using u-boot) but it wouldn't boot, I just got a black screen after the rainbow screen and the green LEDs keep flashing twice continuously. Not sure what I did wrong...

[vikanezrimaya]

@wizeman any output on serial console? Maybe it's a trouble with the framebuffer.

[wizeman]

@wizeman any output on serial console? Maybe it's a trouble with the framebuffer.

Unfortunately, right now I don't have access to the serial console to check. But either the framebuffer is causing a boot failure in itself or there must also be another failure, because I also can't access the machine over the network, whereas with linux-rpi I can.

I'm just surprised nobody on the entire internet has reported whether the mainline 5.5 kernel works on the rpi4 (or at least, I couldn't find any report as of a couple of days ago).

[aaronjanse]

I've circumvented this by using an option to emulate aarch64 automatically via qemu and binfmt

@kisik21 I've spent hours trying to set this up. Could you point me to some learning resources or how you did it?

[vikanezrimaya]

@aaronjanse there's an option called boot.binfmt.emulatedSystems that needs to be set to ["aarch64-linux"].

[lukass01]

Does anyone have builded image with enable SSH? I am beginner with Nixos.

[jbaum98]

Yeah it's hard to get started without that, and you need NixOS to build one so it's a chicken-egg problem. I think there should be a prebuilt image with SSH enabled for this purpose. I can build one for you right now.

[jbaum98]

@lukass01 Let me know if this works

[diamondburned]

As for me, I used an Ubuntu image someone else made for aarch64 and installed Nix on it. I then built my own image with SSH and WiFi enabled.

This initially did not work, which I've made an issue on: https://github.com/NixOS/nixpkgs/issues/73487.

[jbaum98]

I also used an Ubuntu image to bootstrap.

[lukass01]

@jbaum98 Thanks for your image. I am testing it now but SSH doesn't work for me.

[jbaum98]

@lukass01 I'm not sure what the issue is then. This is the configuration that I used, which is identical to the one that worked for me except I've removed my public SSH key. If you want more help get in touch with me at https://matrix.to/#/@jwaksbaum:matrix.org.

[ghost]

Instructions on this page worked perfectly for me (Raspberry Pi 4B 2GB): ssh and hdmi working! https://lunar.computer/posts/nixos-image-raspberry-pi-4/ NixOS config Hardware config

[purcell]

Instructions on this page worked perfectly for me

Thanks, that image has also booted for me on a 4B 4GB.

[Renha]

@cmacrae @diamondburned I had issues with x86-64 host until I told that this helps: https://github.com/NixOS/nixpkgs/pull/82718

[Robertof]

For anyone having issues with building images on x86_64, #82718 will certainly help but it's sufficient to just use at least QEMU 5.0 (which came out a few days ago). There were some bugs with memory allocation which were fixed and there are no more errors thrown around for syscalls/memory!

As a shameless plug too, if it helps anyone, I made a Docker-based SD image builder which can effortlessy produce .img files with OpenSSH built-in (using a key of your choice), so you don't even have to attach your RPi to a display. It automatically sets up QEMU too, if building on x86_64. (note though that the default configuration provided in my repo builds images for Pi 3s, but it should be just a matter of updating sd-card.nix)

[ConradMearns]

@josephtheengineer I tried your image and it seemed to work fine, however any time I reboot my configuration changes are lost. Have you experienced this?

[caseyhoward]

@ConradMearns I have the exact same problem. I thought it might have been just me. My nix store appears intact. And my /nix/var/nix/profiles directory has my generations so I don't know what's going on. I just have to run nixos-rebuild switch when it boots up. I guess I could have that run automatically after boot but that doesn't seem right.

[ConradMearns]

@ConradMearns I have the exact same problem. I thought it might have been just me. My nix store appears intact. And my /nix/var/nix/profiles directory has my generations so I don't know what's going on. I just have to run nixos-rebuild switch when it boots up. I guess I could have that run automatically after boot but that doesn't seem right.

I'm building an image off of the Ubuntu server image for the pi right now, I'm hoping that if I can't fix the reboot problem, then maybe I can at least build an image that has the packages I want...

Still a pain, as I wasn't able to get anything built in QEMU to boot 😥

[santicalcagno]

I had the same issue (rebooting would lose my changes). What I think might've fixed it is adding the /boot partition manually as it wasn't showing up in hardware-configuration.nix. I'm not sure though, I was experimenting until I saw that changes were applied after a reboot. Hope it helps.

[caseyhoward]

@santicalcagno I think you're right that it has to do with the boot since there is nothing mounted at /boot. When I first ssh into the raspberry pi the /etc/nixos directory is empty. The first thing I tried was copying the sd-card.nix used to build the image to /etc/nixos/configuration.nix. It works fine when I make changes and run nixos-rebuild switch but like stated earlier, the changes aren't persisted on reboot.

Where are you getting a hardware-configuration.nix from? If I run nixos-generate-config it gives me some stuff that doesn't work. I get the error "You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable." There are no boot partitions on my SD card though.

Anyway, I think my main confusion is the going from generating an SD card (which the linked project does make trivial) to being able to treat my raspberry pi like a normal NixOS install. There is no mention of that in the wiki, just how to generate the SD card. I supposed I could use the raspberry pi to build the new image (instead of relying on QEMU) then copy it back to another machine, use that machine to reflash the raspberry pi SD card, then boot the raspberry pi back up, but that's pretty tedious.

I should also mention I'm using a raspberry pi 4 btw and using the sd-image-raspberrypi4.nix module.

[Gaelan]

Here's what worked for me, from my recollection:

1. Boot the pi4 from sd-image-raspberrypi4.nix
2. sudo mount /dev/mmcblk0p1 /boot (this doesn't happen by default for some reason, but is necessary to make things work)
3. Run nixos-generate-config. Because we already mounted /boot, the hardware-configuration.nix will be generated correctly.
4. Edit /etc/nixos/configuration.nix by removing (or commenting out) the grub stuff and adding this instead:

   {
   boot.loader.grub.enable = false;
   boot.loader.raspberryPi.enable = true;
   boot.loader.raspberryPi.version = 4;
   boot.kernelPackages = pkgs.linuxPackages_rpi4;
   }

5. sudo nixos-rebuild switch

Also make sure you have some way to log in before rebooting. :)

[santicalcagno]

@caseyhoward yeah, I forgot to add that I ran nixos-generate-config and I compared the output of the configuration.nix and hardware-configuration.nix with the ones that @josephtheengineer posted above. From there I noticed the /boot entry missing as well as the pi-specific bootloader entries that @Gaelan also mentioned. I am now quite certain that his instructions would be enough to get it to work!

[purcell]

I had the issue with /boot, fwiw, having also run nixos-generate-config after booting from the SD image but without having manually mounted /boot first.

[moritzschaefer]

@Gaelan's steps work very well for me, however I don't manage to get sshd started upon boot. May I ask you to post your configuration.nix on gist (or so)?

[Gaelan]

@moritzschaefer My configuration.nix is spread over several files, so it isn't the easiest to share, unfortunately. Does your configuration.nix include <nixpkgs/nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix>, by any chance? That module brings in all the installer features, including the nixos user and disabling SSH on boot. If you followed my instructions, you shouldn't need it.

[ConradMearns]

@moritzschaefer - I have SSH settings in both my sd-image.nix and configuration.nix

sd-image gist configuration.nix

[chrisanthropic]

Thanks all, I've just started playing with Nixos and was able to get it working on my Raspberry Pi 4b using this process: https://gist.github.com/chrisanthropic/2e6d3645f20da8fd4c1f122113f89c06

The short version is:

• use @Robertof repo to build an image with my ssh key
• ssh in, run nixos-generate-config
• tweak the configuration.nix to:
  • boot on pi (gaelon's settings from above)
  • add my non-root user & ssh key
  • enable sshd
• run nixos-rebuild switch
• run passwd so my user can use sudo when I log in
• reboot

all of my settings persist between reboots

Thanks everyone for helping me get this working!

[moritzschaefer]

Thank you @chrisanthropic @ConradMearns @Gaelan. Will this solution be compatible with future versions of NixOS?

And if so, what's missing to close this issue?

[diamondburned]

Someone above said that a PR was around the corner, but the conversation is hidden.

[domenkozar]

Seems like even the official 64bit doesn't yet support hardware acceleration: https://www.raspberrypi.org/forums/viewtopic.php?f=117&t=275370

[janat08]

I believe then that the manjaro arm supports it: https://forum.manjaro.org/t/opengl-3-or-above-for-rpi4/129246/12

[purcell]

I believe then that the manjaro arm supports it

I think they have GL but not accelerated V4L2 or h264 on 64bit. I found a good summary of the state of things fairly recently: https://www.raspberrypi.org/forums/viewtopic.php?p=1628856#p1628856

[aaronjanse]

Is the RPi4 with 8GB of RAM expected to work with NixOS?

I'm trying to decide if I should buy it yet.

[christiantakle]

Ive successfully built and booted a Raspberry Pi 4 4GiB using https://github.com/Robertof/nixos-docker-sd-image-builder

Using the same image in a Raspberry Pi 4 8GiB I get the following error screen

[ScarletHg]

We will need to update raspberrypi-firmware and a couple of other packages such as the Pi kernel to support the 8GB model before you can boot it.

I’d be grateful if you could try bumping the versions of those, test to confirm that it works without issue, and open PRs. ❤️

[SquircleSpace]

I updated the versions for all raspberrypi things I could find and was able to make an sd-image that boots on my rpi4 8gb. Here's my branch. https://github.com/SquircleSpace/nixpkgs/tree/rpi4-8gb

[samueldr]

@SquircleSpace hah, if you end up making a PR you'll be beating me to the punch. CC me when you do.

I was trying to get the raspberrypi-kernel_1.20200601+arm64-1 going, which is 5.4 based, but I don't really know if that branch is even supposed to be working.

Do you (or anyone else) have any insight about that branch?

EDIT²: after resetting everything, and double-checking it seems to work just fine. I can't see what was wrong.

---

EDIT: #89659 updates the firmware (only).

[janat08]

Can you overlock it, or use the config.txt file in boot dir?

[samueldr]

You can configure anything for config.txt using the boot.loader.raspberrypi.firmwareconfig option. Note that this only applies to this non-universal raspberry pi 4 specific image.

[jbaum98]

I wanted to share this nix expression for the rpi-eeprom-update tool used to update the firmware for the RPI4. I'm not sure it belongs in nixpkgs, but I hope it's useful.

[c00w]

@jbaum98 - given it's licensed under bsd 3 clause, any objections to opening a PR and building it for aarch64?

[nagy]

In order to get a second screen to work, I had to add a new line into boot.loader.raspberrypi.firmwareConfig with the content max_framebuffers=2.

[numinit]

The 30 MB firmware partition on the newer builds on hydra seems too small:

[root@nixos:/etc/nixos]# rm -rf /boot/kernel.img* /boot/initrd* /boot/old

[root@nixos:/etc/nixos]# nixos-rebuild boot
building Nix...
building the system configuration...
updating the boot generations directory...
cp: error writing '/boot/old/4vy7pb97ai65masnrrd44q1l8nyykf86-linux-4.19.118-1.20200601-Image.tmp.7898': No space left on device
warning: error(s) occurred while switching to the new configuration

I thought the aarch64 builder used to produce a 128 MB firmware partition? Was there a discussion about changing this somewhere?

(Edit) Ignore the accidental issue close, misclick on my part. Sorry for the notification.

[caseyhoward]

I ran into this also. To work around it I delete the /boot/old directory whenever it fails.