Raspberry Pi 4 support

[majewsky]

Raspberry Pi just announced immediate availability of the Model 4. It will probably be some time until the aarch64 maintainers can get their hands on these, but I figured this can go into the backlog already.

[domenkozar]

It would be especially nice if hardware video acceleration would now finally work due to OSS video firmware

[duckfullstop]

After a definitely-not-impulsive trek to Cambridge yesterday, myself and @nekomimiscience both have Pi 4 4GB's as of yesterday and we can spend some time writing / testing some stuff to get these working properly.

From her experimentation yesterday night, the current kernel shipped with the aarch64 image does not boot, but fudging it by copying the kernel from the new official image (which is now based on debian/buster) at least gets the system to start.

We'll probably need to bring over some of the kernel patches from their fork (see here for a summary, though it'll probably involve quite a lot of the stuff committed on June 21st here), then it will probably be a case of just pulling the firmware Pi3B+ style. Should we be mainlining these patches onto the main kernel tree here, or would it be saner to have a separate package?

[ScarletHg]

Most of the work in getting the Pi 4B to at least boot (without hacks) I've scoped out so far is upstream (U-Boot, Linux kernel).

I'm going to see how far I can get in bringing up U-Boot in my spare time but I don't have too much time to devote to it until Sunday. No promises at this stage.

Otherwise, version bumps on Raspberry Pi related packages seem to be the near-term way forwards (in particular, raspberrypifw and linux_rpi). Then until mainline Linux supports the Pi4B we can at least build images that boot by selecting the linux_rpi kernel.

Extremely excited about the new OSS video firmware though.

[duckfullstop]

There appear to be some blocking issues on the current kernel when built for aarch64 which I would argue block merging it into NixOS.

Some of these are tracked in the following issue: https://github.com/raspberrypi/linux/issues/3032

From what I can ascertain, at the time of writing this comment:

• there seems to be a problem addressing more than 1GB of memory when root is on the SD card, the workaround is to chainload from SD onto a USB drive
• the current tagged version of the kernel (20190620-1) does not compile properly due to a problem with the BCM2835 video driver; this has been fixed in this issue by disabling it altogether, which is far from ideal (but hey, at least it boots and gets to X)

Someone has succeeded in getting Gentoo to boot with an aarch64 kernel, written up here - but that memory problem feels like a bit of a blocker. Keeping an eye on that issue to see if there's a resolution.

I guess once that's fixed, we could target a specific commit of the kernel instead of the 20190620-1 release - though I'm not sure merging that kind of potential instability is a good idea. Open to comment.

[JohnAZoidberg]

If you get it to work, please add the instructions here: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi

[majewsky]

I linked from the wiki page back to this issue so that users consulting the wiki can see what is going on over here.

[tkerber]

I've managed to get nixos fully booting on my new pi4 (and am posting this from it!). I built this fork of the rpi kernel before it was merged: https://github.com/raspberrypi/linux/pull/3144. Compile with bcm2711_defconfig. Beyond that, I had to make the following changes:

• Update wireless firmware.
• Add a derivation to build armstub8-gic.bin. Tbh, I don't actually know what this does, except that it won't boot without it :p
• Update nixos/system/boot/loader/raspberrypi to a) build and copy armstub8-gic.bin, and b) add the corresponding enable_gic=1 and armstub=armstub8-gic.bin lines to config.txt.

Right now, the most obvious deficits you can read about in e.g. the gentoo build, of sd cards, wifi, and RAM not working, are no longer an issue.

Beyond that, it seems like a pretty standard aarch64 build (though it's the first one I've done). I did not look into the sdcard image builder. (I did some messy hack for running the activation script on the chroot'd x86_64 host instead)

I'll try to do a more proper writeup, and make pull reqs for the nixos modules needed over the weekend.

[jbaum98]

@tkerber congrats! I'm excited for the write-up!

Did you get 4GB of ram working? My understanding is that the aarch64 kernel had a bug where it could only use 1GB of ram.

[tkerber]

Not sure to be honest. I've compiled a kernel which I believe to patch the core issue, but I still had symptoms which look very much like those of running without limiting the RAM... I'll try a more up-to-date kernel tomorrow, and see if that persists.

[tkerber]

This issue is known upstream: raspberrypi/linux#3157 - for now, I guess limiting to 1G is the best action to take for a stable system (maybe only while downloading with WiFi as the issue suggests - I wouldn't know, there is little else I do with machines :p).

[yaroslavros]

https://github.com/raspberrypi/linux/pull/3159 should fix WiFi issue

I run arm64 gentoolto on RPI4 4G without any trouble now.

[tkerber]

I've just made a pull request. Here's how to build a working config against it:

Example Config

Replace UUIDs as appropriate.

{ pkgs, ... }:

{
  # Tell the host system that it can, and should, build for aarch64.
  nixpkgs = rec {
    crossSystem = (import <nixpkgs> {}).pkgsCross.aarch64-multiplatform.stdenv.targetPlatform;
    localSystem = crossSystem;
  };

  fileSystems = {
    "/" = {
      fsType = "ext4";
      device = "/dev/disk/by-uuid/67739d77-fe40-46af-8e49-6e6facb79a87";
    };
    "/boot" = {
      fsType = "vfat";
      device = "/dev/disk/by-uuid/5F19-927F";
    };
  };

  services.xserver = {
    enable = true;
    displayManager.slim.enable = true;
    desktopManager.xterm.enable = false;
    windowManager.i3.enable = true;
    videoDrivers = [ "fbdev" ];
  };

  hardware.enableRedistributableFirmware = true;

  users.users.exampleuser = {
    isNormalUser = true;
    password = "badpassword";
  };

  # For the ugly hack to run the activation script in the chroot'd host below. Remove after sd card is set up.
  environment.etc."binfmt.d/nixos.conf".text = ":aarch64:M::\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\xb7\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\xff\\xfe\\xff\\xff\\xff:/run/binfmt/aarch64:";
  boot= {
    kernelPackages = pkgs.linuxPackages_rpi4;
    loader = {
      grub.enable = false;
      raspberryPi = {
        enable = true;
        version = 4;
      };
    };
  };
}

How to Build

This is what I did to set up my microsd. It's quite hacky, but it works. Any command below should be in a root shell (apart from the initial build).

1. Set up your host system with qemu-aarch64 binfmt (https://nixos.wiki/wiki/NixOS_on_ARM#Crosscompiling).
2. Partition your SD card with a small FAT32 boot partition, and a large ext4 root partition.
3. Mount these to /mnt and /mnt/boot.
4. Build your system: NIX_PATH="nixpkgs=path/to/repo" nix build '(import <nixpkgs/nixos> { configuration.imports = [ path/to/config.nix ]; }).system'
5. Run nixos-install --system $(readlink result). Activation will fail.
6. Run install -D /etc/binfmt.d/nixos.conf /mnt/etc/binfmt.d/nixos/conf, and install -D /run/binfmt/aarch64 /mnt/run/binfmt/aarch64 This lets us run aarch64 executables in the chroot.
7. Run the activation scripts: chroot /mnt /nix/var/nix/profiles/system/activate chroot /mnt /run/current-system/bin/switch-to-configuration boot This may still fail. Repeat step 6 if so (nixos really doesn't like that we're trying to force non-sanctioned files in /etc and /run).
8. Find and run passwd as root: ls /mnt/nix/store/*shadow*/bin/passwd chroot /mnt /nix/store/..../bin/passwd to set your password.
9. Unmount, insert into the pi, and enjoy!

[duckfullstop]

The main reason @NekomimiScience and I held off on suggesting that nixpkgs merge the current upstream kernel is because it wasn't possible to address more than 1G of memory on aarch64 (which, in our opinion, was an indication that it probably wasn't merge ready). @tkerber, am I reading the previous comments right in that this is no longer a problem, or is it still not possible to use more than 1G?

[tkerber]

Looking again at raspberrypi/linux#3159, it does seem it's not solved yet. I'm withdrawing my pull req until this gets a consistent upstream fix. I'll leave up the branch, as people can build against it for now. It does seem like it'll still be a few weeks before the kernel is stable, though.

[duckfullstop]

@tkerber If you're able to use the full range of memory, then I'd say it can probably be merged with a "probably unstable" note attached to it. Probably worth a test!

[vikanezrimaya]

I don't plan to use WiFi, since there is Gigabit Ethernet onboard, so if the WiFi is the only issue, I'm ok with it! Are Gigabit Ethernet and USB 3.0 ports working?

I hope to get an RPi4 ~soon-ish, so I might be able to pitch in and help with testing it (mostly in headless mode)

[tkerber]

I can't personally attest to either at the moment. That said, I haven't seen them being brought up as an issue anywhere, and things seem to be moving fast to patch the issues that are remaining.

[yaroslavros]

FWIW, I have just submitted a fresh PR https://github.com/raspberrypi/linux/pull/3164 to address the WIFI DMA issue.

[satmandu]

@tkerber FYI U-boot DOES work on the RPI4 using some work which doesn't look like it has been integrated into upstream yet. My RPI4 is able to boot using this: https://github.com/agherzan/u-boot/tree/ag/v2019.07-rpi4-wip

[lategoodbye]

@satmandu One of the reasons why these changes hasn't integrated into U-Boot upstream yet is the fact that the devicetree sources are under review during Linux upstreaming. It's not a good idea to adopt the downstream files directly.

https://marc.info/?l=linux-arm-kernel&m=156571347332483

[jbaum98]

@tkerber I've tried to follow your instructions, but I don't have a computer with NixOS installed on it. I cloned your branch on an arm AWS instance and tried to build an sd card image, but I could only get a rainbow screen on the pi. Do you have any suggestions on getting the sd card image working?

[tkerber]

@jbaum98 I'm not sure - I don't know enough about the sdcard making process to know what could go wrong. It sounds like an issue in early boot though, so likely the boot partition isn't set up correctly. The pi loads the first partition, which should be fat32, and does a bunch of stuff with it, before booting a specified kernel and initramfs. Sounds like this could be either a) this not being properly set up for whatever reason, or b) it straight up failing to execute the kernel.

[jbaum98]

I got it working following your instructions! I just booted from a live USB of NixOS and it all worked great.

[numinit]

How do I add your git repo as a channel?

Edit: We got it working. You have to export nixpkgs=/path/to/repo in NIX_PATH before switching to the new config.

[wizeman]

It seems like the major issues mentioned in this thread have been solved using the latest commit from the https://github.com/raspberrypi/linux kernel repo?

Specifically, I got the full 4GB of RAM to work fine (and it seems like the Wi-Fi issues have been solved as well, although I don't use Wi-Fi so I can't test it).

Anyone willing to make a PR?

[wizeman]

BTW, note that this wiki page has a config template suggesting to add cma=32M to boot.kernelParams on a RPi3 for the virtual console to work.

Note that if you do that on a RPi4 with the full 4G of RAM, the USB ports will be disabled (however, they will work if you limit RAM to 3G or less).

If you want to make full use of the 4G of RAM and the USB ports, you either need to remove this kernel parameter or increase it to at least 64M (which is the default on a RPi4 if you don't set any value).

I added a comment to the template, although I'm not sure if that's enough of a warning.

[wizeman]

I also just noticed that the nixos/modules/installer/cd-dvd/sd-image-aarch64.nix file in the nixpkgs repo also has the cma=32M parameter (in both master and release-19.03 branches), which most likely will result in the problem I mentioned in my previous comment...

[duckfullstop]

I think we were pretty much set on having to use a separate image builder (potentially targeting a pi4 kernel package) due to some incompatible kernel changes (from what I remember?). Shipping a separate image in the interim is, at least in my opinion, not totally world ending.

(I wonder how other distributions with consoles are going to handle this?)

[samueldr]

See: https://github.com/NixOS/nixpkgs/pull/70796

To be tested, then to be integrated into the generic u-boot-based sd_image. Though sharing already as I may not do those things today.

[sbourdeauducq]

Some changes in nixos-configs are needed for 19.09

diff --git a/overlays/qemu/qemu/default.nix b/overlays/qemu/qemu/default.nix
index 96e6022..9ab5943 100644
--- a/overlays/qemu/qemu/default.nix
+++ b/overlays/qemu/qemu/default.nix
@@ -4,7 +4,7 @@ makeStaticLibraries, glibc, qemu, fetchFromGitHub }:
 let
   env2 = makeStaticLibraries stdenv;
   myglib = (glib.override { stdenv = env2; }).overrideAttrs (drv: {
-    mesonFlags = (drv.mesonFlags or []) ++ [ "--default-library both" ];
+    mesonFlags = (drv.mesonFlags or []) ++ [ "-Ddefault_library=both" ];
   });
   riscv_src = fetchFromGitHub {
     owner = "riscv";
diff --git a/qemu.nix b/qemu.nix
index 0488712..0a7ab81 100644
--- a/qemu.nix
+++ b/qemu.nix
@@ -35,7 +35,7 @@ in {
     nixpkgs = {
       overlays = [ (import ./overlays/qemu) ];
     };
-    boot.binfmtMiscRegistrations =
+    boot.binfmt.registrations =
       optionalAttrs cfg.arm { inherit arm; } //
       optionalAttrs cfg.aarch64 { inherit aarch64; } //
       optionalAttrs cfg.riscv64 { inherit riscv64; };

[ardumont]

Hello, thanks @samueldr and @tkerber for your work on this.

I have a rpi4 running with nixos (building an sd-image based on @tkerber's initial work, from his branch, i did not realize his work got merged at the time). [1]

[1] https://gist.github.com/ardumont/5c40f992dd11d98e51707784365de094

Status: wifi is fine, 4G of ram is used, usb as well (both usb2 and usb3).

My main issue being that it always boots as if it was a fresh sd-card install (nixos-rebuild switch after that and back on my feet to my latest generation). I guess it's because the boot files are not updating during the generation switch (I am not sure whether i am at fault or not here).

So, i was trying to enable u-boot since @samueldr seemed done ;)

Trying to make the rpi4 boot with u-boot enabled, i'm stuck on kernel boot. u-boot starts up fine but then the kernel loading step hangs. But that's expected for now as @samueldr explained [2]

[2] https://github.com/NixOS/nixpkgs/pull/70796#issuecomment-552238845

Again thanks.

Cheers,

[ardumont]

My main issue being that it always boots as if it was a fresh sd-card install (nixos-rebuild switch after that and back on my feet to my latest generation). I guess it's because the boot files are not updating during the generation switch (I am not sure whether i am at fault or not here).

It's indeed not updated when switching generation (in my current setup at least). For now, i overcome my issue by manually editing /boot/firmware/cmdline.txt to target my latest generation (as per referenced in the /boot/extlinux/extlinux.conf entries).

---

More details for those interested:

root@rpi4> cat /boot/firmware/cmdline-original.txt
loglevel=7 init=/nix/store/7zqncsqnaq2d0y2kiwzwsrvqxx5f2m49-nixos-system-nixos-19.09.git.cc5baf2/init  # this matches the 'first' generation in /boot/extlinux/extlinux.conf
root@rpi4> cat /boot/firmware/cmdline.txt                                                                                                                                                                                                    
loglevel=7 init=/nix/store/9bvpahky50f33xp4wmny6q5rw0grgvia-nixos-system-rpi4-19.09.git.cc5baf2/init  # last generation

Note: cmdline.txt is the default file looked up by the current bootloader when nothing is specified in /boot/config/config.txt

[tkerber]

@ardumont this seems like your system config's boot section is not properly defined.

You should have something along the lines of

{
  boot.loader.raspberryPi = {
    enable = true;
    version = 4;
  };
}

Also, I think the bootloader assumes the firmware partition is mounted to /boot, while the sd image mounts it to /boot/firmware.

[ardumont]

@ardumont this seems like your system config's boot section is not properly defined.

Yes, it sure does.

But that's already what i declare though:

    loader = {
      # NixOS wants to enable GRUB by default
      grub.enable = false;
      raspberryPi = {
        enable = true;
        version = 4;
        # not working yet https://github.com/NixOS/nixpkgs/pull/70796#issuecomment-552238845
        uboot = false;
        # possibility to add some config.txt stanza
        # firmwareConfig = ''
      };
      # Enables the generation of /boot/extlinux/extlinux.conf
      generic-extlinux-compatible.enable = true;
    };

  # for the next point
  fileSystems = {
    "/" = {  # bootable as per other images
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
    "/boot/firmware" = {  # for ease of introspection, it should not really be needed... i think?
      device = "/dev/disk/by-label/FIRMWARE";
      fsType = "vfat";
    };
  };

Also, I think the bootloader assumes the firmware partition is mounted to /boot, while the sd image mounts it to /boot/firmware.

That's the part where i'm a bit fuzzy reading through the different image definitions (and tryouts). So thanks for pointing it out. I'll have a look again there ;)

[ScarletHg]

My main issue being that it always boots as if it was a fresh sd-card install (nixos-rebuild switch after that and back on my feet to my latest generation). I guess it's because the boot files are not updating during the generation switch (I am not sure whether i am at fault or not here).

I have run into this issue before with a new image! I resolved this by explicitly mounting the “firmware” partition as /boot in my configuration, and re-switching twice (I don't recall why twice but once was not enough for some reason in my specific case).

There might be more that I did but I recall the issue in my case being that due to the faulty mounts, Nix was writing bootloader information to a /boot folder on the main partition, while the Pi firmware was reading from the boot/firmware partition. The missed connection there between Nix and the Pi firmware caused the Pi to always boot as fresh no matter how many times I ran the rebuild.

[ardumont]

I resolved this by explicitly mounting the “firmware” partition as /boot in my configuration, and re-switching twice (I don't recall why twice but once was not enough for some reason in my specific case).

Maybe some kind of race condition between mounting the filesystem and writing the files there (during the switch)...

There might be more that I did but I recall the issue in my case being that due to the faulty mounts, Nix was writing bootloader information to a /boot folder on the main partition, while the Pi firmware was reading from the boot/firmware partition. The missed connection there between Nix and the Pi firmware caused the Pi to always boot as fresh no matter how many times I ran the rebuild.

Yes, that's something i saw as well. Thus the difficulty troubleshooting. I had both a /boot filled in in the NIXOS_SD:/boot/{nixos,extlinux,...} disk and another in FIRMWARE:/boot (the image creation routine creates that partition with that label).

Reading back the documentation nixos-on-arm [1], the partition for /boot is named NIXOS_BOOT. So i renamed the FIRMWARE labelled one to that just in case. I also removed the NIXOS_SD:/boot one (manually and declaratively as well).

I'm not sure that solved my issue yet though (cmdline.txt did not seem to have moved during switch). In any case, thanks for the feedback ;)

[1] https://nixos.wiki/wiki/NixOS_on_ARM#NixOS_installation_.26_configuration

[diamondburned]

Has anyone tried building with SSH enabled out of the box? I'm getting a Missing privilege separation directory: /var/empty on startup, even though SSH starts fine after logging in.

[diamond@nixos:/etc/nixos]$ systemctl --user status nixos-activation.service
● nixos-activation.service - Run user-specific NixOS activation
   Loaded: loaded (/nix/store/jfkpsxaidqm74y4708iydiqbgycxahd1-unit-nixos-activation.service/nixos-activation.service; static; vendor preset: enabled)
   Active: inactive (dead)

Edit1: It seems like this is very much just a one-timer after flashing the image. You can manually create this directory and it would just work.

Question: How do I add commands to run after image creation?

[ardumont]

Question: How do I add commands to run after image creation?

maybe you you could create a nixos sd-image based on the existing one [1] which enables openssh service by default... (assuming you plug the pi to some passwordless network, ethernet or something for it to be useful ;)

[1] https://nixos.wiki/wiki/NixOS_on_ARM#Build_your_own_image

[diamondburned]

Question: How do I add commands to run after image creation?

maybe you you could create a nixos sd-image based on the existing one [1] which enables openssh service by default... (assuming you plug the pi to some passwordless network, ethernet or something for it to be useful ;)

[1] https://nixos.wiki/wiki/NixOS_on_ARM#Build_your_own_image

nixpkgs/nixos/modules/installer/cd-dvd/sd-image-aarch64.nix imports ../../profiles/installation-device.nix which actually disables SSH by default:

{
    systemd.services.sshd.wantedBy = mkOverride 50 [];
}

Edit1: My current imports:, will update later

{
    imports = [
        <nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix>
        <nixpkgs/nixos/modules/profiles/clone-config.nix>
        <nixpkgs/nixos/modules/profiles/headless.nix>
        <nixpkgs/nixos/modules/installer/cd-dvd/channel.nix>
        <nixpkgs/nixos/modules/installer/scan/detected.nix>
        <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
    ];
}

[diamondburned]

Tried with the <nixpkgs/nixos/modules/installer/cd-dvd/sd-image-raspberrypi4.nix> approach and nixos-rebuild switched. The end results are a bit... weird.

SSH is still not autostarted, nixos user is still there (even though none of my new generated configs import anything other than hardware-configuration.nix), but all of the things in the new config are preserved.

Edit1: made a new issue https://github.com/NixOS/nixpkgs/issues/73487

[cmacrae]

Would someone mind sharing the configuration.nix they're using with their RPi4? :pray:
I'm having issues with my config switches not being persistent across boots (I'm chucked back to the boot configuration of the installer). Guessing I'm messing something up with my boot config.

[cmacrae]

I'm mounting the labelled FIRMWARE partition as /boot, but I've just seen that someone mentioned doing a switch twice! I'll try that when I get home tomorrow and report back :)

[diamondburned]

Mine's in https://gitlab.com/diamondburned/nix-rpi4

[sbourdeauducq]

https://git.m-labs.hk/M-Labs/nix-scripts/src/branch/master/nixops/rpi.nix

[cmacrae]

Great! Thank you!

[cmacrae]

Alright, I think it's because I was being silly and using boot.loader.generic-extlinux-compatible.enable = true;. Without that, stuff works! Thanks for the example configs.

Now, onto another issue I've run into: we're all mounting the FIRMWARE partition as /boot... turns out this quickly fills up when trying to do NixOps deployments of any significance to the Pi.
So, I was just going to hack around it by setting sdImage.bootSize = 2048; in my image builder expression. However, I get the following assertion:

Failed assertions:
- The option definition `sdImage.bootSize' in `path/to/my/sdImage.nix' no longer has any effect; please remove it.
The boot files for SD image have been moved to the main ext4 partition. The FAT partition now only holds the Raspberry Pi firmware files. Changing its size may not be required.

As per the message, I don't think we should be mounting the FIRMWARE partition as /boot 🤔

[samueldr]

Sorry @cmacrae, this message is a bit misleading as the raspberry pi image builder is piggy-backing on the infrastructure of the generic images builder.

In the non-generic raspberry pi 4 use case, you have to use the FAT partition for /boot.

You can increase its size using sdImage.firmwareSize.

The sd image generation infrastructure is extremely coupled with how the generic images are intended to be built.

[cmacrae]

Ah okay, thanks for the explanation @samueldr :+1:

[diamondburned]

@cmacrae Off-topic, but how did you get NixOps deployment working with the RPi4? I couldn't get it working at all, so I had to use a hacky rsync script.

[sbourdeauducq]

The link I posted is for a nixops deployment and it worked without anything in particular (except the patch for https://github.com/NixOS/nixops/pull/665 but that's not rpi-4 specific). What problem do you have?