Feedback: The bootloader is capable of booting NixOS 23.05 with mainline kernel 6.3.X+

[arbv]

I have been successfully running NixOS on my R5S board using the 1.0 release of the bootloader and the corresponding DTB in UEFI mode for a couple of weeks already (I have tested both GRUB and systemd-boot boot managers - both work just fine). I have /boot on an SD card used as EFI system partition (ESP), as having it on the eMMC did not work for some reason. The bootloader itself is installed on the eMMC, and the dtb file is also loaded from a FAT32 partition on eMMC (the partition is marked for boot, too). The OS and most files are on an NVMe drive (SK Hynix P31 Gold, which I recommend for this board due to high power efficiency. Do use thermal pads anyway, though).

See here for my adventures regarding that.

I should note that the kernel from the NixOS repo does not support PCI, HDMI, and LEDs on this chip with its default configuration, so the kernel needs to be recompiled with some kernel options changed. It is easy to do on NixOS, but it takes a lot of time (12-14 hours) on the board itself - so it is wise to "pin" the kernel packages version and update it from time to time for now. Still, it runs the mainline kernel which is a big win for these boards.

Thus, the main takeaway snippet (I think I should try to upstream some of these kernel configuration options to avoid kernel recompilations):

{ config, pkgs, lib, ... }:

{
  config = lib.mkIf pkgs.stdenv.isAarch64 {
    boot = {
      # partial Rockchip related changes from Debian 12 kernel version 6.1
      # Also, see here:
      # https://discourse.nixos.org/t/how-to-provide-missing-headers-to-a-kernel-build/11422/3
      kernelPatches = [
        {
          name = "rockchip-config.patch";
          patch = null;
          extraConfig = ''
            PCIE_ROCKCHIP_EP y
            PCIE_ROCKCHIP_DW_HOST y
            ROCKCHIP_VOP2 y
          '';
        }
        {
          name = "status-leds.patch";
          patch = null;
          # old:
          # LEDS_TRIGGER_NETDEV y
          extraConfig = ''
            LED_TRIGGER_PHY y
            USB_LED_TRIG y
            LEDS_BRIGHTNESS_HW_CHANGED y
            LEDS_TRIGGER_MTD y
          '';
        }
      ];

      initrd.availableKernelModules = [
        ## Rockhip
        ## Storage
        "sdhci_of_dwcmshc"
        "dw_mmc_rockchip"

        "analogix_dp"
        "io-domain"
        "rockchip_saradc"
        "rockchip_thermal"
        "rockchipdrm"
        "rockchip-rga"
        "pcie_rockchip_host"
        "phy-rockchip-pcie"
        "phy_rockchip_snps_pcie3"
        "phy_rockchip_naneng_combphy"
        "phy_rockchip_inno_usb2"
        "dwmac_rk"
        "dw_wdt"
        "dw_hdmi"
        "dw_hdmi_cec"
        "dw_hdmi_i2s_audio"
        "dw_mipi_dsi"
      ];
    };
    # Most Rockchip CPUs (especcially with hybrid cores) work best with "schedutil"
    powerManagement.cpuFreqGovernor = "schedutil";
  };
}

Bootstrapping the installation required building a custom UEFI install image for me using nixos-generators project (it is easier to do that when u-boot and the DTB file are already on the board's eMMC). You will need the snippet above to build a usable installation image, too.

Another takeaway is to run irqbalance --oneshot on startup to spread interrupts processing between cores, otherwise it is too easy to overwhelm the first core with interrupts, especially on high network interfaces activity (use irqtop for monitoring).

On NixOS this can be achieved using this snippet:

{ config, pkgs, lib, ... }:

{
  config = {
    systemd.services."irqbalance-oneshot" = {
      enable = true;
      description = "Distribute interrupts after boot using \"irqbalance --oneshot\"";
      documentation = [ "man:irqbalance" ];
      wantedBy = [ "sysinit.target" ];
      serviceConfig = {
        Type = "oneshot";
        RemainAfterExit = true;
        ExecStart = "${pkgs.irqbalance.out}/bin/irqbalance --foreground --oneshot";
      };
    };
  };
}

Additionally to the above, I have the following in my configuration (one may, or may not need that depending on the needs):

{ config, pkgs, lib, ... }:

## NanoPi R5S-board specific configuration

# This board can be used with NixOS thanks only to this project:
# https://github.com/inindev/nanopi-r5/
# A lot of information is taken from there

{
  config = lib.mkIf pkgs.stdenv.isAarch64 {
    boot.kernelParams = [
      "console=tty1"
      "console=ttyS2,1500000"
      "earlycon=uart8250,mmio32,0xfe660000"
    ];
    # Let's blacklist the Rockchips RTC module so that the
    # battery-powered HYM8563 (rtc_hym8563 kernel module) will be used
    # by default
    boot.blacklistedKernelModules = [ "rtc_rk808" ];
  };
}

I have been running the device as LXC/libvirt/Docker host for weeks now without any issues. Thank you @inindev for your hard work! You made it possible!

[arbv]

That might be of some interest for @inindev: I am attaching dmesg log shortly after booting NixOS.

dmesg.txt

And a short excerpt which shows that it booted in UEFI mode with device tree passed to the kernel from the bootloader:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[    0.000000] Linux version 6.3.5 (nixbld@localhost) (gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.40) #1-NixOS SMP Tue May 30 13:17:29 UTC 2023
[    0.000000] Machine model: FriendlyElec NanoPi R5S
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000fe660000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] efi: EFI v2.10 by Das U-Boot
[    0.000000] efi: RTPROP=0xecee1040 SMBIOS=0xecee0000 INITRD=0xeaef2040 MEMRESERVE=0xeaef1040 
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000200000-0x00000000efffffff]
[    0.000000] NUMA: NODE_DATA [mem 0xef7d98c0-0xef7dcfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000200000-0x00000000efffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000200000-0x00000000ecedffff]
[    0.000000]   node   0: [mem 0x00000000ecee0000-0x00000000ecee1fff]
[    0.000000]   node   0: [mem 0x00000000ecee2000-0x00000000ecee2fff]
[    0.000000]   node   0: [mem 0x00000000ecee3000-0x00000000ecee4fff]
[    0.000000]   node   0: [mem 0x00000000ecee5000-0x00000000ecee5fff]
[    0.000000]   node   0: [mem 0x00000000ecee6000-0x00000000ecee9fff]
[    0.000000]   node   0: [mem 0x00000000eceea000-0x00000000eff1ffff]
[    0.000000]   node   0: [mem 0x00000000eff20000-0x00000000eff2ffff]
[    0.000000]   node   0: [mem 0x00000000eff30000-0x00000000efffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000200000-0x00000000efffffff]
[    0.000000] On node 0, zone DMA: 512 pages in unavailable ranges
[    0.000000] cma: Reserved 32 MiB at 0x00000000ed600000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
...

[inindev]

I have /boot on an SD card used as EFI system partition (ESP), as having it on the eMMC did not work for some reason.

Is this a path issue? When I have a boot device, the binaries are in the root / rather than /boot then this gets mounted to /boot by Linux. I have to edit the boot.txt file to move the paths then recompile it to the boot.scr file. It it hard to know, but if it is related, there is more detail at step 10 here: https://github.com/inindev/nanopi-r5#booting-from-m2-nvme

it takes a lot of time (12-14 hours) on the board itself

The rk3568 is slow on a full kernel. The rk3588 (I use the rock-5b) does the same compile in 40 minutes.

[arbv]

@inindev I am not using boot.scr at all. u-boot in distroboot mode loads the dtb file from the internal eMMC itself when looking for files to boot, but for some reason, it cannot boot the system in UEFI mode when /boot is on eMMC. I do not know why, but I mentioned it before. For now, I use SD card for /boot with eMMC containing u-boot and the dtb (basically, it is my EFI implementation for now).

The generic idea is very similar to the one in the link you posted.

[arbv]

@inindev A bit unrelated question: how is your experience with RK3588? Does it have good enough mainline kernel support to be usable?

[inindev]

@inindev A bit unrelated question: how is your experience with RK3588? Does it have good enough mainline kernel support to be usable?

I run mainline (see my repo: https://github.com/inindev/rock-5b) which is basically identical to collabra's: https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux

It is getting there but still rough. I have to use a USB SSD as the pcie nvme is not available yet. The command line is stable and great for fast native compiles.

[arbv]

@inindev Thanks! I think I will not jump into buying the board yet, but it indeed looks very interesting and promising.