Board request: NanoPc CM3588 NAS

[NomisIV]

What would be required to add support for the FriendlyElec CM3588 NAS kit board? I have very little experience in this area, but I am motivated to get this implemented. I do own this board myself, so I would be able to perform testing on it.

What I've figured out myself is that the device tree (dts) file needs to be included here. The proper device tree for this board is probably this one.

Are there any more files that need to be included for proper support?

Looking forward for input!

[jjpe]

I too would like to know how support for the CM3588 would be added.

[mariobalanica]

You can use one of the existing platforms as a template. They're all fairly similar, with notable differences being power enable GPIOs (e.g. for USB) and I2C peripheral locations. These details can be found in the device tree and board schematics.

Board-specific code resides in the .DSC file and RockchipPlatformLib.

The DTB should indeed be imported from Armbian's rk 6.1 branch. Right now it's only used for booting Linux, UEFI itself does not care about it.

By the way, the PCIe code does not currently support bifurcation (which the CM3588 appears to be using), so things like NVME drives won't be usable in UEFI.

[jjpe]

Thank you for the response.

By the way, the PCIe code does not currently support bifurcation (which the CM3588 appears to be using), so things like NVME drives won't be usable in UEFI.

What exactly would be needed for PCIe bifurcation support? I ask because my primary use case for the board is as a NAS, with... you guessed it, NVMe drives.

EDIT: OTOH, if it's only in UEFI where bifurcation doesn't work then perhaps it's not that much of a problem for the use case of a NAS?

[mariobalanica]

Bifurcation missing in UEFI means you can't boot from NVME. Maybe one of the slots would work at x1 link depending on how things are wired, but I don't know for sure.

It shouldn't be difficult to implement. It mostly needs changes to the PCIe 3 PHY driver and some nice user-configurable settings for lane mapping. I haven't had the hardware to test this and hence no interest really, but I'm going to accept pull requests if anyone's interested in taking on this task.

If your use-case is booting Linux with a device tree off SD/eMMC/USB, then you probably don't care about any of the above.

[navarro967]

I have begun simple work to get it ported to the device here.

The fork/branch produces a bootable build of EDK2. This device is almost identical to the T6 which I used as the base. I don't particularly require NVMe boot however the first NVMe device did show as a boot option.

I'm relatively new to the ARM scene and not very adept at reading device schematics however if I can reverse engineer the values from the T6 to understand how this is mapped I should be able to clean this up and submit a PR. @mariobalanica Maybe you can give me a simple example of how this mapping done?

Otherwise, if anyone want's to take a crack at it you can build from my fork

[NomisIV]

I tried your fork @navarro967, but my keyboard wouldn't get detected. If I was lucky I could get into the boot options by pressing Escape, but once I got in I couldn't navigate anything.

Were you able to get this working on your board, or are you experiencing the same issue?

[jjpe]

I just got my hardware. I'm not familiar with the intricate parts of bootloading etc, so what can I do to help, short of bricking the device?

[Rippanda12]

short of bricking the device?

@jjpe These devices aren't "brickable" since you have access to flashing modes etc which aren't really present on phones so don't worry about something happening to your device

[jjpe]

These devices aren't "brickable"

Not even if a wrong image is flashed, which at least in theory, could cause electrical damage by driving e.g. some I/O ports incorrectly? If that's true, then that is very useful to know indeed.

[Rippanda12]

These devices aren't "brickable"

Not even if a wrong image is flashed, which at least in theory, could cause electrical damage by driving e.g. some I/O ports incorrectly? If that's true, then that is very useful to know indeed.

Software wise they can't be bricked just like a regular pc, ofc physical damage can be done i.e plugging stuff where they don't belong but that rarely happens

[navarro967]

I tried your fork @navarro967, but my keyboard wouldn't get detected. If I was lucky I could get into the boot options by pressing Escape, but once I got in I couldn't navigate anything.

Were you able to get this working on your board, or are you experiencing the same issue?

I was able to, I plugged my Keychron K10 into a USB A port and was able to enter the BIOS and make changes.

[Jeidnx]

I have tried 5 different Keyboards, and the only one that worked was a Microsoft Sculpt (notably, the only one which uses a wireless dongle). Every one of them worked, once booted into the OS. I couldn't find any error messages, the only thing i noticed was that all keyboards get power when plugged in, but only for a second before turning off again.

[jneem]

Thanks for kicking this off @navarro967! I rebased your fork on the latest master to bring in the PCIe bifurcation support, and I can confirm that the firmware loads and sees my NVMe drive.

I haven't actually managed to boot into an OS yet. I have a working Ubuntu installation on the eMMC -- I know it's working because if I remove the sd card with edk2 then it boots -- but selecting it in the boot configuration menu just flashes the screen black and brings me back to edk2. If anyone has suggestions about how to debug it, I'd be grateful. Otherwise, I'm going to poke around with GPIO settings.

[serpent213]

Got my NAS board today and tried @navarro967's fork, flashed to eMMC via the FriendlyWrt web flasher. Console is serial, connected to the three pin header.

Looks pretty good, menu works fine, I switched from ACPI to DT and it boots into Grub from USB or microSD. I used a NixOS ISO (nixos-minimal-24.05.4469.6e99f2a27d60-aarch64-linux.iso).

After selecting the serial console in Grub and progressing to the kernel it gets stuck, though:

EFI stub: Booting Linux Kernel...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...

Also tried an ISO from another distro with very similar results.

Did not get any output on HDMI, but I didn't try very hard.

Any idea on how to move on?

[serpent213]

I built the NixOS image from @ryan4yin: https://github.com/ryan4yin/nixos-rk3588/tree/main, the rock5a variant, needed a small patch:

diff --git a/flake.nix b/flake.nix
index 5102bbc..ecffe1a 100644
--- a/flake.nix
+++ b/flake.nix
@@ -31,7 +31,7 @@
     ...
   }: let
     # Local system's architecture, the host you are running this flake on.
-    localSystem = "x86_64-linux";
+    localSystem = "aarch64-linux";
     pkgsLocal = import nixpkgs {system = localSystem;};
     # The native system of the target SBC.
     aarch64System = "aarch64-linux";
diff --git a/pkgs/mali-firmware/default.nix b/pkgs/mali-firmware/default.nix
index f2bb1c0..bc48e64 100644
--- a/pkgs/mali-firmware/default.nix
+++ b/pkgs/mali-firmware/default.nix
@@ -7,7 +7,7 @@ stdenv.mkDerivation {
   version = "g21p0-01eac0";

   src = fetchurl {
-    url = "https://github.com/JeffyCN/mirrors/raw/e08ced3e0235b25a7ba2a3aeefd0e2fcbd434b68/firmware/g610/mali_csffw.bin";
+    url = "https://github.com/JeffyCN/mirrors/raw/libmali/firmware/g610/mali_csffw.bin";
     hash = "sha256-jnyCGlXKHDRcx59hJDYW3SX8NbgfCQlG8wKIbWdxLfU=";
   };

Copied to SD; throws errors due to hardware mismatch but boots fully with network working (on serial console, did not try HDMI).

[NomisIV]

I built the NixOS image from @ryan4yin: https://github.com/ryan4yin/nixos-rk3588/tree/main, the rock5a variant, needed a small patch:

diff --git a/flake.nix b/flake.nix
index 5102bbc..ecffe1a 100644
--- a/flake.nix
+++ b/flake.nix
@@ -31,7 +31,7 @@
     ...
   }: let
     # Local system's architecture, the host you are running this flake on.
-    localSystem = "x86_64-linux";
+    localSystem = "aarch64-linux";
     pkgsLocal = import nixpkgs {system = localSystem;};
     # The native system of the target SBC.
     aarch64System = "aarch64-linux";
diff --git a/pkgs/mali-firmware/default.nix b/pkgs/mali-firmware/default.nix
index f2bb1c0..bc48e64 100644
--- a/pkgs/mali-firmware/default.nix
+++ b/pkgs/mali-firmware/default.nix
@@ -7,7 +7,7 @@ stdenv.mkDerivation {
   version = "g21p0-01eac0";

   src = fetchurl {
-    url = "https://github.com/JeffyCN/mirrors/raw/e08ced3e0235b25a7ba2a3aeefd0e2fcbd434b68/firmware/g610/mali_csffw.bin";
+    url = "https://github.com/JeffyCN/mirrors/raw/libmali/firmware/g610/mali_csffw.bin";
     hash = "sha256-jnyCGlXKHDRcx59hJDYW3SX8NbgfCQlG8wKIbWdxLfU=";
   };

Copied to SD; throws errors due to hardware mismatch but boots fully with network working (on serial console, did not try HDMI).

Could you elaborate on the steps you took to get this working? I am unable to reproduce. The rawEfiImage fails to build, and the sdImage seems to have a broken partition table (I cannot mount the second partition, and the first one is empty)

[jneem]

My local system isn't an aarch64-linux, so I only changed the mali firmware url. (But it doesn't look like it matters; localSystem is only used in the cross-compiling targets.) I'm on an x86-64 system with boot.binfmt.emulatedSystems = [ "aarch64-linux" ] in my nixos configuration (otherwise it will probably fail with an error about needing to build on an aarch64 system).

Then I run nix build .#rawEfiImage-rock5a, dd the result to an sdcard, and plugging it into my cm3588 with edk2 already installed just works.

(I've been struggling with updating to a more recent kernel, but that's another story...)

[zatevakhin]

@jneem What kernel versions worked for you?

[jneem]

• the 6.1.43 kernel from orangepi (the one that's currently used in nixos-rk3588) boots and has ethernet
• I tried updating to the armbian/linux-rockchip kernel version 6.1.75. It boots but doesn't have ethernet (the r8169 module is available, it just doesn't get loaded)
• I tried updating to mainline 6.10.9 with config from armbian, but it doesn't boot at all. I get nonsense like this on the serial console:

EFI stub: Booting Linux Kernel...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
I nCiagI dPwi alt
                 I dPilg/SaUwnnld
/eaUinnld
I dC ag/SaPsi aro
                 I nCianISdPsioaro
                                  I dCiagISdPsioaro
                                                   I dCianISdPsioaro
                                                                    [    0.702728] rtc-hym8563 6-0051: hctosys: unable to read the hardware clock
[    0.705401] rockchip-thermal fec00000.tsadc: failed to register sensor 0: -19
[    0.706080] rockchip-thermal fec00000.tsadc: error -ENODEV: failed to register sensor[0].
ISaUwn ldt
          I nCilgI dPwi alt
/SaUwnnldt                 I dPil
          I dPilgISaUwn ldt
                           I dCiag/SaPwi alt
                                            I dCiag/SdPwiomro
                                                             I nC anI dCshtmob
                                                                              I dPilgISaPwi llt
                                                                                               I dCilgISdPwi alt
                                                                                                                I dCilgISaPwi alt
                                                                                                                                 I dCilgISaPwi alo
                                                                                                                                                  I dCianI nC htmoo
                                                                                                                                                                   I nC inI n  c ow
I nC anISdPshtmo

I tried booting the second two options (armbian/linux-rockchip and mainline 6.10) using armbian (instead of nixos) to build an sdcard image. The bootable armbian sdcards skip edk2 and they boot with working ethernet.

[jneem]

Btw, I have an updated branch here. Compared to @navarro967 's branch, it's based on a more recent master and it comes with some updates to the pin definitions (but I didn't actually observe any differences caused by these changes)

[Jeidnx]

Thanks for your work @jneem! Did you use mainline with just the armbian config or did you also apply their kernel patches? I have tried to build the 6.10.{5,9} Kernels with the armbian patches and a modified version of their config (with compressed firmware loading enabled), but haven't been able to successfully build it yet. I am pretty sure it is an issue with my build process though. I have also tried the armbian 6.1.75 and the friendlyarm 6.1.57 kernels, both boot but no Ethernet. The chip doesn't show in lspci, i haven't done more digging yet. Curiously the friendlyarm kernel with almost the same config (i had to enable efi and dmiid) and the same dts has working ethernet when used in the vendor debian 12 image.

I have made a flake to test nixos on the cm3588 a while ago, but just now gotten around to cleaning it up. Here is a link for anyone that's interested.

[jneem]

I've been trying without the armbian patches so far, but maybe I should give them a try. Btw, 6.11 now has mainline support for the cm3588! I'm building that now.

Working ethernet is a priority for me, but I was hoping to get a new kernel booting before digging into the ethernet issue...

[jneem]

but haven't been able to successfully build it yet

If it's the same issue I had, it's because you need to set CONFIG_CRYPTO_AEGIS128_SIMD=n, because the assembler nix is using to build the kernel thinks that the rk3588 doesn't support some instructions. I don't know how armbian gets away with this setting; maybe their assembler thinks differently...

[NomisIV]

I got 6.11 working (with ethernet, but no HDMI for some reason) on my cm3588. Here is the minimal config needed:

  boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux_latest;
  hardware = {
    enableRedistributableFirmware = true;
    deviceTree = {
      enable = true;
      name = "rockchip/rk3588-friendlyelec-cm3588-nas.dtb";
    };
  };

see the entire config here

[DualTachyon]

but haven't been able to successfully build it yet

If it's the same issue I had, it's because you need to set CONFIG_CRYPTO_AEGIS128_SIMD=n, because the assembler nix is using to build the kernel thinks that the rk3588 doesn't support some instructions. I don't know how armbian gets away with this setting; maybe their assembler thinks differently...

The Crypto instructions on RK3588 are disabled by default. Whether they are enabled in your environment depends on what build of BL31 is being used. See this patch for reference: https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/29363

[jneem]

boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux_latest;

Amazing! All that fiddling with armbian forks and custom configs, and it turns out that latest mainline just works! (For anyone following along, it's important here to be using nixos-unstable or else linux_latest is too old.) Now I have edk2 on my eMMC, booting into nixos on an nvme drive.

[samyak-jain]

I am trying to flash the firmware from this branch (https://github.com/jneem/edk2-rk3588/tree/cm3588), however, I am unable to enter the boot options when pressing escape at the boot splash screen. I've tried 2 different keyboards. It boots into openmediavault by default no matter what I do. Any suggestions?

[jneem]

It looks like only one USB port was being powered up (the upper USB 3.0 port). The latest commit should enable them all.

[jjpe]

For the NixOS users here, how did you cross compile jneem's EDK2 branch? I'd rather not have to muck about with Debian and the like.

[sanderhollaar]

@jneem Thanks a lot for your work! 🤗

For the NixOS users here, how did you cross compile jneem's EDK2 branch? I'd rather not have to muck about with Debian and the like.

Until someone actually answers your question ..

I did not want to trial and error the required tools and went the Debian route (on x68_64):

Enable podman with virtualisation.podman.enable = true;

Create /tmp/mnt/: mkdir /tmp/mnt/ Start Debian: podman run --rm -it --network=host --volume=/tmp/mnt:/root debian:trixie /bin/bash

In Debian: apt update && apt upgrade -y apt install -y git gcc g++ build-essential gcc-aarch64-linux-gnu acpica-tools python3-pyelftools uuid-dev python-is-python3 device-tree-compiler cd ~ git clone --recursive https://github.com/jneem/edk2-rk3588.git cd edk2-rk3588 git switch cm3588 ./build.sh --device nanopc-cm3588-nas --release Release

Find RK3588_NOR_FLASH.img in /tmp/mnt/edk2-rk3588/

Have not tried the UEFI image yet.

I also have a NanoPi R6C running NixOS.

[jneem]

There were a few moving parts, and I haven't packaged it all up nicely.

1) You need to rebuild the mkimage binary because it isn't nix-friendly. I have a flake here for building it, and then you can copy it to misc/tools/<arch>/mkimage 2) I needed to apply this patch (attached) to the edk2/ submodule 3) I used the attached shell.nix 4) I had some trouble with OpenTFA that I didn't feel like debugging, so I build without it: bash build.sh --device nanopc-cm3588-nas --open-tfa 0 --release RELEASE

edk2-patch.txt shell.nix.txt

[jjpe]

First off, thank you for the instruction, they were most helpful.

EDIT: I got the firmware installed, and can get into the UEFI config menu. Where the story seems to end for now is the fact that when I try to build this nix file with nix-build '<nixpkgs/nixos>' -I nixos-config=./sd-image.nix:

{pkgs, ...}: {
  imports = [
    <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix>
  ];

  boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux_latest;

  hardware = {
    enableRedistributableFirmware = true;
    deviceTree = {
      enable = true;
      name = "rockchip/rk3588-friendlyelec-cm3588-nas.dtb";
    };
  };
}

I get an error complaining that zfs is broken:

trace: evaluation warning: system.stateVersion is not set, defaulting to 24.11. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'nixos-system-nixos-24.11pre-git'
         whose name attribute is located at /nix/store/y6205wq8hxvpqvl8l9d1n9xah01kg0lq-source/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'buildCommand' of derivation 'nixos-system-nixos-24.11pre-git'

         at /nix/store/y6205wq8hxvpqvl8l9d1n9xah01kg0lq-source/nixos/modules/system/activation/top-level.nix:53:5:

           52|     passAsFile = [ "extraDependencies" ];
           53|     buildCommand = systemBuilder;
             |     ^
           54|

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: Package ‘zfs-kernel-2.2.6-6.11’ in /nix/store/y6205wq8hxvpqvl8l9d1n9xah01kg0lq-source/pkgs/os-specific/linux/zfs/generic.nix:216 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

          Note: When using `nix shell`, `nix build`, `nix develop`, etc with a flake,
                then pass `--impure` in order to allow use of environment variables.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.

So unless someone has an idea I guess I'll have to wait until the zfs package is no longer broken.

[Jeidnx]

@jjpe OpenZFS doesn't support the 6.11 kernel yet, that's why the package is marked as broken. I am still trying to get the 6.10 Kernel to work, but no luck so far.

[jneem]

@samyak-jain did that patch end up getting your USB to work? If so, I might go ahead and submit this as a PR, because I think the OpenZFS issue is separate.

[picnoir]

@jneem I've been using your branch without any major issue, the usb keyboard works as expected, the NVME drives are detected. I think it's at least ready to get PR-d.

I have one remaining issue wrt. thermals. lm_sensors does not report back any temperature data for the SOC on a 6.11 live NixOS installer. All the required sensor device trees should be there, they've been merged part of 6.10. Not sure if it's related to the edk2 boot though.

[samyak-jain]

Hey @jneem. I don't have an answer on what the root cause of my problem is so I can't say for sure. Unfortunately, the patch did not fix my issue. I've tried each usb port with all my keyboards after fetching the latest version of the branch. I am taken to openmediavault every single time and my keyboard is working then so it's unclear to me if this is a usb issue or if the problem lies elsewhere. I've also noticed that some symbols like "^]" will appear on the top when I spam escape at the start so it looks like something might be registering even if I'm not taken to boot options.

[jjpe]

@jjpe OpenZFS doesn't support the 6.11 kernel yet, that's why the package is marked as broken. I am still trying to get the 6.10 Kernel to work, but no luck so far.

@Jeidnx Yeah I figured as much. It makes me feel a bit better about not prioritizing my NAS build a bit more highly though, since I would have hit this (temporary) roadblock anyway.

@jneem I should mention that while I got EDK2 installed, and could boot into the EFI settings menu, the AArch64 nixos-unstable image I dd'd onto an SD card got as far as displaying the image's boot menu, but none of the available options booted successfully.

Instead I got a black screen with this text on it:

EFI stub: Booting Linux Kernel...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Generating empty DTB
EFI stub: Exiting boot services...

Now granted, that was just a very recent image I downloaded from the servers, and not any of the forks people here have been working on. I'll try one of those next.

@picnoir which image did you use? Apparently you got further along in the boot process than I have so far.

@samyak-jain I originally got taken to OMV too. I had to follow the unbricking instructions here to get rid of that. Essentially it flashes a minimal loader to the device, putting it into a "factory mode" of sorts. After that (and only then), (this fork of) EDK2 can be installed.

[picnoir]

@jjpe I used the iso_minimal_new_kernel_no_zfs.aarch64-linux image (to get linux 6.11, which contain important fixes for the rk3588 power management) flashed on a usb drive.

You can download the latest one there: https://hydra.nixos.org/build/273957457