Closed jakubgs closed 4 months ago
I tried booting a normal NixOS aarch64
image but got no output on the UAR console:
When i boot the Armbian image I do get output from the UART console though:
Terminal ready
DDR Version V1.08 20220617
LPDDR4X, 2112MHz
channel[0] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
channel[1] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
channel[2] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
channel[3] BW=16 Col=10 Bk=8 CS0 Row=17 CS1 Row=17 CS=2 Die BW=16 Size=4096MB
Manufacturer ID:0xff
CH0 RX Vref:30.7%, TX Vref:21.8%,21.8%
CH1 RX Vref:30.7%, TX Vref:21.8%,21.8%
CH2 RX Vref:27.7%, TX Vref:23.8%,23.8%
CH3 RX Vref:29.7%, TX Vref:21.8%,21.8%
change to F1: 528MHz
change to F2: 1068MHz
change to F3: 1560MHz
change to F0: 2112MHz
out
INFO: Preloader serial: 2
NOTICE: BL31: v2.3():v2.3-405-gb52c2eadd:derrick.huang
NOTICE: BL31: Built : 11:23:47, Aug 15 2022
INFO: spec: 0x1
INFO: ext 32k is not valid
INFO: GICv3 without legacy support detected.
INFO: ARM GICv3 driver initialized in EL3
INFO: system boots from cpu-hwid-0
INFO: idle_st=0x21fff, pd_st=0x11fff9, repair_st=0xfff70001
INFO: dfs DDR fsp_params[0].freq_mhz= 2112MHz
INFO: dfs DDR fsp_params[1].freq_mhz= 528MHz
INFO: dfs DDR fsp_params[2].freq_mhz= 1068MHz
INFO: dfs DDR fsp_params[3].freq_mhz= 1560MHz
INFO: BL31: Initialising Exception Handling Framework
INFO: BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR: Error initializing runtime service opteed_fast
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0x200000
INFO: SPSR = 0x3c9
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
done.
Begin: Will now check root file system ... fsck from util-linux 2.38.1
[/sbin/fsck.ext4 (1) -- /dev/mmcblk1p2] fsck.ext4 -a -C0 /dev/mmcblk1p2
armbi_root: clean, 44590/127488 files, 398514/509947 blocks
done.
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.
Welcome to Armbian 23.8.1 bookworm!
...
There is a work-in-progress U-Boot build using Tow-Boot that supports Rock 5B: https://github.com/samueldr/Tow-Boot/tree/wip/rock5-vendor/boards/radxa-rock5b
I did build it successfully:
~/soft/Tow-Boot wip/rock5-vendor
> nix-build -A radxa-rock5b.config.build.firmwareSPI
trace: Using default Nixpkgs revision '1905f5f2e55e0db0bb6244cfe62cb6c0dbda391d'...
trace: ***********************************
trace: * Evaluating device: radxa-rock5b *
trace: ***********************************
trace: Building with crossSystem?: aarch64-linux != x86_64-linux → we are.
trace: crossSystem: config: aarch64-unknown-linux-gnu
trace: Building with crossSystem?: aarch64-linux != x86_64-linux → we are.
trace: crossSystem: config: aarch64-unknown-linux-gnu
/nix/store/4v5mfwr31av176qcwf6i9q85id2iankz-tow-boot-rock-5b-rk3588_defconfig-spi-aarch64-unknown-linux-gnu-2017.09-005-pre
~/soft/Tow-Boot wip/rock5-vendor
> ls -l /nix/store/4v5mfwr31av176qcwf6i9q85id2iankz-tow-boot-rock-5b-rk3588_defconfig-spi-aarch64-unknown-linux-gnu-2017.09-005-pre/binaries/
total 1796
-r--r--r-- 1 root root 1866752 Jan 1 1970 Tow-Boot.spi.bin
But flashing it is another thing.
Apparently the correct way to flash this is using rkdeveloptool
tool as described here:
https://wiki.radxa.com/Rock5/install/spi#Advanced_.28external.29_method
And Nix does provide the tool:
> nix-shell -p rkdeveloptool
[nix-shell:~/soft/Tow-Boot]$ which rkdeveloptool
/nix/store/0i7wdad3f1imbvv1k3qk0047wf8qyhhq-rkdeveloptool-unstable-2021-02-03/bin/rkdeveloptool
[nix-shell:~/soft/Tow-Boot]$ rkdeveloptool --version
rkdeveloptool ver 1.3
By holding the MASKROM button and connecting the USB-C to the PC I can detect it:
[nix-shell:~/rk3588]$ sudo rkdeveloptool ld
DevNo=1 Vid=0x2207,Pid=0x350b,LocationID=903 Maskrom
But when I try to follow the rest of instructions to load rk3588_spl_loader_v1.08.111.bin
:
[nix-shell:~/rk3588]$ sudo rkdeveloptool db ./rk3588_spl_loader_v1.08.111.bin
Opening loader failed, exiting download boot!
It just fails.
This is the part that fails:
pImage = new CRKImage(szLoader, bRet);
if (!bRet){
ERROR_COLOR_ATTR;
printf("Opening loader failed, exiting download boot!");
I guess there's some issue with the rk3588_spl_loader_v1.08.111.bin
image itself, since the reading of the file works:
583584 newfstatat(AT_FDCWD, "./rk3588_spl_loader_v1.08.111.bin", {st_mode=S_IFREG|0644, st_size=448960, ...}, 0) = 0
583584 openat(AT_FDCWD, "./rk3588_spl_loader_v1.08.111.bin", O_RDONLY) = 7
583584 mmap(NULL, 450560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1c763ba000
583584 newfstatat(7, "", {st_mode=S_IFREG|0644, st_size=448960, ...}, AT_EMPTY_PATH) = 0
583584 lseek(7, 0, SEEK_SET) = 0
583584 read(7, "LDR f\0\v\1\0\0\0\0\0\1\346\7\10\36\0203\r8853\2f\0\0\09\1"..., 442368) = 442368
583584 read(7, "\265\360\342\301P\306$\236\22\367X\216@\271\267\276\213\372%\335t\327oYF~\23A\356\375\365\221"..., 8192) = 6592
583584 newfstatat(1, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x8), ...}, AT_EMPTY_PATH) = 0
583584 write(1, "\33[30;41mOpening loader failed, e"..., 59) = 59
The code doesn't have any indication which check actually fails:
char szName[256];
strcpy(szName, filename.c_str());
if(stat(szName, &statBuf) < 0) {
bCheck = false;
return;
}
if (S_ISDIR(statBuf.st_mode)) {
bCheck = false;
return;
}
m_fileSize = statBuf.st_size;
bool bOnlyBootFile=false;
transform(filename.begin(), filename.end(), filename.begin(), (int(*)(int))tolower);
if (filename.find(".bin") != string::npos) {
bOnlyBootFile = true;
}
m_pFile = fopen(szName, "rb");
if (!m_pFile) {
bCheck = false;
return;
}
But I guess I could patch this code and figure that out.
The folder that contains rk3588_spl_loader_v1.08.111.bin
doesn't have any checksums, so I can't verify if my image is correct:
https://dl.radxa.com/rock5/sw/images/loader/rock-5b/release/
These links seem related:
But don't really help.
Building newer rkdeveloptool
from master
branch of nixpkgs
does fix it:
[nix-shell:~/rk3588]$ rkdeveloptool --version
rkdeveloptool ver 1.32
[nix-shell:~/rk3588]$ sudo rkdeveloptool db rk3588_spl_loader_v1.08.111.bin
Downloading bootloader succeeded.
And now flashing LBA works:
[nix-shell:~/rk3588]$ sudo rkdeveloptool wl 0 Tow-Boot.spi.bin
Write LBA from file (100%)
The Tow-Boot.spi.bin
from samueldr/Tow-Boot didn't really work. But then I flashed the rock-5b_UEFI_Release_v0.9.1.img
from:
https://github.com/edk2-porting/edk2-rk3588/releases/tag/v0.9.1
[nix-shell:~/rk3588]$ sudo rkdeveloptool wl 0 rock-5b_UEFI_Release_v0.9.1.img
Write LBA from file (100%)
And now the UART console gives me this:
Which is pretty darn cool.
It has a boot manager:
This is fucking amazing.
And there's a device manager too:
I'm in awe.
And after reconnecting the NVMe it shows up in the boot manager:
Why the fuck isn't this the default firmware?
And after adding the SD card to boot options it does boot Armbian witout issues:
Armbian 23.8.1 Bookworm ttyFIQ0
rock-5b login:
Except now we have UEFI boot, which in theory should work fine for NixOS too.
Armbian bootpartition doesn't seem to have an u-boot.bin
:
root@rock-5b:~# ls -l /mnt/plugged/
total 71072
-rw-r--r-- 1 1009 1009 209 Sep 1 00:50 armbianEnv.txt
-rw-r--r-- 1 root root 1536 Sep 1 2023 armbian_first_run.txt.template
-rw-r--r-- 1 root root 38518 Sep 1 2023 boot.bmp
-rw-r--r-- 1 1009 1009 3820 Sep 1 00:35 boot.cmd
-rw-rw-r-- 1 root root 3892 Sep 1 2023 boot.scr
-rw-r--r-- 1 root root 219655 Aug 30 02:17 config-5.10.160-legacy-rk35xx
lrwxrwxrwx 1 root root 26 Sep 1 00:53 dtb -> dtb-5.10.160-legacy-rk35xx
drwxr-xr-x 3 root root 4096 Sep 1 00:53 dtb-5.10.160-legacy-rk35xx
lrwxrwxrwx 1 root root 30 Sep 1 00:53 Image -> vmlinuz-5.10.160-legacy-rk35xx
-rw-r--r-- 1 root root 14784948 Sep 1 2023 initrd.img-5.10.160-legacy-rk35xx
drwx------ 2 root root 16384 Sep 1 2023 lost+found
-rw-r--r-- 1 root root 7979679 Aug 30 02:17 System.map-5.10.160-legacy-rk35xx
lrwxrwxrwx 1 root root 30 Sep 1 2023 uInitrd -> uInitrd-5.10.160-legacy-rk35xx
-rw-r--r-- 1 root root 14785012 Sep 1 2023 uInitrd-5.10.160-legacy-rk35xx
-rw-r--r-- 1 root root 34920960 Aug 30 02:17 vmlinuz-5.10.160-legacy-rk35xx
But it does have armbianEnv.txt
:
root@rock-5b:~# cat /mnt/plugged/armbianEnv.txt
verbosity=1
bootlogo=false
overlay_prefix=rockchip-rk3588
fdtfile=rockchip/rk3588-rock-5b.dtb
rootdev=UUID=5aad2a43-6260-4306-bb85-e0483bc08548
rootfstype=ext4
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u
I could try to install NixOS from the Armbian on my SD card, since that is doable:
I've installed Nix multi-user way via the script and then installed nixos-install-tools
package to get tools like nixos-generate-config
and nixos-install
, the partitioned the NVMe using parted
:
wipefs -a /dev/nvme0n1
parted -s --align optimal /dev/nvme0n1 -- mklabel msdos
parted -s --align optimal /dev/nvme0n1 -- mkpart primary 0% 5GiB set 2 esp on
parted -s --align optimal /dev/nvme0n1 -- mkpart primary 5GiB 20GiB
parted -s --align optimal /dev/nvme0n1 -- mkpart primary 20GiB 100%
mkswap /dev/nvme0n1p2
mkfs.vfat /dev/nvme0n1p1
mkfs.ext4 /dev/nvme0n1p3
Then mounted the volumes:
swapon /dev/nvme0n1p2
mount /dev/nvme0n1p3 /mnt
mkdir /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot
And generated a config, then modified it accordingly:
{ config, lib, pkgs, ... }:
{
imports = [ ./hardware-configuration.nix ];
# Use the GRUB 2 boot loader.
boot.loader = {
grub.enable = false;
efi.efiSysMountPoint = "/boot/efi";
efi.canTouchEfiVariables = true;
systemd-boot = {
enable = true;
configurationLimit = 10;
};
};
networking = {
hostName = "israfel";
hostId = "d22a9a6c";
useDHCP = true;
};
time.timeZone = "Europe/Warsaw";
i18n.defaultLocale = "en_US.UTF-8";
console = {
font = "Lat2-Terminus16";
keyMap = "pl";
};
environment.systemPackages = with pkgs; [ neovim wget htop ];
services.openssh.enable = true;
networking.firewall.allowedTCPPorts = [ 22 ];
system.copySystemConfiguration = true;
system.stateVersion = "23.11";
}
But of course it didn't work:
>>Start PXE over IPv4
PXE-E21: Remote boot cancelled.
>>Start PXE over IPv6.
PXE-E21: Remote boot cancelled.
>>Start HTTP Boot over IPv4.
Holy shit I got it working:
<<< Welcome to NixOS 23.11pre537863.038b2922be3f (aarch64) - ttyS0 >>>
Run 'nixos-help' for the NixOS manual.
israfel login:
And this isn't even using the most recent kernel, it's just 6.1
. Which means Nixos 23.05 might work fine. And ZFS as well.
Here's the partitioning I used:
format() {
parted -s --align optimal "$1" -- mklabel gpt
parted -s --align optimal "$1" -- mkpart 'BIOS' 1MB 2MB set 1 bios_grub on
parted -s --align optimal "$1" -- mkpart 'EFI' 2MB 5GiB set 2 esp on
parted -s --align optimal "$1" -- mkpart 'SWAP' 5GiB 20GiB set 2 esp on
parted -s --align optimal "$1" -- mkpart 'ROOT' 20GiB '100%'
parted -s --align optimal "$1" -- print
}
And the EFI
partition is vfat
,while ROOT
is ext4
for now. And the boot config is:
boot.loader = {
efi = {
efiSysMountPoint = "/boot";
canTouchEfiVariables = true;
};
systemd-boot = {
enable = true;
configurationLimit = 10;
};
grub.enable = false;
};
Added first working config and added docs:
nixos-config#12dcf0f4
- hosts/israfel: create first working configurationnixos-config#0113a913
- hosts/israfel: add basic readmenixos-config#3e6e905e
- hosts/israfel: add installation instructionsnixos-config#b8a964bf
- hosts/israfel: add steps to install nixos-install-toolsnixos-config#059e289f
- hosts/israfel: add bootloader screenshotBut the final setup will use ZFS.
I got Armbian_23.02.0-trunk.0252_Rock-5b_lunar_edge_6.2.0-rc2_minimal.img
from armbian/build
repo releases:
And it boots:
Welcome to Armbian 23.02.0-trunk.0251 Lunar!
But it doesn't see the ethernet interface:
root@rock-5b:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
Nor does it see the NVMe:
root@rock-5b:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mmcblk2 179:0 0 58.3G 0 disk
├─mmcblk2p1 179:1 0 256M 0 part /boot
└─mmcblk2p2 179:2 0 57.4G 0 part /var/log.hdd
/
zram0 252:0 0 7.8G 0 disk [SWAP]
zram1 252:1 0 50M 0 disk /var/log
zram2 252:2 0 0B 0 disk
I tried Armbian_23.02.0-trunk.0254_Uefi-arm64_jammy_edge_6.1.11_minimal.img.xz
which does show GRUB at boot time:
But when the first option is selected I get:
Scanning for Btrfs filesystems
done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=d43d82c7-6153-495f-a001-05f9d068cfdb does not exist. Dropping to a shell!
So that's not great.
Tried Armbian_23.02.0-trunk.0254_Uefi-arm64_jammy_current_5.15.93_xfce_desktop.img.xz
and the result:
Loading Linux 5.15.93-arm64 ...
Loading initial ramdisk ...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
I/TC: Secondary CPU 2 initializing
I/TC: Secondary CPU 2 switching to normal world boot
I/TC: Secondary CPU 3 initializing
I/TC: Secondary CPU 3 switching to normal world boot
I/TC: Secondary CPU 4 initializing
I/TC: Secondary CPU 4 switching to normal world boot
I/TC: Secondary CPU 5 initializing
I/TC: Secondary CPU 5 switching to normal world boot
I/TC: Secondary CPU 6 initializing
I/TC: Secondary CPU 6 switching to normal world boot
I/TC: Secondary CPU 7 initializing
I/TC: Secondary CPU 7 switching to normal world boot
And nothing more.
I tried rock-5b_debian_bullseye_cli_b39.img.xz
from official repo: https://github.com/radxa-build/rock-5b/releases
It boots(login is rock
/rock
) and sees the NVMe just fine:
rock@rock-5b:~$ lspci
0000:00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0000:01:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics Co., Ltd. Device 1602 (rev 01)
0004:40:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
But like with otherrs I can't get ZFS installed on the rock-5b
Debian SD card image:
rock@rock-5b:~$ sudo apt install linux-headers-5.10.110-15-rockchip
...
Setting up zfs-dkms (2.0.3-9+deb11u1) ...
Loading new zfs-2.0.3 DKMS files...
Building for 5.10.110-15-rockchip
Building initial module for 5.10.110-15-rockchip
configure: error:
*** None of the expected "PDE_DATA" interfaces were detected.
*** This may be because your kernel version is newer than what is
*** supported, or you are using a patched custom kernel with
*** incompatible modifications.
***
*** ZFS Version: zfs-2.0.3-9+deb11u1
*** Compatible Kernels: 3.10 - 5.10
Error! Bad return status for module build on kernel: 5.10.110-15-rockchip (aarch64)
Consult /var/lib/dkms/zfs/2.0.3/build/make.log for more information.
dpkg: error processing package zfs-dkms (--configure):
Opened an issue about it:
Created a post on NixOS Discourse to ask about proper way to make an UEFI bootable SD card image for aarch64
:
Building a raw-efi
using nixos-generators
did get the booting process going, but it doesn't finish:
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Generating empty DTB
EFI stub: Exiting boot services...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
I/TC: Secondary CPU 2 initializing
I/TC: Secondary CPU 2 switching to normal world boot
I/TC: Secondary CPU 3 initializing
I/TC: Secondary CPU 3 switching to normal world boot
I/TC: Secondary CPU 4 initializing
I/TC: Secondary CPU 4 switching to normal world boot
I/TC: Secondary CPU 5 initializing
I/TC: Secondary CPU 5 switching to normal world boot
I/TC: Secondary CPU 6 initializing
I/TC: Secondary CPU 6 switching to normal world boot
I/TC: Secondary CPU 7 initializing
I/TC: Secondary CPU 7 switching to normal world boot
Probably because I was using 6.1
kernel.
I managed to boot an stock NixOS ISO image written onto a USB pendrive instead of an SD card:
I have no idea why the SD card refuses to boot with the same image, but that's fine.
And it can see the NVMe just fine, not eMMC tho:
[nixos@nixos:~]$ sudo fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: SAMSUNG MZVLB512HAJQ-000L7
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 247F9A1E-4131-434C-A523-5DA93926BABD
Device Start End Sectors Size Type
/dev/nvme0n1p1 4096 12582911 12578816 6G EFI System
/dev/nvme0n1p2 12582912 33554431 20971520 10G Linux filesystem
/dev/nvme0n1p3 33554432 1000214527 966660096 460.9G Linux filesystem
Unfortunately the new kernel builds don't have a ZFS option:
And there we have it. NixOS 23.05.4466.b3ddf9649fda
image booted from USB pendrive via edk2-rk3588
UEFI firmware:
[nixos@nixos:~]$ uname -a
Linux nixos 6.1.59 #1-NixOS SMP Thu Oct 19 21:08:58 UTC 2023 aarch64 GNU/Linux
[nixos@nixos:~]$ sudo fdisk -l
Disk /dev/loop0: 771.47 MiB, 808947712 bytes, 1579976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/nvme0n1: 3.73 TiB, 4096805658624 bytes, 8001573552 sectors
Disk model: Lexar SSD NM790 4TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 28.91 GiB, 31037849600 bytes, 60620800 sectors
Disk model: USB Flash Disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Decent I/O results:
[nixos@nixos:~]$ sudo hdparm -tT /dev/nvme0n1
/dev/nvme0n1:
Timing cached reads: 2824 MB in 2.00 seconds = 1413.28 MB/sec
Timing buffered disk reads: 1604 MB in 3.00 seconds = 534.24 MB/sec
Research into running NixOS on a Radxa Rock 5B device.
Links: