Research booting NanoPi R6C

[jakubgs]

Research into running NixOS on a NanoPi R6C device.

Links:

• https://www.friendlyelec.com/index.php?route=product/product&product_id=291
• https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C

[jakubgs]

I installed Ubuntu using rk3588-eflasher-ubuntu-jammy-minimal-5.10-arm64-20230527.img image. Here's the storage:

pi@NanoPi-R6C:~$ sudo fdisk -l /dev/mmcblk2 /dev/nvme0n1 
Disk /dev/mmcblk2: 28.91 GiB, 31037849600 bytes, 60620800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 73987B6B-4974-4C94-A3E8-58AB2EB7A946

Device           Start      End  Sectors  Size Type
/dev/mmcblk2p1   16384    24575     8192    4M unknown
/dev/mmcblk2p2   24576    32767     8192    4M unknown
/dev/mmcblk2p3   32768    40959     8192    4M unknown
/dev/mmcblk2p4   40960    73727    32768   16M unknown
/dev/mmcblk2p5   73728   155647    81920   40M unknown
/dev/mmcblk2p6  155648   221183    65536   32M unknown
/dev/mmcblk2p7  221184   286719    65536   32M unknown
/dev/mmcblk2p8  286720  5267455  4980736  2.4G unknown
/dev/mmcblk2p9 5267456 60620766 55353311 26.4G unknown

Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: SAMSUNG MZVLB512HAJQ-000L7              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[jakubgs]

aWe can get more insight into the partitions on the eMMC by using parted:

pi@NanoPi-R6C:~$ sudo parted /dev/mmcblk2 print
Model: MMC A3A551 (sd/mmc)
Disk /dev/mmcblk2: 31.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name      Flags
 1      8389kB  12.6MB  4194kB               uboot
 2      12.6MB  16.8MB  4194kB               misc
 3      16.8MB  21.0MB  4194kB               dtbo
 4      21.0MB  37.7MB  16.8MB               resource
 5      37.7MB  79.7MB  41.9MB               kernel
 6      79.7MB  113MB   33.6MB               boot
 7      113MB   147MB   33.6MB               recovery
 8      147MB   2697MB  2550MB  ext4         rootfs
 9      2697MB  31.0GB  28.3GB  ext4         userdata

https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#Install_to_target_board

[jakubgs]

Apparently the kernel needs to be written to the 5th partition based on instructions:

dd if=resource.img of=/dev/mmcblk2p4 bs=1M
dd if=kernel.img of=/dev/mmcblk2p5 bs=1M
dd if=uboot.img of=/dev/mmcblk2p1 bs=1M

And the Ubuntu image actually has no kernel or configuration in /boot:

pi@NanoPi-R6C:~$ ls -l /boot
total 0

Which is different from how NanoPC-T4 works, since there you actually have an extlinux config file in /boot and kernels.

[jakubgs]

I put in it a Samsung PM981 512 GB M.2 2280 (MZVLB512HAJQ) NVMe and here's some very basic benchmarks:

 > sudo hdparm -tT /dev/nvme0n1
/dev/nvme0n1:
 Timing cached reads:   8034 MB in  2.00 seconds = 4020.01 MB/sec
 Timing buffered disk reads: 1030 MB in  3.00 seconds = 343.10 MB/sec

/dev/nvme0n1:
 Timing cached reads:   8342 MB in  2.00 seconds = 4174.52 MB/sec
 Timing buffered disk reads: 1032 MB in  3.00 seconds = 343.92 MB/sec

/dev/nvme0n1:
 Timing cached reads:   8054 MB in  2.00 seconds = 4029.94 MB/sec
 Timing buffered disk reads: 1032 MB in  3.00 seconds = 343.72 MB/sec

 > sudo dd if=/dev/zero of=/mnt/test.img bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.06961 s, 528 MB/s

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.77926 s, 449 MB/s

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.48503 s, 479 MB/s

[jakubgs]

Some potentially interesting resources:

• https://gitlab.com/leonardp/nanopi-r4s-nixos - Someone's U-Boot and SD card image for Nanopi R4S.
• https://github.com/samueldr/Tow-Boot/tree/wip/rock5-vendor - Samueldr setup for building U-Boot for Radxa ROCK 5B
• https://wiki.radxa.com/Rock5/guide/build-u-boot-on-5b - Instructions for building U-Boot for ROCK B5
• https://u-boot.readthedocs.io/en/latest/board/amlogic/nanopi-k2.html - Instructions for building U-Boot for NanoPi K2
• https://wiki.friendlyelec.com/wiki/index.php/Building_U-boot_and_Linux_for_H5/H3/H2%2B - U-Boot for NanoPi H2/H3/H5

[jakubgs]

I have tried booting a generic SD card image of Nixos, since it works on NanoPC-T4, but it does not work: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.sd_image.aarch64-linux

[jakubgs]

I built the image:

 > ls -l $(nix-build -A radxa-rock5b.config.build.firmwareSPI)/binaries
trace: Using default Nixpkgs revision '1905f5f2e55e0db0bb6244cfe62cb6c0dbda391d'...
trace: ***********************************
trace: * Evaluating device: radxa-rock5b *
trace: ***********************************
trace: Building with crossSystem?: aarch64-linux != x86_64-linux → we are.
trace:     crossSystem: config: aarch64-unknown-linux-gnu
trace: Building with crossSystem?: aarch64-linux != x86_64-linux → we are.
trace:     crossSystem: config: aarch64-unknown-linux-gnu
total 1796
-r--r--r-- 1 root root 1866752 Jan  1  1970 Tow-Boot.spi.bi

But it's not clear how I'm supposed to install it. But these instructions for ROCK 5B just use dd:

sudo dd if=spi-image.img of=/dev/mtdblock0

https://wiki.radxa.com/Rock5/install/spi

[jakubgs]

I tried installing it in the U-Boot partition to see if I can maybe get the generic SD image to boot with it:

pi@NanoPi-R6C:~$ sudo dd if=/dev/zero of=/dev/mmcblk2p1
dd: writing to '/dev/mmcblk2p1': No space left on device
8193+0 records in
8192+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.410807 s, 10.2 MB/s

pi@NanoPi-R6C:~$ sudo dd if=Tow-Boot.spi.bin of=/dev/mmcblk2p1
3646+0 records in
3646+0 records out
1866752 bytes (1.9 MB, 1.8 MiB) copied, 0.153187 s, 12.2 MB/s

But of course that did not work. And now nothing boots. That was pure wishful thinking.

[jakubgs]

Using picocom with baud 1500000 via a USB debug port I was able to get the UART to work:

 > sudo minicom -b 1500000 -D /dev/ttyUSB0
...
DDR V1.10 75d050770f typ 23/02/28-20:47:10
LPDDR4X, 2112MHz
channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
Manufacturer ID:0x1
CH0 RX Vref:26.3%, TX Vref:21.8%,21.8%
CH1 RX Vref:26.3%, TX Vref:22.8%,21.8%
CH2 RX Vref:27.5%, TX Vref:19.8%,18.8%
CH3 RX Vref:26.7%, TX Vref:20.8%,21.8%
change to F1: 528MHz
change to F2: 1068MHz
change to F3: 1560MHz
change to F0: 2112MHz
out
U-Boot SPL board init
U-Boot SPL 2017.09-gd91b448214-221107 #root (Apr 10 2023 - 14:05:21)
unknown raw ID 0 0 0
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
part_get_info_efi: *** ERROR: Invalid GPT ***
part_get_info_efi: ***        Using Backup GPT ***
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
Trying to boot from MMC1
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###
# Reset the board to bootrom #

As we can see it fails to boot: SPL: failed to boot from all boot devices I wonder how this looked when I had a working Ubuntu image. Lets see.

[jakubgs]

There are instructions on how to build U-Boot from FriendlyELEC source:

git clone https://github.com/friendlyarm/uboot-rockchip --depth 1 -b nanopi6-v2017.09
UBOOT_SRC=$PWD/uboot-rockchip ./build-uboot.sh friendlycore-focal-arm64

https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#Compile_the_uboot

[jakubgs]

The Ubuntu flash image boots without issues:

U-Boot SPL board init
U-Boot SPL 2017.09-gd91b448214-221107 #root (Apr 10 2023 - 14:05:21)
unknown raw ID 0 0 0
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
Trying fit image at 0x4000 sector
## Verified-boot: 0
## Checking atf-1 0x00040000 ... sha256(ac799e4aa0...) + OK
## Checking uboot 0x00200000 ... sha256(ecc2ca052f...) + OK
## Checking fdt 0x0032ce40 ... sha256(e936f08b25...) + OK
## Checking atf-2 0xff100000 ... sha256(c0f2f7769f...) + OK
## Checking atf-3 0x000f0000 ... sha256(338cd7cb83...) + OK
## Checking optee 0x08400000 ... sha256(fde0860845...) + OK
Jumping to U-Boot(0x00200000) via ARM Trusted Firmware(0x00040000)

boot.log

[jakubgs]

At least we can see the NVMe shows up properly at boot here, so maybe NixOS won't have same issue as on NanoPC-T4.

[    4.475674] nvme nvme0: pci function 0004:41:00.0
[    4.475754] nvme 0004:41:00.0: enabling device (0000 -> 0002)
[    4.478778] nvme nvme0: Shutdown timeout set to 8 seconds
[    4.497511] nvme nvme0: 8/0/0 default/read/poll queues
[    4.524958] rk-pcie fe180000.pcie: PCIe Link up, LTSSM is 0x130011
[    4.525205] rk-pcie fe180000.pcie: PCI host bridge to bus 0003:30

boot.log

[jakubgs]

If we take a look at an SD card image for Ubuntu we can see the same layout as on the eMMC:

rk3588-eflasher-ubuntu-jammy-minimal-5.10-arm64-20230527.img

 > sudo fdisk -l /dev/loop3 
Disk /dev/loop3: 7.26 GiB, 7799999488 bytes, 15234374 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 73987B6B-4974-4C94-A3E8-58AB2EB7A946

Device         Start      End  Sectors  Size Type
/dev/loop3p1 3563520 15228927 11665408  5.6G unknown
/dev/loop3p2   16384    24575     8192    4M unknown
/dev/loop3p3   24576    32767     8192    4M unknown
/dev/loop3p4   32768    40959     8192    4M unknown
/dev/loop3p5   40960    73727    32768   16M unknown
/dev/loop3p6   73728   155647    81920   40M unknown
/dev/loop3p7  155648   221183    65536   32M unknown
/dev/loop3p8  221184   286719    65536   32M unknown
/dev/loop3p9  286720  3563519  3276800  1.6G unknown

Partition table entries are not in disk order.

While the NixOS image uses a FAT32 partition for the boot files, which I guess is why it doesn't work:

 > sudo fdisk -l /dev/loop2 
Disk /dev/loop2: 2.54 GiB, 2723512320 bytes, 5319360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2178694e

Device       Boot Start     End Sectors  Size Id Type
/dev/loop2p1      16384   77823   61440   30M  b W95 FAT32
/dev/loop2p2 *    77824 5319359 5241536  2.5G 83 Linux

[jakubgs]

Here are official RockChip docs about the boot process: https://opensource.rock-chips.com/wiki_Boot_option

[jakubgs]

The docs state that:

By default, the system will be booted from the TF card first, but this is not the case under all conditions. This section will explain all situations in detail;

Refer to rockchip official document [1], there are two types of loader program: 1) U-Boot TPL/SPL (i.e. upsream U-Boot, also called mainline U-Boot) 2) Rockchip MiniLoader

Things to note: 1) FriendlyELEC's image uses Rockchip MiniLoader 2) The third-party image usually uses U-Boot TPL/SPL

https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#The_Boot_order_between_eMMC_and_SD_card

[jakubgs]

There are instructions for building an U-Boot using FriendlyELEC scripts:

git clone https://github.com/friendlyarm/rkbin --single-branch --depth 1 -b nanopi6
git clone https://github.com/friendlyarm/uboot-rockchip --single-branch --depth 1 -b nanopi6-v2017.09
export PATH=/opt/FriendlyARM/toolchain/11.3-aarch64/bin/:$PATH
cd uboot-rockchip/
./make.sh nanopi6

https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#Build_u-boot_v2017.09

But I'm still don't know how that could be modified to boot NixOS. I might have to read up on U-Boot itself.

[jakubgs]

I found some interesting info in this issue. For example this comment describes how to load kernel from Ext4 partition:

setenv m "1080p60hz"
setenv m_bpp "32"
setenv condev "console=ttyS0,115200n8 console=tty0"   # on both
setenv bootargs "root=/dev/mmcblk0p2 rootwait rw ${condev} no_console_suspend hdmimode=${m} m_bpp=${m_bpp} vout=${vout} fsck.repair=yes"
setenv loadaddr "0x11000000"
setenv dtb_loadaddr "0x1000000"
setenv initrd_loadaddr "0x13000000"
ext4load mmc 0:2 ${initrd_loadaddr} /boot/uInitrd
ext4load mmc 0:2 ${loadaddr} /boot/Image
ext4load mmc 0:2 ${dtb_loadaddr} /boot/meson64_odroidc2.dtb
booti ${loadaddr} ${initrd_loadaddr} ${dtb_loadaddr}

• https://github.com/hypriot/image-builder-odroid-c2/issues/5#issuecomment-183966128

[jakubgs]

And indeed, the NanoPi branch in uboot-rockchip repo has a README about handling Ext4 filesystems: https://github.com/friendlyarm/uboot-rockchip/blob/nanopi6-v2017.09/doc/README.ext4

[jakubgs]

I notice that ext4 support is not enabled in the NanoPi 6 build config:

~/soft/uboot-rockchip nanopi6-v2017.09
 > grep ext4 configs/nanopi*          

~/soft/uboot-rockchip nanopi6-v2017.09
 > grep ext4 configs/odroid_defconfig                       
CONFIG_CMD_EXT4=y
CONFIG_CMD_EXT4_WRITE=y

[jakubgs]

In theory this line suggests I could provide U-Boot with custom commands at startup via UART console:

Hit key to stop autoboot('CTRL+C'):  0 

But the real question is how can I modify the config to read extlinux.conf file from an Ext4 /boot partition.

[jakubgs]

And it doesn't seem like it supports FAT filesystem either:

~/soft/uboot-rockchip nanopi6-v2017.09
 > grep fat configs/nanopi*          

~/soft/uboot-rockchip nanopi6-v2017.09
 > grep fat configs/odroid_defconfig
CONFIG_CMD_FAT=y

But there appears to be several FAT filestem flags:

 > grep 'CONFIG_.*FAT' fs/fs.c cmd/pxe.c 
fs/fs.c:#ifdef CONFIG_FS_FAT
fs/fs.c:#ifdef CONFIG_FAT_WRITE
cmd/pxe.c:#ifdef CONFIG_CMD_FAT

For example some usage can be seen in fs/fs.c: https://github.com/friendlyarm/uboot-rockchip/blob/93ceeb4da7efbabefe9b88b57093f90ea731d502/fs/fs.c#L147-L163

Not yet sure how the CMD ones are different.

[jakubgs]

According to the README:

/cmd            U-Boot commands functions

So it seems to me that just means you can enable FAT filesystem support with CONFIG_FS_FAT, but if you want interactive commands at boot time like fatload you need CONFIG_CMD_FAT.

[jakubgs]

So I think a pretty good assumption would be that current U-Boot can't load the config from generic NixOS sd card because it lacks CONFIG_FS_FAT enabled. So lets try that first.

[jakubgs]

Building U-Boot from Rockchip was failing due to lack of cross platform GCC:

 > ./make.sh nanopi6
## make nanopi6_defconfig -j24
#
# configuration written to .config
#
which: no aarch64-linux-gnu-gcc in (/nix/store/fwc0m6fsccir8zznkl7m8iznv5wsjznw-bash-interactive-5.1-p16/bin:/nix/store/n845kl3915ri14zlkqq7wf3n3xabmgal-patchelf-0.14.5/bin:/nix/store/ykcrnkiicqg1pwls9kgnmf0hd9qjqp4x-gcc-wrapper-11.3.0/bin:/nix/store/65v2c245h5qa9mpc7dxhqkfjinl6phx0-gcc-11.3.0/bin:/nix/store/rvgp96bwfmgz4b163flszfaygmhx8wl4-glibc-2.34-210-bin/bin:/nix/store/qarssrazji0q9xp80xg8shsm2crckfr0-coreutils-9.0/bin:/nix/store/n5ypzvl7dijcg24isyngvw8fx0ri6hff-binutils-wrapper-2.38/bin:/nix/store/0q9hm42fapihzj1d64nxqmbml7fpb2d6-binutils-2.38/bin:/nix/store/9l2pdqd2cs9hqyk53w543lpnp0vqaysd-ncurses-6.3-p20220507-dev/bin:/nix/store/sb1wx6xz9yj05d13w5lpdljspllarbim-ncurses-6.3-p20220507/bin:/nix/store/qarssrazji0q9xp80xg8shsm2crckfr0-coreutils-9.0/bin:/nix/store/j25abvpcbappy74w23l8lfcz7gkrsjhy-findutils-4.9.0/bin:/nix/store/2kz1pihzg1jfif46mdm917xmj6r9xyz6-diffutils-3.8/bin:/nix/store/yyg26p5j2mrjwpkbk1djh4nxlsm2p4rw-gnused-4.8/bin:/nix/store/p7pqs5c4zfc4y977p626zch11msmmpj8-gnugrep-3.7/bin:/nix/store/pmh9q9k0g9s189v0iqrxpdp8j1g77gmd-gawk-5.1.1/bin:/nix/store/3fw5n1g3bb925hfll7fj7x94bd0q6k0r-gnutar-1.34/bin:/nix/store/nrvhb0yvawiqgrwbmbfmhjrmy934hhs5-gzip-1.12/bin:/nix/store/n7208v30hf3z4sz6127947vmzpfl46nb-bzip2-1.0.6.0.2-bin/bin:/nix/store/c5myz1zs9bsaq3y4s0rcxgkzb11irwra-gnumake-4.3/bin:/nix/store/9zm6br2ri10a0b71dll2wrim5bnhg6b6-bash-5.1-p16/bin:/nix/store/8636bkd1gg5s8675ipb9wf98wrc5mdpw-patch-2.7.6/bin:/nix/store/5ywhaxrasrp3pj72dnyy40mimx13fi0p-xz-5.2.5-bin/bin:/run/wrappers/bin:/home/jakubgs/.nix-profile/bin:/etc/profiles/per-user/jakubgs/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin:/home/jakubgs/go/bin:/home/jakubgs/bin:/home/jakubgs/go/bin:/home/jakubgs/bin:/home/jakubgs/go/bin:/home/jakubgs/bin)
ERROR: No find aarch64-linux-gnu-gcc

So I will need to create Nix shell for this.

[jakubgs]

So I have created a Nix shell using this example:

{ pkgs ? import <nixpkgs> {
    crossSystem = { config = "aarch64-unknown-linux-gnu"; };
  }
}:

pkgs.mkShell {
  buildInputs = [ gcc ];
}

https://nixos.wiki/wiki/Cross_Compiling#How_to_obtain_a_shell_with_a_cross_compiler

But I'm getting some issues, for example:

aarch64-unknown-linux-gnu-ld: cannot find rcs: No such file or directory
aarch64-unknown-linux-gnu-ld: cannot find arch/arm/cpu/built-in.o: No such file or directory
aarch64-unknown-linux-gnu-ld: cannot find arch/arm/lib/lib.a: No such file or directory

[jakubgs]

This link shows build dependencies for U-Boot on Debian: https://u-boot.readthedocs.io/en/latest/build/gcc.html#debian-based

[jakubgs]

I tried building with a derivation:

{ pkgs ? import <nixpkgs> {
    crossSystem = { config = "aarch64-unknown-linux-gnu"; };
  }
}:

pkgs.stdenv.mkDerivation {
  name = "uboot-nanopi6";
  version = "v2017.09";

  src = pkgs.fetchFromGitHub {
    owner = "friendlyarm";
    repo = "uboot-rockchip";
    rev = "nanopi6-v2017.09";
    sha256 = "sha256-8zg/7py9FXn+6GY+D5zSAoOk/3NgQj1FMLWBYzZJKZM=";
  };
  srcs = [
    (pkgs.fetchFromGitHub {
      owner = "friendlyarm";
      repo = "rkbin";
      rev = "nanopi6";
      sha256 = "sha256-XGnRUW5Foz0U1V08Kd6Cr/OkwY81xzCzS05irdawWjo=";
    })
  ];

  buildInputs = with pkgs; [ gcc linuxHeaders ];
  nativeBuildInputs = with pkgs; [ which glibc autoPatchelfHook ];

  unpackPhase = ''
    cp -r $srcs rkbin
    cp -r $src uboot-rockchip
    find ./ -name 'built-in.o'
    cd uboot-rockchip
  '';

  preBuild = ''
    chmod -R +w ./ ../rkbin
    autoPatchelf ./ ../rkbin
    patchShebangs make.sh scripts arch

    function makeWrapper() {
      AR=$(which $1) 
      echo -e "#!$SHELL\nexec $CC \$@" > $TMP_BIN/$2
      chmod +x $TMP_BIN/$2
    }
    TMP_BIN=$(mktemp -d)
    export PATH="$TMP_BIN:$PATH"
    makeWrapper aarch64-unknown-linux-gnu-gcc cc
    makeWrapper aarch64-unknown-linux-gnu-gcc aarch64-linux-gnu-gcc
    makeWrapper aarch64-unknown-linux-gnu-ld aarch64-linux-gnu-ld
    makeWrapper aarch64-unknown-linux-gnu-ar aarch64-linux-gnu-ar
  '';

  buildPhase = ''
    runHook preBuild
    ./make.sh nanopi6
    runHook postBuild
  '';
}

But it just fails with the same errors:

aarch64-unknown-linux-gnu-gcc: error: rcs: No such file or directory
aarch64-unknown-linux-gnu-gcc: error: arch/arm/cpu/built-in.o: No such file or directory

I have no idea what the rcs part is about, but the arch/arm/cpu/built-in.o part seems like it should come from Linux kernel.

[jakubgs]

I thought that the rcs thing might come from the toolchain provided by FriendlyELC: https://drive.google.com/file/d/16sJLrExcaCnCba13cc22Sw2k9pEJdEMw/view?usp=share_link But when I search the tarball I can't find anything like that:

 > tar tvf cross_compiler.tgz | grep rcs  

[jakubgs]

But the build instructions in the wiki do say we need the toolchain in the PATH anyway:

export PATH=/opt/FriendlyARM/toolchain/11.3-aarch64/bin/:$PATH

So maybe trying to build this with Nix is a bad idea, if the only easy way is using their toolchain.

[jakubgs]

I tried building in an Ubuntu Docker container but it also fails:

root@7d39d1851d79:/build/uboot-rockchip# ./make.sh nanopi6
## make nanopi6_defconfig -j24
  HOSTCC  scripts/basic/fixdep
/bin/sh: 1: scripts/basic/fixdep: not found
make[1]: *** [scripts/Makefile.host:99: scripts/basic/fixdep] Error 127
make: *** [Makefile:400: scripts_basic] Error 2

root@7d39d1851d79:/build/uboot-rockchip# file scripts/basic/fixdep
scripts/basic/fixdep: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=de98b94d832293bf2ebe88c0558c01374d85fc94, for GNU/Linux 4.4.0, not stripped

root@7d39d1851d79:/build/uboot-rockchip# arch
x86_64

Seems to me like it might be simpler to build this on the NanoPi itself.

[jakubgs]

But if I try to build on the NanoPi itself I get:

pi@NanoPi-R6C:/mnt/uboot-rockchip$ ./make.sh nanopi6
grep: .config: No such file or directory
## make nanopi6_defconfig -j16
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  SHIPPED scripts/kconfig/zconf.tab.c
  SHIPPED scripts/kconfig/zconf.lex.c
  SHIPPED scripts/kconfig/zconf.hash.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
/mnt/FriendlyARM/toolchain/11.3-aarch64/bin/aarch64-linux-gnu-gcc: 1: ELF: not found
/mnt/FriendlyARM/toolchain/11.3-aarch64/bin/aarch64-linux-gnu-gcc: 2: Syntax error: "(" unexpected
aarch64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
dirname: missing operand
Try 'dirname --help' for more information.
scripts/kconfig/conf  --silentoldconfig Kconfig
  CHK     include/config.h
  UPD     include/config.h
  CFG     u-boot.cfg
  GEN     include/autoconf.mk.dep
  CFG     spl/u-boot.cfg
  CFG     tpl/u-boot.cfg
aarch64-linux-gnu-gcc: fatal error: cannot execute 'cc1': execvp: No such file or directory
compilation terminated.
aarch64-linux-gnu-gcc: fatal error: cannot execute 'cc1': execvp: No such file or directory
compilation terminated.
aarch64-linux-gnu-gcc: fatal error: cannot execute 'cc1': execvp: No such file or directory
compilation terminated.
aarch64-linux-gnu-gcc: fatal error: cannot execute 'cc1': execvp: No such file or directory
make[1]: *** [scripts/Makefile.autoconf:83: spl/u-boot.cfg] Error 1
compilation terminated.
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [scripts/Makefile.autoconf:87: tpl/u-boot.cfg] Error 1
make[1]: *** [scripts/Makefile.autoconf:79: u-boot.cfg] Error 1
make[1]: *** [scripts/Makefile.autoconf:50: include/autoconf.mk.dep] Error 1
make: *** No rule to make target 'include/config/auto.conf', needed by 'include/config/uboot.release'.  Stop.

So it seems like the toolchain is indeed intended to be used on an x86_64 system to cross-compile U-Boot.

[jakubgs]

If I set export KBUILD_VERBOSE=1 I can see the commands being executed by the build scripts:

 > export MAKEFLAGS=-j1
 > export KBUILD_VERBOSE=1
 > ./make.sh nanopi6
...
make -f ./scripts/Makefile.build obj=arch/arm/cpu
make -f ./scripts/Makefile.build obj=arch/arm/cpu/armv8
   rm -f arch/arm/cpu/built-in.o; aarch64-linux-gnu-ar rcs arch/arm/cpu/built-in.o
make -f ./scripts/Makefile.build obj=arch/arm/lib
aarch64-unknown-linux-gnu-gcc: error: rcs: No such file or directory
aarch64-unknown-linux-gnu-gcc: error: arch/arm/cpu/built-in.o: No such file or directory
make[1]: *** [scripts/Makefile.build:359: arch/arm/cpu/built-in.o] Error 1

It appears the command in play here is: aarch64-linux-gnu-ar rcs arch/arm/cpu/built-in.o According to the manual the first argument is the archive and the other ones are files in the archive:

 > ar --help
Usage: ar [emulation options] [-]{dmpqrstx}[abcDfilMNoOPsSTuvV] [member-name] [count] archive-file file...

It seems to me like the arch/arm/cpu/built-in.o file is what we want out of a file called rcs, which is apparently an archive.

[jakubgs]

I have two locations that are the source of this:

# If the list of objects to link is empty, just create an empty built-in.o
cmd_link_o_target = $(if $(strip $(obj-y)),\
              $(LD) $(ld_flags) -r -o $@ $(filter $(obj-y), $^) \
              $(cmd_secanalysis),\
              rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) $@)

https://github.com/friendlyarm/uboot-rockchip/blob/93ceeb4d/scripts/Makefile.build#L352-L356

And:

cmd_link_l_target = rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) $@ $(lib-y)

https://github.com/friendlyarm/uboot-rockchip/blob/93ceeb4d/scripts/Makefile.build#L382

[jakubgs]

I modified the files to check which one it is and it appears it's both:

aarch64-unknown-linux-gnu-gcc: error: rcsYYY: No such file or directory
aarch64-unknown-linux-gnu-gcc: error: arch/arm/cpu/built-in.o: No such file or directory
make[1]: *** [scripts/Makefile.build:359: arch/arm/cpu/built-in.o] Error 1
make: *** [Makefile:1308: arch/arm/cpu] Error 2
make: *** Waiting for unfinished jobs....
  AR      arch/arm/lib/lib.a
aarch64-unknown-linux-gnu-gcc: error: rcsXXX: No such file or directory
aarch64-unknown-linux-gnu-gcc: error: arch/arm/lib/lib.a: No such file or directory
make[1]: *** [scripts/Makefile.build:385: arch/arm/lib/lib.a] Error 1

Which suggests that KBUILD_ARFLAGS is empty and should not be.

[jakubgs]

It seems to me like rcs are just flags, and not a filename:

root@0b0fef29089b:/# /build/FriendlyARM/toolchain/11.3-aarch64/bin/aarch64-linux-gnu-ar --help 
Usage: /build/FriendlyARM/toolchain/11.3-aarch64/bin/aarch64-linux-gnu-ar [emulation options] [-]{dmpqrstx}[abcDfilMNoOPsSTuvV] [--plugin <name>] [member-name] [count] archive-file file...
       /build/FriendlyARM/toolchain/11.3-aarch64/bin/aarch64-linux-gnu-ar -M [<mri-script]
 commands:
  d            - delete file(s) from the archive
  m[ab]        - move file(s) in the archive
  p            - print file(s) found in the archive
  q[f]         - quick append file(s) to the archive
  r[ab][f][u]  - replace existing or insert new file(s) into the archive
  s            - act as ranlib
  t[O][v]      - display contents of the archive
  x[o]         - extract file(s) from the archive
 command specific modifiers:
  [a]          - put file(s) after [member-name]
  [b]          - put file(s) before [member-name] (same as [i])
  [D]          - use zero for timestamps and uids/gids (default)
  [U]          - use actual timestamps and uids/gids
  [N]          - use instance [count] of name
  [f]          - truncate inserted file names
  [P]          - use full path names when matching
  [o]          - preserve original dates
  [O]          - display offsets of files in the archive
  [u]          - only replace files that are newer than current archive contents
 generic modifiers:
  [c]          - do not warn if the library had to be created
  [s]          - create an archive index (cf. ranlib)
  [l <text> ]  - specify the dependencies of this library
  [S]          - do not build a symbol table
  [T]          - deprecated, use --thin instead
  [v]          - be verbose
  [V]          - display the version number
  @<file>      - read options from <file>
  --target=BFDNAME - specify the target object format as BFDNAME
  --output=DIRNAME - specify the output directory for extraction operations
  --record-libdeps=<text> - specify the dependencies of this library
  --thin       - make a thin archive

The version of ar from the toolchain appears to support flags without the dash prefix based on this part:

[-]{dmpqrstx}[abcDfilMNoOPsSTuvV] 

[jakubgs]

And indeed, I can see the built-in.o files:

root@0b0fef29089b:/# find /build/ -iname built-in.o
/build/uboot-rockchip/board/rockchip/nanopi6/built-in.o
/build/uboot-rockchip/common/built-in.o
/build/uboot-rockchip/common/init/built-in.o
/build/uboot-rockchip/cmd/built-in.o
/build/uboot-rockchip/arch/arm/lib/built-in.o
/build/uboot-rockchip/arch/arm/mach-rockchip/rk3588/built-in.o
/build/uboot-rockchip/arch/arm/mach-rockchip/built-in.o
/build/uboot-rockchip/arch/arm/cpu/armv8/built-in.o

Which means the ar command provided with GCC from Nix does not have the same format.

[jakubgs]

And indeed. The ar from the toolchain is the GNU ar:

root@0b0fef29089b:/# /build/FriendlyARM/toolchain/11.3-aarch64/bin/aarch64-linux-gnu-ar --version
GNU ar (GNU Binutils) 2.38

While the one I'm setting up in Nix is the GCC one:

[nix-shell:~/soft/uboot/uboot-rockchip]$ aarch64-linux-gnu-ar --version
aarch64-unknown-linux-gnu-gcc (GCC) 9.3.0

[jakubgs]

I managed to get it fixed by linking ar from binutils to aarch64-linux-gnu-ar, but now it's failing(much later) due to:

ERROR: No 'dtc', please: apt-get install device-tree-compiler

[jakubgs]

Oh shit, I think it worked:

Info:Pack loader ok.
pack loader okay! Input: /build/rkbin/RKBOOT/RK3588MINIALL.ini
/build/uboot-rockchip

Image(no-signed, version=0): uboot.img (FIT with uboot, trust...) is ready
Image(no-signed): rk3588_spl_loader_v1.09.112.bin (with spl, ddr...) is ready
pack uboot.img okay! Input: /build/rkbin/RKTRUST/RK3588TRUST.ini

Platform RK3588 is build OK, with new .config(make nanopi6_defconfig -j24)
aarch64-linux-gnu-
Sun Jun  4 22:50:58 UTC 2023
buildPhase completed in 48 seconds

[jakubgs]

The resulting image is nearly 4 MB in size:

 > du -hs /nix/store/sca27khvfl08iyaxjg2i744l6vr71886-uboot-nanopi6-aarch64-unknown-linux-gnu/uboot.img
3.8M    /nix/store/sca27khvfl08iyaxjg2i744l6vr71886-uboot-nanopi6-aarch64-unknown-linux-gnu/uboot.img

Which fits with the size of the uboot partition on the eMMC:

Device           Start      End  Sectors  Size Type
/dev/mmcblk2p1   16384    24575     8192    4M unknown

I wonder how the size changes if I add EXT4 support.

[jakubgs]

I flashed it to the uboot partition:

pi@NanoPi-R6C:~$ sudo dd if=uboot.img of=/dev/mmcblk2p1
8192+0 records in
8192+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.452276 s, 9.3

And rebooted:

Net:   eth1: ethernet@fe1c0000
Hit key to stop autoboot('CTRL+C'):  0 
## Booting FIT Image FIT: No fit blob
FIT: No FIT image
ANDROID: reboot reason: "(none)"
optee api revision: 2.0
TEEC: Waring: Could not find security partition
Not AVB images, AVB skip
No valid android hdr
Android image load failed
Android boot failed, error -1.

## Booting Rockchip Format Image
fdt      @ 0x08300000 (0x000350d5)
kernel   @ 0x00400000 (0x02107008)
ramdisk  @ 0x0a200000 (0x007b2bc0)

And it works. So that's progress.

[jakubgs]

At least we can see that opening the filesystem works:

=> mmc list
mmc@fe2c0000: 1
mmc@fe2e0000: 0 (eMMC)
=> ls mmc 0:9 /
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
               0 .ext4.resized
<DIR>       4096 root
<DIR>       4096 work
=> ls mmc 0:8 /
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
<SYM>          7 bin
<DIR>       4096 boot
...

[jakubgs]

The U-Boot by default has some pre-defined boot* scripts:

=> printenv
arch=arm
autoload=no
baudrate=1500000
board=nanopi6
board_name=nanopi6
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}extlinux/extlinux.conf
boot_net_usb_start=usb start
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_targets=mmc1 mmc0 mtd2 mtd1 mtd0 usb0 pxe dhcp 
bootargs=storagemedia=emmc androidboot.storagemedia=emmc androidboot.mode=normal androidboot.dtbo_idx=0
bootcmd=boot_fit;boot_android ${devtype} ${devnum};bootrkp;run distro_bootcmd;
bootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;
bootcmd_mmc0=setenv devnum 0; run mmc_boot
bootcmd_mmc1=setenv devnum 1; run mmc_boot
bootcmd_mtd0=setenv devnum 0; run mtd_boot
bootcmd_mtd1=setenv devnum 1; run mtd_boot
bootcmd_mtd2=setenv devnum 2; run mtd_boot
bootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fi
bootcmd_usb0=setenv devnum 0; run usb_boot
bootdelay=1
...
mmc_boot=if mmc dev ${devnum}; then setenv devtype mmc; run scan_dev_for_boot_part; fi
mtd_boot=if mtd_blk dev ${devnum}; then setenv devtype mtd; run scan_dev_for_boot_part; fi
...
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}extlinux/extlinux.conf; then echo Found ${prefix}extlinux/extlinux.conf; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; do
...

[jakubgs]

I tried some of the existing ones but they dont' work:

=> run bootcmd_mmc0
switch to partitions #0, OK
mmc0(part 0) is current device
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **

=> run bootcmd_mmc1
switch to partitions #0, OK
mmc1 is current device
Failed to mount ext2 filesystem...
** Unrecognized filesystem type *

[jakubgs]

This page explains mmc sub-commands in U-Boot quite well: https://u-boot.readthedocs.io/en/latest/usage/cmd/mmc.html

=> mmc list
mmc@fe2c0000: 1 (SD)
mmc@fe2e0000: 0 (eMMC)
=> mmc dev 1
switch to partitions #0, OK
mmc1 is current device
=> mmc part 

Partition Map for MMC device 1  --   Partition Type: EFI

Part    Start LBA       End LBA         Name
        Attributes
        Type GUID
        Partition GUID
  1     0x00366000      0x00e85fff      "userdata"
        attrs:  0x0000000000000000
        type:   f808d051-1602-4dcd-9452-f9637fefc49a
        guid:   b750e44e-833f-4a30-c38c-b117241d84d4
  2     0x00004000      0x00005fff      "uboot"
        attrs:  0x0000000000000000
        type:   c6d08308-e418-4124-8890-f8411e3d8d87
        guid:   a1c81622-7741-47ad-b846-c6972488d396
  3     0x00006000      0x00007fff      "misc"
        attrs:  0x0000000000000000
        type:   2a583e58-486a-4bd4-ace4-8d5454e97f5c
        guid:   43784a32-a03d-4ade-92c6-ede64ff9b794
...

But the result from mmc part doesn't look correct, since the NixOS SD card image has only two partitions.

[jakubgs]

I can actually read the SD card FAT partition with boot images and scripts:

=> ls mmc 1:2 /
      512   armstub8-gic.bin
    52593   bcm2711-rpi-4-b.dtb
    52682   bcm2711-rpi-400.dtb
    53202   bcm2711-rpi-cm4.dtb
    50504   bcm2711-rpi-cm4s.dtb
    52476   bootcode.bin
      946   config.txt
     7266   fixup.dat
     5399   fixup4.dat
     3173   fixup4cd.dat
     8381   fixup4db.dat
     8387   fixup4x.dat
     3173   fixup_cd.dat
    10230   fixup_db.dat
    10228   fixup_x.dat
  2975680   start.elf
  2251392   start4.elf
   805756   start4cd.elf
  3747688   start4db.elf
  2998888   start4x.elf
   805756   start_cd.elf
  4819624   start_db.elf
  3722504   start_x.elf
   573544   u-boot-rpi3.bin
   620720   u-boot-rpi4.bin

25 file(s), 0 dir(s)

But there's no extlinux.conf, so I guess that's why it doesn't treat it as bootable.

[jakubgs]

Actually, the second partition on the SD card contains a /boot with an extlinux.conf:

pi@NanoPi-R6C:~$ cat /media/pi/NIXOS_SD/boot/extlinux/extlinux.conf 
# Generated file, all changes will be lost on nixos-rebuild!

# Change this to e.g. nixos-42 to temporarily boot to an older configuration.
DEFAULT nixos-default

MENU TITLE ------------------------------------------------------------
TIMEOUT 30

LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/8qajvgdwgl58xahzwk4xd8ay3sgyynzj-linux-6.1.31-Image
  INITRD ../nixos/8kqlms92cy9ycvh2x189xalay0d4fj39-initrd-linux-6.1.31-initrd
  APPEND init=/nix/store/hfbp8ig1n2jsnq9hxv2sqxxsiyxwj03x-nixos-system-nixos-23.05pre-git/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 cma=32M console=ttyS2,115200n8 console=tty0 nohibernate loglevel=7
  FDTDIR ../nixos/8qajvgdwgl58xahzwk4xd8ay3sgyynzj-linux-6.1.31-dtbs

[jakubgs]

But I can't actually get to that partition via U-Boot:

=> ls mmc 1:3
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **

It works fine for the EXT4 on the eMMC though:

=> ls mmc 0:9
<DIR>       4096 .
<DIR>       4096 ..
<DIR>      16384 lost+found
               0 .ext4.resized
<DIR>       4096 root
<DIR>       4096 work

[jakubgs]

It appears the boot from eMMC is done using the bootrkp command, which is a custom one added by Rockchip: https://github.com/friendlyarm/uboot-rockchip/blob/nanopi6-v2017.09/cmd/bootrkp.c

And in theory running distro_bootcmd should work, but it doesn't.

[jakubgs]

At least committed the derivation to make the uboot.img with FAT and EXT4 support:

• https://github.com/jakubgs/nixos-config/commit/71006c13 - hosts/arael: add U-Boot derivation
• https://github.com/jakubgs/nixos-config/commit/3746028e - hosts/arael: add FAT and EXT4 to uboot image

Next I need to figure out why it's not seeing the SD card partition with /boot.