[Radxa 4B] overlay for PCIe Gen2 no longer working

[jalsco]

Possibly a bug somewhere ? May be problems in my config, or maybe some config changes from ambian ?

Gen2 speeds for the NVMe on RockPi4 were working for several months, since 27 Nov 2022 and now is not. I didn't check after mid-Jan as was in different site, only checked in last few days, so I do not know exactly when it stopped working. Was working on earlier kernels : 5.15.76-rockchip64 and 5.15.80-rockchip64 in Dec 2022 and into Jan 2023. Now (not working) kernel is on 5.15.93-rockchip64 and can only get Gen1 speeds.

My published benchmarks https://pibenchmarks.com/user/SeanPH/ cover 27-Nov-2022 - 9-Dec-2022. They show Gen2 speeds. Great numbers actually ! (thanks DietPi 😉). Click on test-num to see system specs at time of each test (eg kernel release etc).

Within those benchmarks included are hdparm direct-read numbers, plus others. By just using: hdparm -Tt --direct /dev/nvme0n1p1 , and looking at the DIRECT disk reads, it pretty clear if Gen2 is being used or not:

Gen1:   430 - 520 MB/Sec for DIRECT disk reads (mostly 460-480)
Gen2:   660 - 985 MB/sec for DIRECT disk reads (mostly 730-850)

[Nov 27] Timing O_DIRECT disk reads: 2416 MB in 3.00 seconds = 805.08 MB/sec ( 5.15.76 ) 
[Dec 04] Timing O_DIRECT disk reads: 2814 MB in 3.00 seconds = 937.75 MB/sec ( 5.15.80 )
[Jun 13] Timing O_DIRECT disk reads: 1414 MB in 3.00 seconds = 470.85 MB/sec  ( 5.15.93 )  <-- today

Required Information

[Tue Jun 13] root@batopi: ~$ cat /boot/dietpi/.version
G_DIETPI_VERSION_CORE=8
G_DIETPI_VERSION_SUB=18
G_DIETPI_VERSION_RC=2
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
G_LIVE_PATCH_STATUS[0]='not applicable'
[Tue Jun 13] root@batopi: ~$ echo $G_DISTRO_NAME
bullseye
[Tue Jun 13] root@batopi: ~$ echo $G_HW_MODEL_NAME
ROCK 4 (aarch64)

My ambianEnv.txt :

bootlogo=false
overlay_prefix=rockchip
overlays=i2c7 pcie-gen2
fdtfile=rockchip/rk3399-rock-pi-4b.dtb
rootdev=UUID=dfa3ab95-cede-4159-9841-4e31bafe853e
rootfstype=ext4
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u
docker_optimizations=on
console=none

In rk3399-rock-pi.4b : note max-link-speed = <0x03>;

    pcie@f8000000 {
        compatible = "rockchip,rk3399-pcie";
        reg = <0x00 0xf8000000 0x00 0x2000000 0x00 0xfd000000 0x00 0x1000000>;
        reg-names = "axi-base\0apb-base";
        device_type = "pci";
        #address-cells = <0x03>;
        #size-cells = <0x02>;
        #interrupt-cells = <0x01>;
        aspm-no-l0s;
        bus-range = <0x00 0x1f>;
        clocks = <0x08 0xc5 0x08 0xc4 0x08 0x147 0x08 0xa0>;
        clock-names = "aclk\0aclk-perf\0hclk\0pm";
        interrupts = <0x00 0x31 0x04 0x00 0x00 0x32 0x04 0x00 0x00 0x33 0x04 0x00>;
        interrupt-names = "sys\0legacy\0client";
        interrupt-map-mask = <0x00 0x00 0x00 0x07>;
        interrupt-map = <0x00 0x00 0x00 0x01 0x13 0x00 0x00 0x00 0x00 0x02 0x13 0x01 0x00 0x00 0x00 0x03 0x13 0x02 0x00 0x00 0x00 0x04 0x13 0x03>;
        max-link-speed = <0x03>;
        msi-map = <0x00 0x14 0x00 0x1000>;
        phys = <0x15 0x00 0x15 0x01 0x15 0x02 0x15 0x03>;
        phy-names = "pcie-phy-0\0pcie-phy-1\0pcie-phy-2\0pcie-phy-3";
        ranges = <0x82000000 0x00 0xfa000000 0x00 0xfa000000 0x00 0x1e00000 0x81000000 0x00 0xfbe00000 0x00 0xfbe00000 0x00 0x100000>;
        resets = <0x08 0x82 0x08 0x83 0x08 0x84 0x08 0x85 0x08 0x86 0x08 0x81 0x08 0x80>;
        reset-names = "core\0mgmt\0mgmt-sticky\0pipe\0pm\0pclk\0aclk";
        status = "okay";
        ep-gpios = <0x16 0x1b 0x00>;
        num-lanes = <0x04>;
        pinctrl-0 = <0x17>;
        pinctrl-names = "default";
        vpcie12v-supply = <0x18>;
        vpcie3v3-supply = <0x19>;
        bus-scan-delay-ms = <0x5dc>;
        phandle = <0xca>;

        interrupt-controller {
            interrupt-controller;
            #address-cells = <0x00>;
            #interrupt-cells = <0x01>;
            phandle = <0x13>;
        };
    };

The overlay rockchip-pcie-gen2:

/dts-v1/;
/plugin/;

/ {
    model = "Radxa ROCK Pi 4B";
    compatible = "rockchip,rockpi", "rockchip,rk3399";

    fragment@0 {
        target = <0xffffffff>;

        __overlay__ {
            max-link-speed = <0x03>;
        };
    };

    __fixups__ {
        pcie0 = "/fragment@0:target:0";
    };
};

Now for something weird: The NVMe was fully running and installed on my system by 20 Nov 2022 in Gen1 speed. And I tried many, many variations of config settings to get the pcie-gen2 overlay working 20-21 Nov, but I could not. Speed for direct-reads always near 450-505 MB/sec. Even tried bypassing the overlay as per this: https://forum.armbian.com/topic/23574-howto-enable-pcie-gen2-to-get-max-speed-of-nvme-rockpi-4b/ -- but -- nothing worked.

So I left it alone, untouched, and a few days later (27 Nov) I came back to write it up and document all the things I had tried... and before I started writing I checked the speed, and AMAZINGLY it was 800+ MB/sec (ie Gen2) . And after then , even kernel update xxx.76 -> xx.80 , it always stayed at Gen2 speeds (until I tested it in June 2023). I am curious if there was an kernel update near 21-27 Nov, maybe that could explain it ?

[andreagdipaolo]

I am exeperiencing the same issue with the same board on the latest DietPi os and an intel 600p SSD.

[MichaIng]

Since Armbian did not push any new kernel update to their APT servers since February, we'd need to check with the latest build that comes with their images.

Could you test it with one of their image? https://www.armbian.com/rockpi4/

[andreagdipaolo]

I will check in the next days with another SSD. I can't do that on the same I use now because I have just set it up to be a server. The other (better, intel 7600p) drive won't work for boot, I will boot from USB and then do some tests. In order to have more than one overlay I just have to put a space between them in the env file, right?

[MichaIng]

In order to have more than one overlay I just have to put a space between them in the env file, right?

Exactly

[andreagdipaolo]

I'm testing on my Rock 4B with 4GB of RAM the Armbian_23.5.1_Rockpi-4b_bookworm_current_6.1.30_minimal image. I'm using a 256GB intel 600p SSD more than capable of reaching higher speeds with Gen2 enabled.

Without touching anything, the results are:

/dev/nvme0n1p1:
 Timing O_DIRECT cached reads:   1036 MB in  2.00 seconds = 517.31 MB/sec
 Timing O_DIRECT disk reads: 1710 MB in  3.00 seconds = 569.33 MB/sec

Enabling Gen2 via overlays=pcie-gen2 in the ENV file the results are:

/dev/nvme0n1p1:
 Timing O_DIRECT cached reads:   1504 MB in  2.00 seconds = 751.65 MB/sec
 Timing O_DIRECT disk reads: 2710 MB in  3.00 seconds = 902.90 MB/sec

It seems to work here with Armbian. After taking more than 2-3 consecutive tests the results increade by something like 20% in both cases. I usually run DietPi with a slight overclock at 2 GHz.

Linux rockpi-4b 6.1.30-rockchip64 #3 SMP PREEMPT Wed May 24 16:32:53 UTC 2023 aarch64 GNU/Linux

Strange thing is that with an SSD that I bought one week ago to expand my storage it seems to get Gen2 speeds even on Dietpi.

/dev/nvme0n1p1:
 Timing O_DIRECT cached reads:   2034 MB in  2.00 seconds = 1017.24 MB/sec
 Timing O_DIRECT disk reads: 2680 MB in  3.00 seconds = 892.90 MB/sec

So there is no issue, maybe. I can't check the intel ssd with Dietpi today, maybe tomorrow. Let me know.

[MichaIng]

Can you retest with DietPi v8.21? It applies a major Linux upgrade, being aligned with the version shipped with Armbian's images, or newer.

[andreagdipaolo]

Can you retest with DietPi v8.21? It applies a major Linux upgrade, being aligned with the version shipped with Armbian's images, or newer.

Sure, I will upgrade it tonight.

[andreagdipaolo]

It's doing fine!

 Timing O_DIRECT cached reads:   2010 MB in  2.00 seconds = 1006.07 MB/sec
 Timing O_DIRECT disk reads: 2714 MB in  3.00 seconds = 904.27 MB/sec
dietpi@DietPi:~/Docker$ sudo hdparm -Tt --direct  /dev/nvme0n1p1

Linux DietPi 6.1.46-current-rockchip64 #3 SMP PREEMPT Wed Aug 16 16:27:31 UTC 2023 aarch64 GNU/Linux

[MichaIng]

Okay great, many thanks for testing. Looks like there have been quite a bunch of bugs fixed since February. I'm working on an own APT repo for next DietPi release, form where we can ship own kernel builds (like this v6.1.46 one, with the Armbian build system) and update them more regularly for our users.