CM3 on RPi CM4 IO Board

[engineering-and-development]

Hello everyone. Hello Tidone, thanks for all the work!

Can anyone confirm or deny the functionality of the PCIe socket on the RPi Compute Module 4 IO Board? I tweaked the DTS for more then one day now without any luck. Is it even possible that it can work at all?

Greetings

[Tidone]

Hi,

We did not test PCIe on the CM3, but it should work in theory because the relevant pins are the same on the Raspberry CM4 and the Radxa CM3.

PCIe is disabled by default (see the Linux kernel dts), and if you enable it you have to add the respective pinctrl definitions. You can check the dts for the Quartz64 for the correct definitions. You also have to enable the PCIe power regulator (see here).

Best Regards

[engineering-and-development]

Hey, thanks for the fast response!

Thanks for pointing out what is disabled by default. I enabled the pinctrl definitions. Reset is on GPIO1 not GPIO0. I think I saw it once wrongly during the search.

&pcie2x1 {
        pinctrl-0 = <&pcie20m2_pins>;
        reset-gpios = <&gpio1 RK_PB2 GPIO_ACTIVE_HIGH>;
        status = "okay";
        vpcie3v3-supply = <&vcc3v3_sys>;
};

PCIe is tied to VCCIO1 power domain if I see it correctly. It should be on acodec, but still I added / tested:

        vcc3v3_sys: vcc3v3_sys {
                compatible = "regulator-fixed";
                enable-active-high;
                gpio = <&gpio0 RK_PD3 GPIO_ACTIVE_HIGH>;
                pinctl-names = "default";
                pinctrl-0 = <&pcie_enable_h>;
                regulator-name = "vcc3v3_sys";
                regulator-always-on;
                regulator-boot-on;
                regulator-min-microvolt = <3300000>;
                regulator-max-microvolt = <3300000>;
                vin-supply = <&vcc_sys>;
        };

GPIO0 RD3 goings into nothing, so still I tested it like above with and without enable pin.

VCCIO1 again:

&pmu_io_domains {
        pmuio1-supply = <&vcc3v3_pmu>;
        pmuio2-supply = <&vcc_3v3>;
        vccio1-supply = <&vccio_acodec>;
        vccio2-supply = <&vcc_1v8>;
        vccio3-supply = <&vccio_sd>;
        vccio4-supply = <&vcc_1v8>;
        vccio5-supply = <&vcc_3v3>;
        vccio6-supply = <&vcc_3v3>;
        vccio7-supply = <&vcc_3v3>;
        status = "okay";
};

The acodec is working. I changed the voltage under vccio_acodec: LDO_REG4 from 3V3 to 2V8 and could measure it working (with oscilloscope at R44). I did this because I saw a lot of warnings during boot up and wanted to make sure.

[    0.656833] fan53555-regulator 0-001c: FAN53555 Option[12] Rev[15] Detected!
[    0.657503] fan53555-regulator 0-001c: Looking up vin-supply from device tree
[    0.658200] vdd_cpu: supplied by vcc_sys
[    0.658568] vcc_sys: could not add device link regulator.3: -ENOENT
[    0.662595] vdd_cpu: 712 <--> 1200 mV at 1025 mV, enabled
[    0.668128] rk808 0-0020: chip id: 0x8170
[    0.750187] rk808-regulator rk808-regulator: there is no dvs0 gpio
[    0.750798] rk808-regulator rk808-regulator: there is no dvs1 gpio
[    0.751526] rk808 0-0020: Looking up vcc1-supply from device tree
[    0.752092] vdd_logic: supplied by vcc_sys
[    0.752472] vcc_sys: could not add device link regulator.4: -ENOENT
[    0.757927] vdd_logic: 500 <--> 1200 mV at 900 mV, enabled
[    0.759323] rk808 0-0020: Looking up vcc2-supply from device tree
[    0.759885] vdd_gpu: supplied by vcc_sys
[    0.760248] vcc_sys: could not add device link regulator.5: -ENOENT
[    0.764763] vdd_gpu: 500 <--> 1200 mV at 900 mV, enabled
[    0.766105] rk808 0-0020: Looking up vcc3-supply from device tree
[    0.766668] vcc_ddr: supplied by vcc_sys
[    0.767030] vcc_sys: could not add device link regulator.6: -ENOENT
[    0.770566] vcc_ddr: at 500 mV, enabled
[    0.771798] rk808 0-0020: Looking up vcc4-supply from device tree
[    0.772361] vcc_3v3: supplied by vcc_sys
[    0.772724] vcc_sys: could not add device link regulator.7: -ENOENT
[    0.777136] vcc_3v3: 3300 mV, enabled
[    0.778356] rk808 0-0020: Looking up vcc5-supply from device tree
[    0.778918] vcca1v8_pmu: supplied by vcc_sys
[    0.779310] vcc_sys: could not add device link regulator.8: -ENOENT
[    0.783355] vcca1v8_pmu: 1800 mV, enabled
[    0.784617] rk808 0-0020: Looking up vcc5-supply from device tree
[    0.785178] vdda_0v9: supplied by vcc_sys
[    0.785551] vcc_sys: could not add device link regulator.9: -ENOENT
[    0.789095] vdda_0v9: 900 mV, enabled
[    0.790289] rk808 0-0020: Looking up vcc5-supply from device tree
[    0.790851] vdda0v9_pmu: supplied by vcc_sys
[    0.791292] vcc_sys: could not add device link regulator.10: -ENOENT
[    0.795405] vdda0v9_pmu: 900 mV, enabled
[    0.796635] rk808 0-0020: Looking up vcc6-supply from device tree
[    0.797197] vccio_acodec: supplied by vcc_sys
[    0.797599] vcc_sys: could not add device link regulator.11: -ENOENT
[    0.802426] vccio_acodec: 3300 mV, enabled
[    0.803676] rk808 0-0020: Looking up vcc6-supply from device tree
[    0.804238] vccio_sd: supplied by vcc_sys
[    0.804655] vcc_sys: could not add device link regulator.12: -ENOENT
[    0.808726] vccio_sd: 1800 <--> 3300 mV at 3300 mV, enabled
[    0.810090] rk808 0-0020: Looking up vcc6-supply from device tree
[    0.810652] vcc3v3_pmu: supplied by vcc_sys
[    0.811084] vcc_sys: could not add device link regulator.13: -ENOENT
[    0.815533] vcc3v3_pmu: 3300 mV, enabled
[    0.816778] rk808 0-0020: Looking up vcc7-supply from device tree
[    0.817340] vcc_1v8: supplied by vcc_sys
[    0.817750] vcc_sys: could not add device link regulator.14: -ENOENT
[    0.821316] vcc_1v8: 1800 mV, enabled
[    0.822518] rk808 0-0020: Looking up vcc7-supply from device tree
[    0.823080] vcc1v8_dvp: supplied by vcc_sys
[    0.823467] vcc_sys: could not add device link regulator.15: -ENOENT
[    0.827015] vcc1v8_dvp: 1800 mV, enabled
[    0.828242] rk808 0-0020: Looking up vcc7-supply from device tree
[    0.828804] vcc2v8_dvp: supplied by vcc_sys
[    0.829191] vcc_sys: could not add device link regulator.16: -ENOENT
[    0.832764] vcc2v8_dvp: 2800 mV, enabled
[    0.834023] rk808 0-0020: Looking up vcc8-supply from device tree
[    0.834632] boost: supplied by vcc_sys
[    0.834984] vcc_sys: could not add device link regulator.17: -ENOENT
[    0.836116] boost: Bringing 4700000uV into 5000000-5000000uV
[    0.841412] boost: 5000 mV, enabled
[    0.843588] otg_switch: no parameters, disabled
[    0.844225] rk808 0-0020: Looking up vcc9-supply from device tree
[    0.844852] otg_switch: supplied by boost
[    0.852595] input: rk805 pwrkey as /devices/platform/fdd40000.i2c/i2c-0/0-0020/rk805-pwrkey/input/input0
[    0.866588] rk808-rtc rk808-rtc: registered as rtc0
[    0.869864] rk808-rtc rk808-rtc: setting system clock to 2017-08-05T09:00:04 UTC (1501923604)
[    0.872953] arm-scmi firmware:scmi: Failed. SCMI protocol 21 not active.
[    0.880140] cpu cpu0: Looking up cpu-supply from device tree

Overall I guess I have to clean up my dts... I tested many different ways and options and went baby steps (a lot).

PCIe shares a line with SATA, so I turned all off. The comments might be from quartz64 files. Found them somewhere...

/* sata1 is muxed with the usb3 port */
&sata1 {
        status = "disabled";
};

/* sata2 is muxed with the pcie2 slot*/
&sata2 {
        status = "disabled";
};

I also played with switch both on / off (all combinations):

&combphy2 {
        status = "okay";
};

&combphy1 {
        status = "disabled";
};

By best guess would be that I have to turn something else off in order to get PCIe running. Maybe it some other switch in buildroot or somewhere else? I would not argue that my PCBs are broken. At least not yet ;)

Any angle point I could look into?

[engineering-and-development]

I was just wondering: Do I have to enable a certain I2C interface? Then I checked schematic and saw, that I2C is not wired to the PCIe header. I am not familiar the the PCIe standard / protocol, but I hope I2C is just a feature and not necessary.

[Tidone]

Just to make sure, but you power the IO Board via the 12V DC barrel jack, right?

You could check out flatmax's repo because that one uses the Radxa kernel. Maybe the upstream 6.1 kernel broke PCIe suport on the CM3?

You could also try to install one of the official Debian images to check if PCIe is working there. If it is working you can copy the dtb and decompile it with dtc (dtc -I dtb -O dts -o out.dts in.dtb). This should give you a working device tree which you can compare to your own one (although the device trees are not compatible because of the different kernel versions)

I2C should not be necessary. AFAIK the I2C bus is only used to read some specific device info from the PCIe card to make sure the kernel loads the correct drivers.

[engineering-and-development]

I power via J20. It is easier to me to crimp some connector than finding a DC barrel... It is on the same trace and I have 3V3 on PCIe (TP1).

I tried flatmax's repos and was not able to compile without patching a dozen of files. On one point I stopped the process and came to your repo, as this worked for me out of the box.

I thought about going with an official repo. I was hesitant as I have a Radxa CM3 in a RPi CM4 board. I do not want to break anything in case some miss-configured pin goes H / L. Although I will give this a test now as I now know how to get back the dts file. Thanks!

[Tidone]

I just compared my kernel config with the one on the Radxa kernel repo and saw that I most likely disabled PCIe support to save some resources.

You have to edit the linux.defconfig file, add the following lines anr rebuild the kernel:

CONFIG_PCI=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIE_ROCKCHIP_HOST=y
CONFIG_PCIE_ROCKCHIP_DW_HOST=y
CONFIG_PHY_ROCKCHIP_SNPS_PCIE3=y

Also, every time you edit the linux defconfig or device tree you have to let buildroot recompile the kernel. You have to delete the output/build/linux-6.1 folder and run brmake again to do so.

[Tidone]

The /boot directory is empty. This directory is just a placeholder. The real boot folder is most likely in a FAT partition at the start of the image.

[engineering-and-development]

Sorry. I was wrong and deleted the old comment of mine. I had to mount boot.img (not rootfs.img) and then I had the files. Anyways. I run the image and changed to correct dtb. This is the end of dmesg:

[    5.523314] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    6.536891] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    7.550198] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    8.563542] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    9.576673] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[   10.590070] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x1
[   11.603516] rk-pcie 3c0000000.pcie: PCIe Link Fail
[   11.603622] rk-pcie 3c0000000.pcie: failed to initialize host

image: radxa-cm3-io-ubuntu-focal-server-arm64-20221101-0254-gpt.img.xz

Your hint to edit linux.defconfig sounds like gold to me. This could be the issue. I will check! Thanks for pointing out the trick with deleting output/build/linux-6.1. I had to recompile everything for more then a day strait. I already figured this trick out :smile: :smiling_face_with_tear:

[engineering-and-development]

Thanks! Changing linux.defconfig helped. Now I have at least some info about a PCIe interface at all!

[   11.367331] rockchip-dw-pcie 3c0000000.pcie: Looking up vpcie3v3-supply from device tree
[   11.371553] platform 3c0000000.pcie: deferred probe pending

I will tweak my dts now again.

[Tidone]

I just pushed two additional branches which use the default Radxa Kernels (one for 4.19 and one for 5.10). These branches use the default kernel configs and device trees. Everything that works on the default Debian image should also work on these branches.

If you are switching from main to another branch make sure to read the Readme and clean the buildroot directory to make sure the buildroot patches are applied correctly.

[engineering-and-development]

Thanks! I will checkout and test 4.19 and 5.10.

Taking the image radxa-cm3-io-ubuntu-focal-server-arm64-20221101-0254-gpt.img, it's not working:

root@radxa-cm3-io:~# dmesg | grep pci
[    0.461046] rk-pcie 3c0000000.pcie: Looking up vpcie3v3-supply from device tree
[    0.461158] rk-pcie 3c0000000.pcie: Linked as a consumer to regulator.2
[    0.461584] rk-pcie 3c0000000.pcie: missing legacy IRQ resource
[    0.461644] rk-pcie 3c0000000.pcie: Missing *config* reg space
[    0.461718] rk-pcie 3c0000000.pcie: host bridge /pcie@fe260000 ranges:
[    0.461792] rk-pcie 3c0000000.pcie:   err 0x300000000..0x3007fffff -> 0x00000000
[    0.461844] rk-pcie 3c0000000.pcie:    IO 0x300800000..0x3008fffff -> 0x00800000
[    0.461894] rk-pcie 3c0000000.pcie:   MEM 0x300900000..0x33fffffff -> 0x00900000
[    0.690925] ehci-pci: EHCI PCI platform driver
[    1.470009] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    2.484098] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    3.496732] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x1
[    4.509995] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    5.523314] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    6.536827] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    7.550174] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    8.563511] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[    9.576838] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[   10.590176] rk-pcie 3c0000000.pcie: PCIe Linking... LTSSM is 0x0
[   11.603496] rk-pcie 3c0000000.pcie: PCIe Link Fail
[   11.603602] rk-pcie 3c0000000.pcie: failed to initialize host

Compiling this image (from repo) after change in linux.defconfig I get:

# dmesg | grep pci
[   11.366678] rockchip-dw-pcie 3c0000000.pcie: Looking up vpcie3v3-supply from device tree
[   11.371014] platform 3c0000000.pcie: deferred probe pending

[engineering-and-development]

I can say that out of the box 4.19 and 5.10 seem promising. I changed in buildroot.cm3/board/RK3566.cm3/linux/rk3566-radxa-cm3-spa.dts

&combphy1 {
        status = "okay";
};

into

&combphy2 {
        status = "okay";
};

&pcie2x1 {
        pinctrl-0 = <&pcie20m2_pins>;
        reset-gpios = <&gpio1 RK_PB2 GPIO_ACTIVE_HIGH>;
        status = "okay";
        vpcie3v3-supply = <&vcc_sys>;
};

and I can say, that I have lspci showing up some good values, e.g.:

USB3.0 adapter card:

01:00.0 Class 0c03: 1b73:1100
00:00.0 Class 0604: 1d87:3566

Intel NIC:

01:00.0 Class 0200: 8086:10d3
00:00.0 Class 0604: 1d87:3566

I have a bunch of errors, e.g. kernel 4.19 constant spamming of:

[   21.763121] dwmmc_rockchip fe2b0000.dwmmc: failed to enable vmmc regulator
[   21.799539] dwmmc_rockchip fe2b0000.dwmmc: could not set regulator OCR (-22)
[   21.799605] dwmmc_rockchip fe2b0000.dwmmc: failed to enable vmmc regulator
[   21.836288] dwmmc_rockchip fe2b0000.dwmmc: could not set regulator OCR (-22)

kernel 5.10 constant spamming of:

[    9.571057] mmc_host mmc0: Bus speed (slot 0) = 375000Hz (slot req 375000Hz, actual 375000HZ div = 0)
[   10.729273] mmc_host mmc0: Bus speed (slot 0) = 375000Hz (slot req 400000Hz, actual 375000HZ div = 0)
[   10.771010] mmc_host mmc0: Bus speed (slot 0) = 375000Hz (slot req 375000Hz, actual 375000HZ div = 0)
[   11.902602] mmc_host mmc0: Bus speed (slot 0) = 375000Hz (slot req 400000Hz, actual 375000HZ div = 0)
[   11.944177] mmc_host mmc0: Bus speed (slot 0) = 375000Hz (slot req 375000Hz, actual 375000HZ div = 0)

I will checkout the original 6.10 branch as well and test it again with the minor change to dts and defconfig out of curiosity.

[engineering-and-development]

Linux 6.10 branch

I updated buildroot.cm3/board/RK3566.cm3/linux/rk3566-radxa-cm3-spa.dts (see above) and buildroot.cm3/board/RK3566.cm3/linux/linux.defconfig (see above what your wrote).

Furthermore I activated:

make menuconfig Kernel Linux Kernel Tools pci :heavy_check_mark: (I don't know if necessary) Target packages Hardware handling pcitools :heavy_check_mark: (for lspci command)

lspci shows nothing.

# lspci
#

Tail of dmesg:

[   11.363906] rockchip-dw-pcie 3c0000000.pcie: Looking up vpcie3v3-supply from device tree
[   11.366720] platform fd000000.usb: deferred probe pending
[   11.367260] platform 3c0000000.pcie: deferred probe pending

So overall kernel 6.10 branch does not work for PCIe like the others. I guess something else needs to get activated. @Tidone if you have any clue, please let me know. I can test it as everything is on my table at the moment. Depends on the fact if you want to get PCIe working on your fork or not?!

I can pick 4.19 or 5.10 for my intentions.

[Tidone]

Yeah, I also noticed the constant spamming of log messages, and added some workarounds to the Readme files of the respective branches. The 5.10 branch can be easily fixed by inserting an SD Card into the slot, or by deactivating mmc0 in the device tree. I didn't test if the 4.19 branch can be fixed.

The deferred probe pending messages mean that a kernel driver is missing. I could compare the kernel configs but there's hundreds of changes and renamed parameters between them. I don't really need PCIe, so I don't think I will fix this right now.

You could run lsmod on the 4.19, 5.10 and 6.1 branches to see which modules are loaded. Maybe one of them is responsible for PCIe.