Forlinx RK3568-based SoM cannot initialize PCIE30X2 on 5.10-develop

[kkursor]

I am trying to develop Buildroot-based firmware based on Linux 5.10 instead of stock 4.19 one. Using 5.10-develop branch. SoM is https://www.forlinx.net/product/rk3568-som-125.html

Stock DT is:

&pcie30phy {
    status = "okay";
};

&pcie3x2 {
    reset-gpios = <&gpio0 RK_PC6 GPIO_ACTIVE_HIGH>;
    enable-gpios = <&gpio0 RK_PA5 GPIO_ACTIVE_HIGH>;
    vpcie3v3-supply = <&vcc3v3_sys>;
    status = "okay";
};

Device tree fragment of PCIE3.0 X2 in my DT is:

&pcie30phy {
    status = "okay";
};

&pcie3x2 {
    compatible = "rockchip,rk3568-pcie";
    reset-gpios = <&gpio0 RK_PC6 GPIO_ACTIVE_HIGH>;
    prsnt-gpios = <&gpio0 RK_PC1 GPIO_ACTIVE_HIGH>;
    enable-gpios = <&gpio0 RK_PA5 GPIO_ACTIVE_HIGH>;
    vpcie3v3-supply = <&vcc3v3_sys>;
    status = "okay";
};

rk3568.dtsi contains different Compatible string:

pcie3x2: pcie@fe280000 {
        compatible = "rockchip,rk3568-pcie", "snps,dw-pcie";

but it hangs on boot, so I switched it off.

With this compatible param string the kernel boots but fails to initialize PCIe 3.0 X2 interface. It seems that discrete PCIe PHY is not being initialized.

# dmesg | grep -E "(pci|phy)"
[    0.585806] bus: 'pci': add driver serial
[    0.585897] bus: 'pci': add driver exar_serial
[    0.619275] device class 'sas_phy': registering
[    0.619802] bus: 'pci': add driver megaraid_sas
[    0.621330] bus: 'pci': add driver hisi_sas_v3_hw
[    0.621410] bus: 'pci': add driver ahci
[    0.622716] ahci fc800000.sata: supply phy not found, using dummy regulator
[    0.623422] device: 'phy:phy-fe840000.phy.6--platform:fc800000.sata': device_add
[    0.623513] ahci fc800000.sata: Linked as a consumer to phy-fe840000.phy.6
[    0.631080] bus: 'pci': add driver sata_sil24
[    0.634942] bus: 'platform': add driver physmap-flash
[    0.648670] bus: 'pci': add driver mdio_thunder
[    0.655396] bus: 'pci': add driver amd-xgbe
[    0.658063] bus: 'pci': add driver thunder_xcv
[    0.658454] bus: 'pci': add driver thunder_bgx
[    0.658764] bus: 'pci': add driver nicpf
[    0.661693] bus: 'pci': add driver fsl_enetc
[    0.661765] bus: 'pci': add driver fsl_enetc_vf
[    0.661819] bus: 'pci': add driver fsl_enetc_mdio
[    0.661882] bus: 'pci': add driver fsl_enetc_ptp
[    0.665271] bus: 'pci': add driver hns3
[    0.666299] bus: 'pci': add driver e1000
[    0.667337] bus: 'pci': add driver e1000e
[    0.668380] bus: 'pci': add driver igb
[    0.669540] bus: 'pci': add driver igbvf
[    0.671346] bus: 'pci': add driver sky2
[    0.671418] bus: 'pci': add driver r8168
[    0.671473] bus: 'pci': add driver r8169
[    0.972877] bus: 'pci': add driver stmmaceth
[    0.975165] bus: 'pci': add driver vfio-pci
[    0.975879] bus: 'platform': add driver tegra-phy
[    0.977406] bus: 'pci': add driver dwc3-pci
[    0.977485] bus: 'pci': add driver dwc3-haps
[    0.979349] device: 'platform:fe820000.phy--platform:fcc00000.dwc3': device_add
[    0.979440] platform fcc00000.dwc3: Linked as a consumer to fe820000.phy
[    0.979460] device: 'platform:fe8a0000.usb2-phy--platform:fcc00000.dwc3': device_add
[    0.979551] platform fcc00000.dwc3: Linked as a consumer to fe8a0000.usb2-phy
[    0.981203] device: 'phy:phy-fe8a0000.usb2-phy.1--platform:fcc00000.dwc3': device_add
[    0.981338] dwc3 fcc00000.dwc3: Linked as a consumer to phy-fe8a0000.usb2-phy.1
[    0.981473] device: 'phy:phy-fe820000.phy.4--platform:fcc00000.dwc3': device_add
[    0.981566] dwc3 fcc00000.dwc3: Linked as a consumer to phy-fe820000.phy.4
[    0.984326] dwc3-of-simple usbdrd: Dropping the link to fe820000.phy
[    0.984335] device: 'platform:fe820000.phy--platform:usbdrd': device_unregister
[    0.984412] dwc3-of-simple usbdrd: Dropping the link to fe8a0000.usb2-phy
[    0.984420] device: 'platform:fe8a0000.usb2-phy--platform:usbdrd': device_unregister
[    0.985172] device: 'platform:fe830000.phy--platform:fd000000.dwc3': device_add
[    0.985264] platform fd000000.dwc3: Linked as a consumer to fe830000.phy
[    0.985298] device: 'platform:fe8a0000.usb2-phy--platform:fd000000.dwc3': device_add
[    0.985404] platform fd000000.dwc3: Linked as a consumer to fe8a0000.usb2-phy
[    0.987144] device: 'phy:phy-fe8a0000.usb2-phy.0--platform:fd000000.dwc3': device_add
[    0.987249] dwc3 fd000000.dwc3: Linked as a consumer to phy-fe8a0000.usb2-phy.0
[    0.987324] device: 'phy:phy-fe830000.phy.5--platform:fd000000.dwc3': device_add
[    0.987425] dwc3 fd000000.dwc3: Linked as a consumer to phy-fe830000.phy.5
[    0.990727] dwc3-of-simple usbhost: Dropping the link to fe830000.phy
[    0.990735] device: 'platform:fe830000.phy--platform:usbhost': device_unregister
[    0.990865] dwc3-of-simple usbhost: Dropping the link to fe8a0000.usb2-phy
[    0.990874] device: 'platform:fe8a0000.usb2-phy--platform:usbhost': device_unregister
[    0.995058] bus: 'pci': add driver isp1760
[    0.995733] ehci-pci: EHCI PCI platform driver
[    0.996141] bus: 'pci': add driver ehci-pci
[    0.997279] device: 'phy:phy-fe8b0000.usb2-phy.3--platform:fd800000.usb': device_add
[    0.997402] ehci-platform fd800000.usb: Linked as a consumer to phy-fe8b0000.usb2-phy.3
[    0.997426] phy phy-fe8b0000.usb2-phy.3: illegal mode
[    1.019950] device: 'phy:phy-fe8b0000.usb2-phy.2--platform:fd880000.usb': device_add
[    1.020058] ehci-platform fd880000.usb: Linked as a consumer to phy-fe8b0000.usb2-phy.2
[    1.042588] ohci-pci: OHCI PCI platform driver
[    1.043006] bus: 'pci': add driver ohci-pci
[    1.043959] device: 'phy:phy-fe8b0000.usb2-phy.3--platform:fd840000.usb': device_add
[    1.044074] ohci-platform fd840000.usb: Linked as a consumer to phy-fe8b0000.usb2-phy.3
[    1.044089] phy phy-fe8b0000.usb2-phy.3: illegal mode
[    1.111271] device: 'phy:phy-fe8b0000.usb2-phy.2--platform:fd8c0000.usb': device_add
[    1.111369] ohci-platform fd8c0000.usb: Linked as a consumer to phy-fe8b0000.usb2-phy.2
[    1.180494] bus: 'pci': add driver xhci_hcd
[    1.195095] bus: 'pci': add driver ci_hdrc_pci
[    1.354129] devices_kset: Moving 3c0800000.pcie to end of list
[    1.354140] PM: Moving platform:3c0800000.pcie to end of list
[    1.354154] platform 3c0800000.pcie: Retrying from deferred list
[    1.354886] bus: 'platform': driver_probe_device: matched device 3c0800000.pcie with driver rk-pcie
[    1.354908] platform 3c0800000.pcie: probe deferral - supplier vcc3v3-sys not ready
[    1.354916] platform 3c0800000.pcie: Added to deferred list
[    1.379380] devices_kset: Moving 3c0800000.pcie to end of list
[    1.379386] PM: Moving platform:3c0800000.pcie to end of list
[    1.379415] devices_kset: Moving pcie30-avdd0v9 to end of list
[    1.379420] PM: Moving platform:pcie30-avdd0v9 to end of list
[    1.379426] devices_kset: Moving pcie30-avdd1v8 to end of list
[    1.379431] PM: Moving platform:pcie30-avdd1v8 to end of list
[    1.386185] devices_kset: Moving pcie30-avdd0v9 to end of list
[    1.386192] PM: Moving platform:pcie30-avdd0v9 to end of list
[    1.386202] platform pcie30-avdd0v9: Retrying from deferred list
[    1.386480] bus: 'platform': driver_probe_device: matched device pcie30-avdd0v9 with driver reg-fixed-voltage
[    1.386493] bus: 'platform': really_probe: probing driver reg-fixed-voltage with device pcie30-avdd0v9
[    1.386515] reg-fixed-voltage pcie30-avdd0v9: no pinctrl handle
[    1.386632] pcie30_avdd0v9: supplied by vcc3v3_sys
[    1.387389] driver: 'reg-fixed-voltage': driver_bound: bound to device 'pcie30-avdd0v9'
[    1.387445] bus: 'platform': really_probe: bound device pcie30-avdd0v9 to driver reg-fixed-voltage
[    1.387460] devices_kset: Moving pcie30-avdd1v8 to end of list
[    1.387467] PM: Moving platform:pcie30-avdd1v8 to end of list
[    1.387476] platform pcie30-avdd1v8: Retrying from deferred list
[    1.387734] bus: 'platform': driver_probe_device: matched device pcie30-avdd1v8 with driver reg-fixed-voltage
[    1.387747] bus: 'platform': really_probe: probing driver reg-fixed-voltage with device pcie30-avdd1v8
[    1.387767] reg-fixed-voltage pcie30-avdd1v8: no pinctrl handle
[    1.387879] pcie30_avdd1v8: supplied by vcc3v3_sys
[    1.388642] driver: 'reg-fixed-voltage': driver_bound: bound to device 'pcie30-avdd1v8'
[    1.388696] bus: 'platform': really_probe: bound device pcie30-avdd1v8 to driver reg-fixed-voltage
[    1.487128] devices_kset: Moving 3c0800000.pcie to end of list
[    1.487134] PM: Moving platform:3c0800000.pcie to end of list
[    1.487144] platform 3c0800000.pcie: Retrying from deferred list
[    1.487743] bus: 'platform': driver_probe_device: matched device 3c0800000.pcie with driver rk-pcie
[    1.487758] bus: 'platform': really_probe: probing driver rk-pcie with device 3c0800000.pcie
[    1.487785] rk-pcie 3c0800000.pcie: no pinctrl handle
[    1.487840] rk-pcie 3c0800000.pcie: adding to PM domain pipe
[    1.487850] rk-pcie 3c0800000.pcie: genpd_add_device()
[    1.487880] rk-pcie 3c0800000.pcie: Clock aclk_pcie30x2_mst con_id (null) managed by runtime PM.
[    1.487895] rk-pcie 3c0800000.pcie: Clock aclk_pcie30x2_slv con_id (null) managed by runtime PM.
[    1.487913] rk-pcie 3c0800000.pcie: Clock aclk_pcie30x2_dbi con_id (null) managed by runtime PM.
[    1.487931] rk-pcie 3c0800000.pcie: Clock pclk_pcie30x2 con_id (null) managed by runtime PM.
[    1.487950] rk-pcie 3c0800000.pcie: Clock clk_pcie30x2_aux_ndft con_id (null) managed by runtime PM.
[    1.488119] rk-pcie 3c0800000.pcie: invalid prsnt-gpios property in node
[    1.488883] device: 'regulator:regulator.12--platform:3c0800000.pcie': device_add
[    1.488987] devices_kset: Moving 3c0800000.pcie to end of list
[    1.488996] PM: Moving platform:3c0800000.pcie to end of list
[    1.489007] rk-pcie 3c0800000.pcie: Linked as a consumer to regulator.12
[    1.489015] => rk_pcie_enable_power
[    1.489028] rk_pcie_enable_power ret=0
[    1.489485] device: 'phy:phy-fe8c0000.phy.7--platform:3c0800000.pcie': device_add
[    1.489901] devices_kset: Moving 3c0800000.pcie to end of list
[    1.489910] PM: Moving platform:3c0800000.pcie to end of list
[    1.489920] rk-pcie 3c0800000.pcie: Linked as a consumer to phy-fe8c0000.phy.7
[    1.490453] rockchip_p3phy_rk3568_init: lock failed 0x6890000, check input refclk and power supply
[    1.491248] phy phy-fe8c0000.phy.7: phy init failed --> -110
[    1.491756] rk-pcie 3c0800000.pcie: fail to init phy, err -110
[    1.492276] rk-pcie 3c0800000.pcie: phy init failed
[    1.492745] => rk_pcie_disable_power
[    1.492759] rk_pcie_disable_power ret=0 =>
[    1.493086] rk-pcie 3c0800000.pcie: removing from PM domain pipe
[    1.493456] rk-pcie 3c0800000.pcie: genpd_remove_device()
[    1.493549] rk-pcie 3c0800000.pcie: Dropping the link to phy-fe8c0000.phy.7
[    1.493557] device: 'phy:phy-fe8c0000.phy.7--platform:3c0800000.pcie': device_unregister
[    1.493795] rk-pcie 3c0800000.pcie: Dropping the link to regulator.12
[    1.493805] device: 'regulator:regulator.12--platform:3c0800000.pcie': device_unregister
[    1.493932] rk-pcie: probe of 3c0800000.pcie failed with error -110

As I see on oscilloscope, REFCLK 100MHz signal is there in U-Boot, but disappears on kernel boot, and I can't find out why. :(

I am not very experienced in kernel and driver developing, but I see no significant differences in DT and involved drivers in stock vendors' 4.19 kernel and 5.10-develop branch.

Could you please tell how to troubleshoot this?

[gosha-z]

enable-gpio has no effect in DW PCI-Express driver. As a workaround, you may create a fictive power supply, mention it in 'vpcie3v3-supply' in DT entry and move RK_PA5 gpio to this supply. But it will not resolve hang problem, event with or without ptsnt-gpios

[Pasha-From-Russia]

did you manage to get PCIE working? If so, could you please share your solution?

[kkursor]

@Pasha-From-Russia привет! Нет, не получилось. Пришлось юзать ядро от производителя. В чём конкретно была проблема - так и не разобрались.

[Pasha-From-Russia]

@Pasha-From-Russia привет! Нет, не получилось. Пришлось юзать ядро от производителя. В чём конкретно была проблема - так и не разобрались.

Жаль... спасибо за ответ! А ядро от производителя - какой версии?

EDIT: а, 4.19, я слепой =_=

[kkursor]

4.19, если я правильно помню. В целом, для нас это не критично, хотелось завести 5.10 чисто по приколу, но второй PCIe нашим нужен, поэтому решили не тратить время и откатиться.

[Pasha-From-Russia]

4.19, если я правильно помню. В целом, для нас это не критично, хотелось завести 5.10 чисто по приколу, но второй PCIe нашим нужен, поэтому решили не тратить время и откатиться.

а сможете подсказать, какая у вас eMMC стоит (на фотографиях вашего SoM-а не удалось разобрать)? Мы на 5.10 перешли только потому, что наша eMMC стандарта 5.1, а ядро 4.19 поддерживает только до 2-го (что довольно странно, так как в спецификации к камню указан стандарт eMMC 5.1). Либо в ваше ядро добавлены патчи для поддержки нужных eMMC...

[kkursor]

какая у вас eMMC стоит

Sandisk SDINBDG4-16G https://www.chipdip.ru/product0/8003306691

[Pasha-From-Russia]

Sandisk SDINBDG4-16G https://www.chipdip.ru/product0/8003306691

спасибо!

[Pasha-From-Russia]

@kkursor приветствую! Можно ли у вас попросить defconfig ядра, defconfig uboot и dts для uboot?