tq-systems / u-boot-tqmaxx

U-Boot tree for TQ Systems ARM based SOM
4 stars 5 forks source link

TQMa8MxML: Problem boot from some micro SD cards #4

Open wico-frad opened 1 week ago

wico-frad commented 1 week ago

Hi everyone,

We are experiencing boot issues with our embedded module TQMa8MxML when using certain micro SD cards, specifically the SanDisk MAX ENDURANCE microSD™ Card - 32GB Class 10.

The u-boot fails to load the Device Tree Blob from the micro SD card, resulting in the error message: 'WARN: Cannot load the DT!'. The boot log below illustrates the failure after enabling #define CONFIG_MMC_TRACE.

Does anyone have any insights into why this might be occurring?


 BuildInfo:
  - ATF cfccd00
  - U-Boot 2020.04-tq-ga04fdffac7

Normal Boot
Hit any key to stop autoboot:  3  2  1  0 
Booting from mmc ...
CMD_SEND:0
        ARG          0x00000000
        MMC_RSP_NONE
CMD_SEND:8
        ARG          0x000001aa
        MMC_RSP_R1,5,6,7     0x000001aa 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x41ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x40ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0xc0ff8000 
CMD_SEND:2
        ARG          0x00000000
        MMC_RSP_R2       0x03534453 
                         0x48333247 
                         0x8097e54d 
                         0x40017b00 
                    DUMPING DATA
                    000 - 03 53 44 53 
                    004 - 48 33 32 47 
                    008 - 80 97 e5 4d 
                    012 - 40 01 7b 00 
CMD_SEND:3
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0xaaaa0520 
CMD_SEND:9
        ARG          0xaaaa0000
        MMC_RSP_R2       0x400e0032 
                         0xdb790000 
                         0xedc87f80 
                         0x0a404000 
                    DUMPING DATA
                    000 - 40 0e 00 32 
                    004 - db 79 00 00 
                    008 - ed c8 7f 80 
                    012 - 0a 40 40 00 
CMD_SEND:7
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000700 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:51
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00000002
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x80fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:13
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
switch to partitions #0, OK
mmc1 is current device
CMD_SEND:0
        ARG          0x00000000
        MMC_RSP_NONE
CMD_SEND:8
        ARG          0x000001aa
        MMC_RSP_R1,5,6,7     0x000001aa 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x40ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x40ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0xc0ff8000 
CMD_SEND:2
        ARG          0x00000000
        MMC_RSP_R2       0x03534453 
                         0x48333247 
                         0x8097e54d 
                         0x40017b00 
                    DUMPING DATA
                    000 - 03 53 44 53 
                    004 - 48 33 32 47 
                    008 - 80 97 e5 4d 
                    012 - 40 01 7b 00 
CMD_SEND:3
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0xaaaa0520 
CMD_SEND:9
        ARG          0xaaaa0000
        MMC_RSP_R2       0x400e0032 
                         0xdb790000 
                         0xedc87f80 
                         0x0a404000 
                    DUMPING DATA
                    000 - 40 0e 00 32 
                    004 - db 79 00 00 
                    008 - ed c8 7f 80 
                    012 - 0a 40 40 00 
CMD_SEND:7
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000700 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:51
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00000002
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x80fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:13
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00004000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00004000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:18
        ARG          0x00004154
        RET          -70
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:0
        ARG          0x00000000
        MMC_RSP_NONE
CMD_SEND:8
        ARG          0x000001aa
        MMC_RSP_R1,5,6,7     0x000001aa 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x40ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x40ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0xc0ff8000 
CMD_SEND:2
        ARG          0x00000000
        MMC_RSP_R2       0x03534453 
                         0x48333247 
                         0x8097e54d 
                         0x40017b00 
                    DUMPING DATA
                    000 - 03 53 44 53 
                    004 - 48 33 32 47 
                    008 - 80 97 e5 4d 
                    012 - 40 01 7b 00 
CMD_SEND:3
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0xaaaa0520 
CMD_SEND:9
        ARG          0xaaaa0000
        MMC_RSP_R2       0x400e0032 
                         0xdb790000 
                         0xedc87f80 
                         0x0a404000 
                    DUMPING DATA
                    000 - 40 0e 00 32 
                    004 - db 79 00 00 
                    008 - ed c8 7f 80 
                    012 - 0a 40 40 00 
CMD_SEND:7
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000700 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:51
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00000002
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x80fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:13
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
switch to partitions #0, OK
mmc1 is current device
CMD_SEND:0
        ARG          0x00000000
        MMC_RSP_NONE
CMD_SEND:8
        ARG          0x000001aa
        MMC_RSP_R1,5,6,7     0x000001aa 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x40ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0x40ff8000 
CMD_SEND:55
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000120 
CMD_SEND:41
        ARG          0x41300000
        MMC_RSP_R3,4         0xc0ff8000 
CMD_SEND:2
        ARG          0x00000000
        MMC_RSP_R2       0x03534453 
                         0x48333247 
                         0x8097e54d 
                         0x40017b00 
                    DUMPING DATA
                    000 - 03 53 44 53 
                    004 - 48 33 32 47 
                    008 - 80 97 e5 4d 
                    012 - 40 01 7b 00 
CMD_SEND:3
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0xaaaa0520 
CMD_SEND:9
        ARG          0xaaaa0000
        MMC_RSP_R2       0x400e0032 
                         0xdb790000 
                         0xedc87f80 
                         0x0a404000 
                    DUMPING DATA
                    000 - 40 0e 00 32 
                    004 - db 79 00 00 
                    008 - ed c8 7f 80 
                    012 - 0a 40 40 00 
CMD_SEND:7
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000700 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:51
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x00000002
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:6
        ARG          0x80fffff1
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:55
        ARG          0xaaaa0000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:13
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000920 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00004000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00000000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:17
        ARG          0x00004000
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:16
        ARG          0x00000200
        MMC_RSP_R1,5,6,7     0x00000900 
CMD_SEND:18
        ARG          0x00004154
        RET          -70
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
CMD_SEND:16
        ARG          0x00000200
        RET          -110
WARN: Cannot load the DT ```
tq-schifferm commented 1 week ago

Hi, the issue was fixed in 19e90b5a4a36c6cd31c9e951f1ae0fdcfbea29c7, so updating to the kirkstone.TQMa8.BSP.SW.0092 release (or the current version of the TQMa8-v2020.04_imx_5.4.70_2.3.0 U-Boot branch if you're not using our Yocto BSP) should help.

(Edit: fixed commit reference, accidentally got the TQMa8MPxL instead of TQMa8MxML version of the fix at first)

wico-frad commented 6 days ago

Hi, the issue was fixed in 19e90b5, so updating to the kirkstone.TQMa8.BSP.SW.0092 release (or the current version of the TQMa8-v2020.04_imx_5.4.70_2.3.0 U-Boot branch if you're not using our Yocto BSP) should help.

(Edit: fixed commit reference, accidentally got the TQMa8MPxL instead of TQMa8MxML version of the fix at first)

Thank you for your prompt reply.

Currently, we are working with the kirkstone.TQMa8MxML.BSP.SW.0091 release (current branch: TQMa8-v2020.04_imx_5.4.70_2.3.0) . We’ve tried cherry-picking the commit 19e90b5, but the issue persists when testing on the hardware.

Any suggestions would be appreciated.

tq-schifferm commented 6 days ago

Hmm, that is unfortunate. The errors we were seeing (also with certain SanDisk SD cards) looked extremely similar, with the card stopping to respond to certain commands after an incomplete reset, resulting in error 110 (ETIMEDOUT) - although I don't remember the error 70 (ECOMM) happening here. However, I think that our issue happened earlier in the boot - as the reset in the SPL was to short, loading the U-Boot proper failed, instead of U-Boot failing to load the OS.

Some things to try:

tq-niebelm commented 6 days ago

We had a similar sounding problem - U-Boot came up but U-Boot proper could nor read from SD-Card. But the error was seen only on one board for exactly one card type. It was blamed on this board. At this time we picked some patches that correct several issues in the USDHC controller driver and as far as I can tell the problem was solved on this board, too.

I rebased and pushed the patches to branch `testing/v2020.04-imx-sd-card-fixes.

wico-frad commented 1 day ago

Hmm, that is unfortunate. The errors we were seeing (also with certain SanDisk SD cards) looked extremely similar, with the card stopping to respond to certain commands after an incomplete reset, resulting in error 110 (ETIMEDOUT) - although I don't remember the error 70 (ECOMM) happening here. However, I think that our issue happened earlier in the boot - as the reset in the SPL was to short, loading the U-Boot proper failed, instead of U-Boot failing to load the OS.

Some things to try:

* Interrupt the boot in the U-Boot command line, remove and reinsert the SD card to power cycle it, continue boot

* U-Boot proper will handle the SD card reset in the real MMC driver, which is the `udelay(2000)` in `mmc_power_cycle` in `drivers/mmc/mmc.c`. Try increasing the delay, maybe to 10000 as as test.

* Check `mmc dev 1; mmc info` for the used mode. The card will likely show a UHS mode like SDR104. You can try disabling `CONFIG_MMC_UHS_SUPPORT` in the U-Boot config to check if the issue disappears in slower modes.

Thank you for the suggestions. Here's what we've tried so far:

The i.MX8M Mini can support high-speed grades in the USDHC controllers. Turning off the corresponding configuration option for the TQMa8MxML variant in Kconfig before booting the kernel seems to resolve the issue.

Additionally, we tried capping the USDHC controller's capabilities by disabling the sd-uhs-sdr104 flag in imx8mm-mba8mx-u-boot.dtsi. Surprisingly, this also resolves the issue. I'm still curious why limiting the USDHC controller's capabilities in this way helps. From what I understand, the generic MMC driver uses these flags to set the host capabilities. If anyone has insights into why reducing the speed grades seems to improve stability, I would appreciate your input.