search

JeffyCN / meta-rockchip

Yocto BSP layer for the Rockchip SOC boards
Other
129 stars 96 forks source link

RK3399 & RK3568 reset/reboot every 10min #91

Closed chrisiberg closed 10 months ago

chrisiberg commented 1 year ago

I have an iot-3399e board with Kernel 4.19 the board is rebooting after about 10min, the CPU Temp is at ~65C. The DTS is from a running ubuntu image with the same kernel version. There is no error on the serial console, just rebooting without reason. Does anybody have a hint for me where to start?

chrisiberg commented 12 months ago

Sorry, the original image kernel is 4.4 I use 4.19, I'll give it a try.

chrisiberg commented 12 months ago

same with kernel 4.4, reboot after 10min.

JeffyCN commented 12 months ago

you csn try to use the working loader & kernel with this yocto's rootfs.

if there's no restart log in serial console, likely something wrong in the kernel/hardware side

chrisiberg commented 11 months ago

I have the same problem with a iot-3568A now (same company)? If I only exchange the rootfs, "everything" is working. I tried to disable u-boot watchdog, even if I think the timeout must be shorter.

The restart ist initiated on u-boot level, if I stop u-boot with SCTRG+C it does the restart after 10 minutes too.

How would it behave if the trusted firmware is wrong, would it even boot, or is there a time limitation? Is 600sec there a number? Is there even a company difference or is for every rk3568 the same trusted firmware?

What can I enable in u-boot or kernel to se why the reboot is initiated, tried many things already but no usable output in the console.

What about this message?

TEEC: Waring: Could not find security partition
**lib/avb/libavb_user/avb_ops_user.c: trusty_read_lock_state failed**
Error determining whether device is unlocked.
Device is: UNLOCKED

maybe the log is saying something?

DDR V1.18 f366f69a7d typ 23/07/17-15:48:58
In
LP4/4x derate en, other dram:1x trefi
ddrconfig:0
LP4 MR14:0x4d
LPDDR4, 324MHz
BW=32 Col=10 Bk=8 CS0 Row=17 CS=1 Die BW=16 Size=4096MB
tdqss: cs0 dqs0: 24ps, dqs1: -72ps, dqs2: -48ps, dqs3: -144ps,

change to: 324MHz
clk skew:0x62

change to: 528MHz
clk skew:0x58

change to: 780MHz
clk skew:0x58

change to: 1560MHz(final freq)
PHY drv:clk:38,ca:38,DQ:30,odt:60
vrefinner:16%, vrefout:29%
dram drv:40,odt:80
vref_ca:00000068
clk skew:0x1c
cs 0:
the read training result:
DQS0:0x34, DQS1:0x35, DQS2:0x38, DQS3:0x31,
min  : 0xe  0xf 0x12  0xf  0x3  0x5  0x8  0x3 , 0xa  0x8  0x2  0x2  0xf  0xb  0xd  0x9 ,
      0x13 0x14  0xd  0xa  0x2  0x1  0x3  0x5 , 0xd  0x8  0x9  0x1 0x12 0x10  0xd  0xf ,
mid  :0x2a 0x2b 0x2d 0x2b 0x1e 0x22 0x24 0x1f ,0x28 0x26 0x1f 0x1e 0x2c 0x28 0x2a 0x26 ,
      0x30 0x31 0x2a 0x27 0x20 0x1e 0x20 0x22 ,0x28 0x23 0x24 0x1d 0x2c 0x2b 0x28 0x2b ,
max  :0x47 0x47 0x49 0x47 0x3a 0x3f 0x40 0x3c ,0x46 0x45 0x3c 0x3b 0x4a 0x45 0x47 0x43 ,
      0x4d 0x4e 0x47 0x45 0x3f 0x3c 0x3e 0x3f ,0x44 0x3f 0x3f 0x39 0x47 0x47 0x43 0x47 ,
range:0x39 0x38 0x37 0x38 0x37 0x3a 0x38 0x39 ,0x3c 0x3d 0x3a 0x39 0x3b 0x3a 0x3a 0x3a ,
      0x3a 0x3a 0x3a 0x3b 0x3d 0x3b 0x3b 0x3a ,0x37 0x37 0x36 0x38 0x35 0x37 0x36 0x38 ,
the write training result:
DQS0:0x20, DQS1:0xe, DQS2:0x13, DQS3:0x0,
min  :0x68 0x6d 0x6f 0x6a 0x5c 0x60 0x65 0x65 0x62 ,0x52 0x4f 0x4b 0x4a 0x57 0x53 0x56 0x54 0x4d ,
      0x5e 0x5f 0x59 0x56 0x50 0x4c 0x4f 0x54 0x54 ,0x4a 0x4a 0x4a 0x43 0x51 0x50 0x4d 0x53 0x45 ,
mid  :0x85 0x89 0x8b 0x85 0x77 0x7b 0x80 0x7e 0x7d ,0x6f 0x6c 0x65 0x64 0x72 0x6e 0x71 0x6d 0x68 ,
      0x7b 0x7b 0x74 0x72 0x6c 0x68 0x6b 0x6e 0x70 ,0x67 0x66 0x63 0x5d 0x6b 0x6b 0x67 0x6e 0x60 ,
max  :0xa3 0xa5 0xa7 0xa1 0x93 0x97 0x9b 0x98 0x98 ,0x8c 0x89 0x7f 0x7f 0x8d 0x89 0x8c 0x87 0x84 ,
      0x99 0x98 0x8f 0x8e 0x88 0x84 0x87 0x88 0x8c ,0x85 0x82 0x7c 0x78 0x86 0x87 0x82 0x89 0x7b ,
range:0x3b 0x38 0x38 0x37 0x37 0x37 0x36 0x33 0x36 ,0x3a 0x3a 0x34 0x35 0x36 0x36 0x36 0x33 0x37 ,
      0x3b 0x39 0x36 0x38 0x38 0x38 0x38 0x34 0x38 ,0x3b 0x38 0x32 0x35 0x35 0x37 0x35 0x36 0x36 ,
CA Training result:
cs:0 min  :0x43 0x38 0x37 0x2c 0x38 0x29 0x3a ,0x42 0x35 0x36 0x29 0x36 0x2a 0x3a ,
cs:0 mid  :0x7d 0x7b 0x6f 0x6f 0x71 0x6a 0x65 ,0x7c 0x78 0x70 0x6b 0x70 0x6c 0x65 ,
cs:0 max  :0xb7 0xbe 0xa8 0xb2 0xab 0xab 0x90 ,0xb7 0xbc 0xaa 0xad 0xaa 0xae 0x90 ,
cs:0 range:0x74 0x86 0x71 0x86 0x73 0x82 0x56 ,0x75 0x87 0x74 0x84 0x74 0x84 0x56 ,
out
U-Boot SPL board init
U-Boot SPL 2017.09-gaaca6ffec1-211203 #zzz (Dec 03 2021 - 18:42:16)
unknown raw ID phN
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
MMC error: The cmd index is 1, ret is -110
Card did not respond to voltage select!
mmc_init: -95, time 9
spl: mmc init failed with error: -95
Trying to boot from MMC1
No misc partition
Trying fit image at 0x4000 sector
## Verified-boot: 0
## Checking atf-1 0x00040000 ... sha256(0d5225a4ab...) + OK
## Checking uboot 0x00a00000 ... sha256(93251e08d9...) + OK
## Checking fdt 0x00b3f2c8 ... sha256(4ba618a9e8...) + OK
## Checking atf-2 0xfdcc1000 ... sha256(3e94d16e6a...) + OK
## Checking atf-3 0x0006b000 ... sha256(fde0ef262b...) + OK
## Checking atf-4 0xfdcce000 ... sha256(c9eb312bf2...) + OK
## Checking atf-5 0xfdcd0000 ... sha256(befba422b8...) + OK
## Checking atf-6 0x00069000 ... sha256(6ede7a3b44...) + OK
## Checking optee 0x08400000 ... sha256(4fcbcd3870...) + OK
Jumping to U-Boot(0x00a00000) via ARM Trusted Firmware(0x00040000)
Total: 236.428 ms

INFO:    Preloader serial: 2
NOTICE:  BL31: v2.3():v2.3-607-gbf602aff1:cl
NOTICE:  BL31: Built : 10:16:03, Jun  5 2023
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    pmu v1 is valid 220114
INFO:    dfs DDR fsp_param[0].freq_mhz= 1560MHz
INFO:    dfs DDR fsp_param[1].freq_mhz= 324MHz
INFO:    dfs DDR fsp_param[2].freq_mhz= 528MHz
INFO:    dfs DDR fsp_param[3].freq_mhz= 780MHz
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 0
INFO:    BL31: Initializing runtime services
INFO:    BL31: Initializing BL32
I/TC:
I/TC: OP-TEE version: 3.13.0-723-gdcfdd61d0 #hisping.lin (gcc version 10.2.1 20201103 (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16))) #2 Wed Jun  7 09:43:57 CST 2023 aarch64
I/TC: Primary CPU initializing
I/TC: Primary CPU switching to normal world boot
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0xa00000
INFO:    SPSR = 0x3c9

U-Boot 2017.09 (Nov 08 2023 - 07:32:24 +0000)

Model: Rockchip RK3568 Evaluation Board
MPIDR: 0x81000000
PreSerial: 2, raw, 0xfe660000
DRAM:  4 GiB
Sysmem: init
Relocation Offset: ed220000
Relocation fdt: eb9f8fa8 - eb9fece0
CR: M/C/I
DM: v1
dwmmc@fe2b0000: 1, dwmmc@fe2c0000: 2, sdhci@fe310000: 0
Bootdev(atags): mmc 0
MMC0: HS200, 200Mhz
PartType: EFI
No misc partition
boot mode: None
RESC: 'boot', blk@0x0000b208
optee api revision: 2.0
TEEC: Waring: Could not find security partition
lib/avb/libavb_user/avb_ops_user.c: trusty_read_lock_state failed
Error determining whether device is unlocked.
Device is: UNLOCKED
DTB: rk-kernel.dtb (136639)
HASH(c): OK
I2c0 speed: 100000Hz
vsel-gpios- not found! Error: -2
vdd_cpu init 900000 uV
PMIC:  RK8090 (on=0x40, off=0x00)
vdd_logic init 900000 uV
vdd_gpu init 900000 uV
vdd_npu init 900000 uV
io-domain: OK
INFO:    ddr dmc_fsp already initialized in loader.
Could not find baseparameter partition
Model: RK3568-ECO2
MPIDR: 0x81000000
Rockchip UBOOT DRM driver version: v1.0.1
VOP have 2 active VP
vp0 have layer nr:3[0 2 4 ], primary plane: 4
vp1 have layer nr:3[1 3 5 ], primary plane: 5
vp2 have layer nr:0[], primary plane: 0
xfer: num: 2, addr: 0x50
xfer: num: 2, addr: 0x50
Monitor has basic audio support
Could not find baseparameter partition
hdmi@fe0a0000:  detailed mode clock 27000 kHz, flags[a]
    H: 0720 0736 0798 0858
    V: 0480 0489 0495 0525
bus_format: 2025
VOP update mode to: 720x480p60, type: HDMI0 for VP0
VP0 set crtc_clock to 27000KHz
VOP VP0 enable Smart0[654x270->654x270@33x105] fmt[2] addr[0xedf04000]
CEA mode used vic=2
final pixclk = 27000000 tmdsclk = 27000000
PHY powered down in 0 iterations
PHY PLL locked 1 iterations
PHY powered down in 1 iterations
PHY PLL locked 1 iterations
sink has audio support
hdmi_set_clk_regenerator: fs=48000Hz ftdms=27.000MHz N=6144 cts=27000
CLK: (sync kernel. arm: enter 816000 KHz, init 816000 KHz, kernel 0N/A)
  apll 1416000 KHz
  dpll 780000 KHz
  gpll 1188000 KHz
  cpll 1000000 KHz
  npll 1200000 KHz
  vpll 24000 KHz
  hpll 27000 KHz
  ppll 200000 KHz
  armclk 1416000 KHz
  aclk_bus 150000 KHz
  pclk_bus 100000 KHz
  aclk_top_high 500000 KHz
  aclk_top_low 400000 KHz
  hclk_top 150000 KHz
  pclk_top 100000 KHz
  aclk_perimid 300000 KHz
  hclk_perimid 150000 KHz
  pclk_pmu 100000 KHz
Net:   eth1: ethernet@fe010000
Hit key to stop autoboot('CTRL+C'):  0
framr=1
already set
Could not find misc partition
ANDROID: reboot reason: "(none)"
Not AVB images, AVB skip
ANDROID: Hash OK
Booting LZ4 kernel at 0x04080000(Uncompress to 0x00280000) with fdt at 0x08300000...

Fdt Ramdisk skip relocation
No misc partition
## Booting Android Image at 0x0407f800 ...
Kernel: 0x00280000 - 0x00cc0331 (10497 KiB)
## Flattened Device Tree blob at 0x08300000
   Booting using the fdt blob at 0x08300000
   Uncompressing LZ4 Kernel Image from 0x04080000 to 0x00280000 ... with 0157b008 bytes OK
   kernel loaded at 0x00280000, end = 0x017fb008
   Using Device Tree in place at 0000000008300000, end 00000000083245be
vp0, plane_mask:0x15, primary-id:4, curser-id:0
vp1, plane_mask:0x2a, primary-id:5, curser-id:1
vp2, plane_mask:0x0, primary-id:0, curser-id:-1
WARNING: could not set reg FDT_ERR_BADOFFSET.
## reserved-memory:
  drm-logo@00000000: addr=edf00000 size=b8000
  drm-cubic-lut@00000000: addr=eff00000 size=8000
  ramoops@110000: addr=110000 size=f0000
Adding bank: 0x00200000 - 0x08400000 (size: 0x08200000)
Adding bank: 0x09400000 - 0xf0000000 (size: 0xe6c00000)
Adding bank: 0x1f0000000 - 0x200000000 (size: 0x10000000)
Total: 736.800/779.549 ms

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
chrisiberg commented 10 months ago

I have identified a solution; there is a chip on all SMDT boards referred as MCU in the device tree. This chip is connected to the I2C bus and needs to be managed as a watchdog. Initially, I attempted desoldering the VDD of this chip, which halted the 10-minute reset. However, modifying the hardware for each board was not a feasible solution for me. Instead, I used a logic analyzer to read out the init and watchdog strings, and fortunately, this approach proved successful. Now, I employ an init script along with a watchdog script that triggers every 10 seconds.

robinZhao commented 4 weeks ago

I have identified a solution; there is a chip on all SMDT boards referred as MCU in the device tree. This chip is connected to the I2C bus and needs to be managed as a watchdog. Initially, I attempted desoldering the VDD of this chip, which halted the 10-minute reset. However, modifying the hardware for each board was not a feasible solution for me. Instead, I used a logic analyzer to read out the init and watchdog strings, and fortunately, this approach proved successful. Now, I employ an init script along with a watchdog script that triggers every 10 seconds.

can you share the watchdog script?