i2c kernel panic on A20-OLinuXino-LIME during poweroff

[ralflici]

I'm encountering a problem during every shutdown/reboot on the A20-OLinuXino-LIME board. As far as I can understand, when systemd tries to power down the board, a kernel panic occurs due to a problem on the i2c driver communicating with the power management IC (AXP209).

Here are the systemd messages

[ 1443.648826] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 1443.681457] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 1443.702110] systemd-journald[114]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 1443.710853] systemd-journald[114]: Failed to send WATCHDOG=1 notification message: Connection refused
[ 1443.735901] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 1443.780708] systemd-shutdown[1]: Unmounting file systems.
[ 1443.794921] [1304]: Remounting '/' read-only with options 'n/a'.
[ 1443.822827] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: disabled.
[ 1443.836692] systemd-shutdown[1]: All filesystems unmounted.
[ 1443.843750] systemd-shutdown[1]: Deactivating swaps.
[ 1443.849950] systemd-shutdown[1]: All swaps deactivated.
[ 1443.855240] systemd-shutdown[1]: Detaching loop devices.
[ 1443.861419] systemd-shutdown[1]: All loop devices detached.
[ 1443.867055] systemd-shutdown[1]: Stopping MD devices.
[ 1443.872653] systemd-shutdown[1]: All MD devices stopped.
[ 1443.878012] systemd-shutdown[1]: Detaching DM devices.
[ 1443.883915] systemd-shutdown[1]: All DM devices detached.
[ 1443.890194] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[ 1443.909690] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 1443.917232] systemd-shutdown[1]: Powering off.
[ 1443.961994] reboot: Power down
[ 1443.965153] ------------[ cut here ]------------
[ 1443.969778] WARNING: CPU: 0 PID: 1 at drivers/i2c/i2c-core.h:40 i2c_transfer+0xfc/0x108
[ 1443.977815] No atomic I2C transfer handler for 'i2c-0'
[ 1443.982956] Modules linked in: cfg80211 gpexe_driver(O) fuse autofs4
[ 1443.989344] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: G           O      5.15.35 #1
[ 1443.997268] Hardware name: Allwinner sun7i (A20) Family
[ 1444.002507] [<c010d7c0>] (unwind_backtrace) from [<c0109d9c>] (show_stack+0x10/0x14)
[ 1444.010275] [<c0109d9c>] (show_stack) from [<c0813730>] (dump_stack_lvl+0x40/0x4c)
[ 1444.017866] [<c0813730>] (dump_stack_lvl) from [<c080fde0>] (__warn+0xb8/0x114)
[ 1444.025187] [<c080fde0>] (__warn) from [<c080fed4>] (warn_slowpath_fmt+0x98/0xc0)
[ 1444.032689] [<c080fed4>] (warn_slowpath_fmt) from [<c05ee000>] (i2c_transfer+0xfc/0x108)
[ 1444.040798] [<c05ee000>] (i2c_transfer) from [<c05ee05c>] (i2c_transfer_buffer_flags+0x50/0x78)
[ 1444.049509] [<c05ee05c>] (i2c_transfer_buffer_flags) from [<c05251e4>] (regmap_i2c_write+0x14/0x30)
[ 1444.058579] [<c05251e4>] (regmap_i2c_write) from [<c0521c84>] (_regmap_raw_write_impl+0x600/0x6bc)
[ 1444.067557] [<c0521c84>] (_regmap_raw_write_impl) from [<c0521db8>] (_regmap_bus_raw_write+0x78/0xa0)
[ 1444.076793] [<c0521db8>] (_regmap_bus_raw_write) from [<c05227d8>] (regmap_write+0x3c/0x60)
[ 1444.085161] [<c05227d8>] (regmap_write) from [<c05297e0>] (axp20x_power_off+0x2c/0x38)
[ 1444.093096] [<c05297e0>] (axp20x_power_off) from [<c014c434>] (__do_sys_reboot+0x148/0x244)
[ 1444.101470] [<c014c434>] (__do_sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[ 1444.109577] Exception stack(0xc1047fa8 to 0xc1047ff0)
[ 1444.114638] 7fa0:                   00000000 00000000 fee1dead 28121969 4321fedc 4321fedc
[ 1444.122822] 7fc0: 00000000 00000000 00000000 00000058 00000000 00000000 00000000 00000000
[ 1444.131004] 7fe0: 00000058 be8b8c44 b6d09405 b6c81ae6
[ 1444.136058] ---[ end trace 073d7082bd2319fa ]---
[ 1446.008426] i2c i2c-0: mv64xxx: I2C bus locked, block: 1, time_left: 0
[ 1446.528434] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[ 1446.536104] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: G        W  O      5.15.35 #1
[ 1446.544030] Hardware name: Allwinner sun7i (A20) Family
[ 1446.549253] [<c010d7c0>] (unwind_backtrace) from [<c0109d9c>] (show_stack+0x10/0x14)
[ 1446.557024] [<c0109d9c>] (show_stack) from [<c0813730>] (dump_stack_lvl+0x40/0x4c)
[ 1446.564608] [<c0813730>] (dump_stack_lvl) from [<c080fb44>] (panic+0xf0/0x2d4)
[ 1446.571848] [<c080fb44>] (panic) from [<c012b108>] (do_exit+0x9f0/0x9fc)
[ 1446.578569] [<c012b108>] (do_exit) from [<c014c43c>] (__do_sys_reboot+0x150/0x244)
[ 1446.586159] [<c014c43c>] (__do_sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[ 1446.594266] Exception stack(0xc1047fa8 to 0xc1047ff0)
[ 1446.599326] 7fa0:                   00000000 00000000 fee1dead 28121969 4321fedc 4321fedc
[ 1446.607510] 7fc0: 00000000 00000000 00000000 00000058 00000000 00000000 00000000 00000000
[ 1446.615693] 7fe0: 00000058 be8b8c44 b6d09405 b6c81ae6
[ 1446.620755] Rebooting in 10 seconds..

Since the problem is related to the AXP209, sometimes the system does not reboot at all and freezes on the line containing ---[ end trace 073d7082bd2319fa ]---. \ I tried to communicate with the AXP209 through i2c-tools but was unsuccessful:

root@D810XXX:~# i2cdetect -l
i2c-0   i2c             mv64xxx_i2c adapter                     I2C adapter
i2c-1   i2c             mv64xxx_i2c adapter                     I2C adapter
root@D810XXX:~# i2cdetect -y 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- UU -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --
root@D810XXX:~# i2cget 0 0x34
Error: Could not set address to 0x34: Device or resource busy

\ Do you know if this is a hw related bug or a kernel bug? Any ideas how to solve the problem?

[nandra]

@ralflici thanks for reaching out. This seems to be related to some kernel changes (like this: https://github.com/torvalds/linux/commit/544a8d75f3d6e60e160cd92dc56321484598a993). I'll check if can find patch for sunxi. You're using kirkstone release right? Can you maybe try master to verify if it works or not? Thanks.

[ralflici]

Thanks for your reply. Yes, I'm using kirkstone but to try master I would need to update the dependencies. I don't know if this is helpful but I found out that only the poweroff command triggers the kernel panic while the reboot does not.

[nandra]

@ralflici I don't have your board but maybe you can backport https://github.com/torvalds/linux/commit/544a8d75f3d6e60e160cd92dc56321484598a993 to kirkstone and try if it will be easier for you maybe? Thanks.

[ralflici]

I just tried the patch and, although I god rid of that kernel panic, a different one appeared:

[  145.711947] reboot: Power down
[  146.228466] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
[  146.236155] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 5.15.35 #1
[  146.242692] Hardware name: Allwinner sun7i (A20) Family
[  146.247930] [<c010d7c0>] (unwind_backtrace) from [<c0109d9c>] (show_stack+0x10/0x14)
[  146.255707] [<c0109d9c>] (show_stack) from [<c0813800>] (dump_stack_lvl+0x40/0x4c)
[  146.263294] [<c0813800>] (dump_stack_lvl) from [<c080fc14>] (panic+0xf0/0x2d4)
[  146.270536] [<c080fc14>] (panic) from [<c012b108>] (do_exit+0x9f0/0x9fc)
[  146.277257] [<c012b108>] (do_exit) from [<c014c43c>] (__do_sys_reboot+0x150/0x244)
[  146.284849] [<c014c43c>] (__do_sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[  146.292956] Exception stack(0xc1047fa8 to 0xc1047ff0)
[  146.298016] 7fa0:                   00000000 00000000 fee1dead 28121969 4321fedc 4321fedc
[  146.306199] 7fc0: 00000000 00000000 00000000 00000058 00000000 00000000 00000000 00000000
[  146.314380] 7fe0: 00000058 bedd4c44 b6cd9405 b6c51ae6
[  146.319441] Rebooting in 10 seconds..

Anyway I found this forum post which suggested to apply both torvalds/linux@544a8d75f3d6e60e160cd92dc56321484598a993 and torvalds/linux@09b343038e3470e4d0da45f0ee09fb42107e5314 and it actually solved the problem. Now I don't get kernel panic on poweroff anymore.\ Thanks for your help.

[nandra]

@ralflici can you please post pull request with this change for kirkstone, so other can benefit from that fix also. Thanks.

[nandra]

@ralflici I've pushed those patches to kirkstone branch. Closing an issue.