Poweroff command is Fail on M5CoreMP135.

nnn112358 commented 3 months ago

On M5 CoreMP135, the "poweroff" command does not shutdown properly. After some time, a reboot occurs. The "reboot" command works correctly. There is no difference between buildroot and Debian, and the same phenomenon occurs in the latest commit of buildroot on github.

By @MeemeeLab , He tells us that he suspects it is because the CPU does not have a shutdown function and the AXP2101 genpd (power management) driver is not available. He has implemented a shutdown function, Linux kernel 6.9 oriented. https://github.com/MeemeeLab/linux/commit/184ad91ecfe37568e7765ab2d1fbe33242efe4cb However, this implementation has compilation problems with Linux kernel 5.15.

Linux Log(USB serial connected to CoreMP135 Port.C, Log in root)


$ poweroff
Stopping /etc/rc.localOK
Stopping dropbear sshd: OK
Stopping dhcpcd...
stopped /sbin/dhcpcd (pid 177)
Stopping network: OK
Stopping tee-supplicant: FAIL
Seeding 256 bits and crediting
Saving 256 bits of creditable seed for next boot
Stopping mdev... stopped process in pidfile '/var/run/mdev.pid' (pid 138)
OK
Stopping modules: rmmod: can't unload module 'soundcore': Resource temporarily unavailable
rmmod: can't unload module 'snd': Resource temporarily unavailable
rmmod: can't unload module 'snd_timer': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm_dmaengine': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_core': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_simple_card_utils': Resource temporarily unavailable
rmmod: can't unload module 'libcomposite': Resource temporarily unavailable
OK
Stopping klogd: OK
Stopping syslogd: OK
umount: devtmpfs busy - remounted read-only
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[  235.243319] reboot: Power down
E/TC:0   Panic at core/arch/arm/plat-stm32mp1/pm/low_power.c:405 <stm32_enter_cstop_shutdown>
E/TC:0   TEE load address @ 0xde200000
E/TC:0   Call stack:
E/TC:0    0xde203f29
E/TC:0    0xde21c0f9
E/TC:0    0xde20a5bb
E/TC:0    0xde20af21
E/TC:0    0xde23e791
E/TC:0    0xde20b233
E/TC:0    0xde20b0dc

[ 0.004017] /cpus/cpu@0 missing clock-frequency property [ 0.435184] stm32-cpufreq stm32-cpufreq: OPP-v2 not supported Starting syslogd: OK Starting klogd: OK Starting modules: soundcore success, snd success, snd_timer success, snd_pcm success, snd_pcm_dmaengine success, snd_soc_core success, snd_soc_simple_card_utils success, snd_soc_audio_graph_card success, snd-soc-simple-card success, snd_soc_stm32_sai success, snd_soc_stm32_sai_sub success, cdc-acm success, libcomposite success, g_serial success, OK Running sysctl: OK Starting mdev... OK Seeding 256 bits and crediting Saving 256 bits of creditable seed for next boot Starting tee-supplicant: Using device /dev/teepriv0. OK Starting network: OK Starting dhcpcd... dhcpcd-9.4.1 starting DUID 00:01:00:01:2d:fa:18:62:ca:7e:a7:e6:2f:fb forked to background, child pid 1[ 5.408396] m_can_platform 4400e000.can can0: bit-timing not yet defined 77 [ 5.421874] m_can_platform 4400e000.can can0: failed to open can device [ 5.436986] m_can_platform 4400f000.can can1: bit-timing not yet defined [ 5.454812] m_can_platform 4400f000.can can1: failed to open can device Starting dropbear sshd: OK Stopping /etc/rc.localOK

Welcome to CoreMP135, Powered by M5Stack. CoreMP135 login:



Reference:
https://github.com/MeemeeLab/linux/tree/m5stack-coremp135

MeemeeLab commented 3 months ago

For reference, panic happens here: https://github.com/STMicroelectronics/optee_os/blob/4bbca1689f6854f660a80327a0d2959537bea222/core/arch/arm/plat-stm32mp1/pm/low_power.c#L405

I'm guessing that this CPU does not have shutdown mode, though STM32_PM_SHUTDOWN exists and I don't really know what it means.

MeemeeLab commented 3 months ago

By the way, I might be able to backport backlight/poweroff driver to 5.15, though it may require fairly large refactoring, like i2c-mfd to use multiple modules for single i2c peripheral: https://github.com/MeemeeLab/linux/commit/7347843c347edc74598502989142b031dec17cc1

I'm not sure M5 will merge PR this big, so I'm not gonna try for now. I feel like patch based modification isn't suitable for this multiple changes, I prefer having linux repository and reference them in defconfig.

dianjixz commented 3 months ago

I have also found this bug. I will fix it later, and it needs to wait to be added to my work schedule list.

dianjixz commented 3 months ago

@MeemeeLab Yes, this is a considerable amount of work, so it was decided during the previous project discussion not to allocate time for the driver of ax2101. If you would like to write a driver for it, I would be happy to add its patch to the m5stack project repository.

MeemeeLab commented 3 months ago

@dianjixz I think I can backport drivers I wrote. Should I try from coremp135_5_15 or coremp135? which one is most widely used one?

dianjixz commented 3 months ago

@MeemeeLab CoreMP135 mainly uses the Linux 6.1 kernel, which is not currently used as the main kernel for building. All current image builds are based on the Linux 5.15 kernel of CoreMP135_5_15. The main reason is that some drivers do not have versions for Linux 6.1. In addition, I have tried to port axp2101-poweroff to kernel 5.15. It reported a kernel error when executing poweroff. During the shutdown phase, the i2c driver gets unloaded, and the kernel can no longer use the i2c driver to write to the AXP2101_REG_COMM_CFG register. I may try to perform the poweroff operation in OP-TEE.

root@M5Core135:~# poweroff
root@M5Core135:~# [  195.233350] reboot: Power down
[  195.235098] ------------[ cut here ]------------
[  195.239706] WARNING: CPU: 0 PID: 1 at drivers/i2c/i2c-core.h:40 i2c_transfer+0xfc/0x138
[  195.247700] No atomic I2C transfer handler for 'i2c-0'
[  195.252813] Modules linked in:
[  195.255898] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 5.15.118-g97e408edb6d7-dirty #6
[  195.264273] Hardware name: STM32 (Device Tree Support)
[  195.269395] [<c0110b84>] (unwind_backtrace) from [<c010c538>] (show_stack+0x10/0x14)
[  195.277177] [<c010c538>] (show_stack) from [<c0c74d74>] (dump_stack_lvl+0x40/0x4c)
[  195.284750] [<c0c74d74>] (dump_stack_lvl) from [<c0125b20>] (__warn+0x88/0x128)
[  195.292019] [<c0125b20>] (__warn) from [<c0c6f24c>] (warn_slowpath_fmt+0x78/0xa8)
[  195.299487] [<c0c6f24c>] (warn_slowpath_fmt) from [<c09be3f0>] (i2c_transfer+0xfc/0x138)
[  195.307566] [<c09be3f0>] (i2c_transfer) from [<c09be47c>] (i2c_transfer_buffer_flags+0x50/0x84)
[  195.316355] [<c09be47c>] (i2c_transfer_buffer_flags) from [<c07ae41c>] (regmap_i2c_write+0x14/0x30)
[  195.325352] [<c07ae41c>] (regmap_i2c_write) from [<c07a8dc4>] (_regmap_raw_write_impl+0x630/0x930)
[  195.334348] [<c07a8dc4>] (_regmap_raw_write_impl) from [<c07a913c>] (_regmap_bus_raw_write+0x78/0xa0)
[  195.343548] [<c07a913c>] (_regmap_bus_raw_write) from [<c07a848c>] (_regmap_write+0x4c/0x158)
[  195.352033] [<c07a848c>] (_regmap_write) from [<c07aa060>] (regmap_write+0x3c/0x60)
[  195.359704] [<c07aa060>] (regmap_write) from [<c09d4378>] (axp2101_poweroff_do_poweroff+0x24/0x60)
[  195.368704] [<c09d4378>] (axp2101_poweroff_do_poweroff) from [<c014fda8>] (__do_sys_reboot+0x15c/0x244)
[  195.378111] [<c014fda8>] (__do_sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[  195.386187] Exception stack(0xc1899fa8 to 0xc1899ff0)
[  195.391202] 9fa0:                   00000000 00000000 fee1dead 28121969 4321fedc 00000000
[  195.399373] 9fc0: 00000000 00000000 00000003 00000058 be82ce74 00000000 be82cbcc 4321fedc
[  195.407641] 9fe0: 00000058 be82cb1c b6cf1b95 b6c61616
[  195.412651] ---[ end trace 6811842759608fc4 ]---
[  195.417800] 8<--- cut here ---
[  195.420346] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[  195.428523] pgd = f3336c55
[  195.431198] [00000000] *pgd=00000000
[  195.434619] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[  195.439943] Modules linked in:
[  195.442923] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: G        W         5.15.118-g97e408edb6d7-dirty #6
[  195.452824] Hardware name: STM32 (Device Tree Support)
[  195.457835] PC is at mutex_lock+0x14/0x3c
[  195.461842] LR is at optee_invoke_func+0x44/0x350
[  195.466559] pc : [<c0c84b4c>]    lr : [<c0aa83d8>]    psr: 60000013
[  195.472790] sp : c1899b40  ip : 00000000  fp : 00000000
[  195.478002] r10: c1dc7244  r9 : c1899ba0  r8 : 00000001
[  195.483314] r7 : c1899b88  r6 : c1dbd8c0  r5 : 00000000  r4 : 00000001
[  195.489849] r3 : c18a4000  r2 : c1899ba0  r1 : c1899b88  r0 : 00000000
[  195.496385] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  195.503534] Control: 10c5387d  Table: c39f806a  DAC: 00000051
[  195.509253] Register r0 information: NULL pointer
[  195.513862] Register r1 information: non-slab/vmalloc memory
[  195.522271] Register r2 information: non-slab/vmalloc memory
[  195.529227] Register r3 information: slab task_struct start c18a4000 pointer offset 0
[  195.538417] Register r4 information: non-paged memory
[  195.544715] Register r5 information: NULL pointer
[  195.550584] Register r6 information: slab kmalloc-64 start c1dbd8c0 pointer offset 0 size 64
[  195.560321] Register r7 information: non-slab/vmalloc memory
[  195.567174] Register r8 information: non-paged memory
[  195.573372] Register r9 information: non-slab/vmalloc memory
[  195.580146] Register r10 information: slab kmalloc-128 start c1dc7200 pointer offset 68 size 128
[  195.590107] Register r11 information: NULL pointer
[  195.595979] Register r12 information: NULL pointer
[  195.601793] Process systemd-shutdow (pid: 1, stack limit = 0xf77d10d9)
[  195.609426] Stack: (0xc1899b40 to 0xc189a000)
[  195.614813] 9b40: 00000000 c18a4080 c18bde80 00000001 c0aa8394 c1f98640 ddac9000 00000001
[  195.624067] 9b60: 00000000 c1dc7244 c19ab410 c0aa3728 00000000 ddac9000 c19c2c48 c1dc7240
[  195.633303] 9b80: c1dc71c0 c0a537f4 00000001 00000006 00000000 00000000 00000000 00000001
[  195.642527] 9ba0: 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  195.651722] 9bc0: 20000013 a57a0adc c1f98640 c1dc71c0 c1884074 c1310500 c19c2c48 0000024a
[  195.660906] 9be0: 00000000 c0a4b4d8 7f57f880 00000000 00000008 c0a4e5a8 00000007 c19ab410
[  195.670080] 9c00: c2871180 00000001 0000002d 00000000 c1f9d9d4 00000017 00000001 c19c2c48
[  195.679230] 9c20: c2871180 00000001 0000002d c0a4e5a8 c1899c38 c0c8966c c1f98640 a57a0adc
[  195.688352] 9c40: c1fa29c0 c1fa29c0 00000000 c130eca0 c2893d40 c069be50 c1fcb9c0 c19c2e10
[  195.697457] 9c60: c19c9410 c2893d40 c19c2c48 c069bdfc c1fcb9c0 c19c2e10 c19c9410 c2893d40
[  195.706562] 9c80: c19c2c48 c069c0f8 c28b0440 c09c7d8c c19c9410 c19c2e10 00000000 c1318530
[  195.715654] 9ca0: c19c2c48 c07a0af8 40000013 c19d8800 c19d7810 7f4baa25 c1204d00 c19c9410
[  195.724752] 9cc0: c07a0a44 00000000 c19c9494 00000001 00000000 00000001 c1204d00 c0794e54
[  195.733874] 9ce0: c19c9410 c19c3410 c07a0a44 c1898000 c19c9494 c19c3494 00000000 c0794f80
[  195.742985] 9d00: c19c9410 c19c3410 c07a0a44 c0795588 00000001 c01825d8 c1899d3c c13387d0
[  195.752069] 9d20: 00000009 a57a0adc 60000093 60000093 00000004 c19c9494 00000000 00000001
[  195.761188] 9d40: c1899dc0 c07957e4 c28b0440 00000000 c19c9410 c09c8560 c28b0440 00000000
[  195.770323] 9d60: 00000000 c1899dc0 00000001 c1203d00 ffffd712 c09bdf10 c1365744 c0fc6f88
[  195.779477] 9d80: c1899da8 c28b0440 00000001 c1899dc0 c130f0c0 c286de00 00000001 c2873e01
[  195.788612] 9da0: 00000001 c09be390 c2873340 00000000 00000002 c2873e01 00000010 c09be47c
[  195.797732] 9dc0: 00000034 00000002 c2873e00 a57a0adc 00000002 c07ae41c c286de00 c07a8dc4
[  195.806856] 9de0: 00000000 00000000 4321fedc a57a0adc 00000000 c2873e00 00000001 00000000
[  195.815992] 9e00: 00000012 00000010 4321fedc c286de00 00000010 00000001 c07a4840 c286de00
[  195.825124] 9e20: c1898000 00000058 4321fedc c07a913c 00000000 00000000 c120e9c8 c286de00
[  195.834238] 9e40: 00000010 00000010 00000001 c07a848c c286de00 00000010 00000001 c120e9c8
[  195.843375] 9e60: fee1dead c1898000 00000058 c07aa060 4321fedc 00000000 00000000 c09d4378
[  195.852521] 9e80: 4321fedc c014fda8 00000000 00000000 00000000 00000000 00000000 a57a0adc
[  195.861684] 9ea0: c3c746c0 00000000 c3969480 c1899ef8 c1899f78 00000000 00000000 00000092
[  195.870837] 9ec0: b6ba6ad4 c03143bc 00000027 c3969480 c1899ef8 c1899f78 00000000 00000000
[  195.879979] 9ee0: 00000092 c0314598 c1899ef4 c1899ef8 00000000 00000000 00010000 00000000
[  195.889124] 9f00: 00000000 c1899f3c 00000000 00000000 be82c914 be82c210 00000004 be82cf68
[  195.898299] 9f20: 00000010 be82c224 00000005 be82c2dc 0000000d b6f7a900 00000001 00000000
[  195.907518] 9f40: 00000001 c1899fb0 00000005 be82c224 be82c210 a57a0adc be82c24c 00000005
[  195.916734] 9f60: c3969480 c3969480 00000000 c0314734 00000000 c013843c 00000000 00000000
[  195.925955] 9f80: 7fe3c0fe a57a0adc 7fe3c0fe a57a0adc 00000000 00000000 00000003 00000058
[  195.935205] 9fa0: c01002a4 c0100060 00000000 00000000 fee1dead 28121969 4321fedc 00000000
[  195.944463] 9fc0: 00000000 00000000 00000003 00000058 be82ce74 00000000 be82cbcc 4321fedc
[  195.953722] 9fe0: 00000058 be82cb1c b6cf1b95 b6c61616 60000030 fee1dead 00000000 00000000
[  195.963020] [<c0c84b4c>] (mutex_lock) from [<c0aa83d8>] (optee_invoke_func+0x44/0x350)
[  195.972090] [<c0aa83d8>] (optee_invoke_func) from [<c0aa3728>] (tee_client_invoke_func+0x7c/0x94)
[  195.982168] [<c0aa3728>] (tee_client_invoke_func) from [<c0a537f4>] (optee_send_message+0x168/0x1f0)
[  195.992565] [<c0a537f4>] (optee_send_message) from [<c0a4b4d8>] (do_xfer+0xb0/0x444)
[  196.001604] [<c0a4b4d8>] (do_xfer) from [<c0a4e5a8>] (scmi_clock_config_set+0x6c/0xb0)
[  196.010750] [<c0a4e5a8>] (scmi_clock_config_set) from [<c069be50>] (clk_core_prepare+0x8c/0x314)
[  196.020826] [<c069be50>] (clk_core_prepare) from [<c069bdfc>] (clk_core_prepare+0x38/0x314)
[  196.030489] [<c069bdfc>] (clk_core_prepare) from [<c069c0f8>] (clk_prepare+0x20/0x30)
[  196.039617] [<c069c0f8>] (clk_prepare) from [<c09c7d8c>] (stm32f7_i2c_runtime_resume+0x3c/0xe4)
[  196.049573] [<c09c7d8c>] (stm32f7_i2c_runtime_resume) from [<c07a0af8>] (genpd_runtime_resume+0xb4/0x2d8)
[  196.060501] [<c07a0af8>] (genpd_runtime_resume) from [<c0794e54>] (__rpm_callback+0x3c/0x108)
[  196.070327] [<c0794e54>] (__rpm_callback) from [<c0794f80>] (rpm_callback+0x60/0x64)
[  196.079435] [<c0794f80>] (rpm_callback) from [<c0795588>] (rpm_resume+0x604/0x82c)
[  196.088346] [<c0795588>] (rpm_resume) from [<c07957e4>] (__pm_runtime_resume+0x34/0x6c)
[  196.097680] [<c07957e4>] (__pm_runtime_resume) from [<c09c8560>] (stm32f7_i2c_xfer+0x38/0x1c4)
[  196.107640] [<c09c8560>] (stm32f7_i2c_xfer) from [<c09bdf10>] (__i2c_transfer.part.0+0xec/0x424)
[  196.117821] [<c09bdf10>] (__i2c_transfer.part.0) from [<c09be390>] (i2c_transfer+0x9c/0x138)
[  196.127712] [<c09be390>] (i2c_transfer) from [<c09be47c>] (i2c_transfer_buffer_flags+0x50/0x84)
[  196.137829] [<c09be47c>] (i2c_transfer_buffer_flags) from [<c07ae41c>] (regmap_i2c_write+0x14/0x30)
[  196.148354] [<c07ae41c>] (regmap_i2c_write) from [<c07a8dc4>] (_regmap_raw_write_impl+0x630/0x930)
[  196.158792] [<c07a8dc4>] (_regmap_raw_write_impl) from [<c07a913c>] (_regmap_bus_raw_write+0x78/0xa0)
[  196.169463] [<c07a913c>] (_regmap_bus_raw_write) from [<c07a848c>] (_regmap_write+0x4c/0x158)
[  196.179426] [<c07a848c>] (_regmap_write) from [<c07aa060>] (regmap_write+0x3c/0x60)
[  196.188575] [<c07aa060>] (regmap_write) from [<c09d4378>] (axp2101_poweroff_do_poweroff+0x24/0x60)
[  196.199059] [<c09d4378>] (axp2101_poweroff_do_poweroff) from [<c014fda8>] (__do_sys_reboot+0x15c/0x244)
[  196.209975] [<c014fda8>] (__do_sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x48)
[  196.219580] Exception stack(0xc1899fa8 to 0xc1899ff0)
[  196.226145] 9fa0:                   00000000 00000000 fee1dead 28121969 4321fedc 00000000
[  196.235891] 9fc0: 00000000 00000000 00000003 00000058 be82ce74 00000000 be82cbcc 4321fedc
[  196.245649] 9fe0: 00000058 be82cb1c b6cf1b95 b6c61616
[  196.252283] Code: e3c33d7f e3c3303f e5933008 f590f000 (e1901f9f) 
[  196.261444] ---[ end trace 6811842759608fc5 ]---
[  196.268059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[  196.277419] Rebooting in 5 seconds..

dianjixz commented 3 months ago

@MeemeeLab This is the relevant code.

// SPDX-License-Identifier: GPL-2.0-or-later
/*
 * Backlight driver for AXP2101, used for M5Stack devices.
 *
 * Copyright 2024 MeemeeLab
 */
#include <linux/module.h>
#include <linux/init.h>
#include <linux/errno.h>
#include <linux/reboot.h>
#include <linux/platform_device.h>
#include <linux/regmap.h>
#include <linux/mod_devicetable.h>
#include <linux/property.h>

#define AXP2101_REG_COMM_CFG 0x10

struct axp2101_poweroff {
    struct device *dev;
    struct regmap *regmap;
};
#define AXP2101_REG_COMM_CFG 0x10

// /* Global variable needed only for axp2101_power_data */
static struct axp2101_poweroff axp2101_power_data;

static void axp2101_poweroff_do_poweroff(void) {
    if(axp2101_power_data.regmap != NULL)
        regmap_write(axp2101_power_data.regmap, AXP2101_REG_COMM_CFG, 1);
    WARN_ONCE(1, "Unable to power off system\n");
}

static int axp2101_probe(struct platform_device *pdev) {

    if (!pdev->dev.parent) {
        dev_err(&pdev->dev, "No parent found!");
        return -ENODEV;
    }

    axp2101_power_data.dev = &pdev->dev;
    axp2101_power_data.regmap = dev_get_regmap(pdev->dev.parent, NULL);
    if (!axp2101_power_data.regmap) {
        dev_err(&pdev->dev, "No regmap found!");
        return -ENODEV;
    }

    pm_power_off = axp2101_poweroff_do_poweroff;

    return 0;
}

static const struct platform_device_id axp2101_id[] = {
    { "axp2101-poweroff", 0 },
    { },
};
MODULE_DEVICE_TABLE(platform, axp2101_id);

static const struct of_device_id axp2101_of_match[] = {
        { .compatible = "x-powers,axp2101-poweroff" },
        { },
};
MODULE_DEVICE_TABLE(of, axp2101_of_match);

static struct platform_driver axp2101_driver = {
    .driver = {
        .name   = KBUILD_MODNAME,
        .of_match_table = axp2101_of_match,
    },
    .probe = axp2101_probe,
    .id_table = axp2101_id,
};

module_platform_driver(axp2101_driver);

MODULE_LICENSE("GPL v2");
MODULE_AUTHOR("MeemeeLab");
MODULE_DESCRIPTION("AXP2101 Poweroff driver");

MeemeeLab commented 3 months ago

@dianjixz I think kernel panic is occurring because of wrong DTS configuration. (probably MFD isn't correctly configured) I'll try backporting my drivers to 6.1 kernel first.

dianjixz commented 3 months ago

ok,i try!

MeemeeLab commented 3 months ago

@dianjixz @nnn112358 I just published PR with backported drivers. Could you test it if you have free time? I'm not sure it will work on different build environment.

dianjixz commented 3 months ago

@dianjixz Ok ,i will.Thank you very much.

nnn112358 commented 3 months ago

@MeemeeLab Thanks for the great job. I did the trial easily.

buildroot (kernel 6.1)

I performed a BuildRoot Linux build on an Ubuntu PC.

git clone https://github.com/m5stack/CoreMP135_buildroot.git
git clone -b upstream-backport https://github.com/MeemeeLab/CoreMP135_buildroot-external-st.git
cd CoreMP135_buildroot
make BR2_EXTERNAL=../CoreMP135_buildroot-external-st/ m5stack_coremp135_defconfig
make -j4

"CoreMP135_buildroot/output/images/sdcard.img" was copied to a microsd, inserted into CoreMP135, and logged into Root. The poweroff command stops in the middle of the command. Is there a problem that is interrupting the process?

And, Since we will be changing fb_ili9341_m5, /etc/rc.local will also need to be changed.

#     #  #######   #####   #######     #      #####   #    #
##   ##  #        #     #     #       # #    #     #  #   #
# # # #  #        #           #      #   #   #        #  #
#  #  #  ######    #####      #     #     #  #        ###
#     #        #        #     #     #######  #        #  #
#     #  #     #  #     #     #     #     #  #     #  #   #
#     #   #####    #####      #     #     #   #####   #    #
Welcome to CoreMP135, Powered by M5Stack.
CoreMP135 login: root

# poweroff
# Stopping /etc/rc.localOK
Stopping dropbear sshd: OK
Stopping dhcpcd...
stopped /sbin/dhcpcd (pid 188)
Stopping network: OK
Stopping tee-supplicant: FAIL
Seeding 256 bits and crediting
Saving 256 bits of creditable seed for next boot
Stopping mdev... stopped process in pidfile '/var/run/mdev.pid' (pid 138)
OK
Stopping modules: rmmod: can't unload module 'soundcore': Resource temporarily u                     navailable
rmmod: can't unload module 'snd': Resource temporarily unavailable
rmmod: can't unload module 'snd_timer': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm_dmaengine': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_core': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_simple_card_utils': Resource temporarily una                     vailable
rmmod: can't unload module 'cdc_acm': No such file or directory
rmmod: can't unload module 'libcomposite': Resource temporarily unavailable

buildroot (kernel 5.15)

The 5.15 kernel changes here do not appear to have been implemented yet. https://github.com/MeemeeLab/CoreMP135_buildroot-external-st/commit/ce791082dc31ad803c9aaa3225f5fd8195e32b4d The 5.15 kernel version is currently the recommended version (The default factory firmware version) in the M5Stack documentation, so we would like to see this version implemented as well.

https://docs.m5stack.com/en/guide/linux/coremp135/buildroot

I will performe a BuildRoot Linux build on an Ubuntu PC.

git clone https://github.com/m5stack/CoreMP135_buildroot.git
git clone -b upstream-backport https://github.com/MeemeeLab/CoreMP135_buildroot-external-st.git
cd CoreMP135_buildroot
make BR2_EXTERNAL=../CoreMP135_buildroot-external-st/ m5stack_coremp135_515_defconfig
make -j4

MeemeeLab commented 3 months ago

@nnn112358 I couldn't reproduce behavior you mentioned. Here is logs taken from my CoreMP135:

#     #  #######   #####   #######     #      #####   #    #
##   ##  #        #     #     #       # #    #     #  #   #
# # # #  #        #           #      #   #   #        #  #
#  #  #  ######    #####      #     #     #  #        ###
#     #        #        #     #     #######  #        #  #
#     #  #     #  #     #     #     #     #  #     #  #   #
#     #   #####    #####      #     #     #   #####   #    #
Welcome to CoreMP135, Powered by M5Stack.
CoreMP135 login: root
# poweroff
# Stopping /etc/rc.localOK
Stopping dropbear sshd: OK
Stopping dhcpcd...
stopped /sbin/dhcpcd (pid 188)
Stopping network: OK
Stopping tee-supplicant: FAIL
Stopping mdev... stopped process in pidfile '/var/run/mdev.pid' (pid 140)
OK
Stopping modules: rmmod: can't unload module 'soundcore': Resource temporarily unavailable
rmmod: can't unload module 'snd': Resource temporarily unavailable
rmmod: can't unload module 'snd_timer': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm_dmaengine': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_core': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_simple_card_utils': Resource temporarily unavailable
rmmod: can't unload module 'cdc_acm': No such file or directory
rmmod: can't unload module 'libcomposite': Resource temporarily unavailable
OK
Stopping klogd: OK
Stopping syslogd: OK
Seeding 256 bits and crediting
Saving 256 bits of creditable seed for next boot
umount: devtmpfs busy - remounted read-only
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[   12.493871] reboot: Power down

I'm guessing your LCD panel power isn't powering-off, but try connecting Ethernet and see LED when powering off. if it turns off when poweroff, it's expected behavior (poweroff)

For 5.15 kernel, see conversations above: https://github.com/m5stack/CoreMP135_buildroot-external-st/issues/3#issuecomment-2185605697

MeemeeLab commented 3 months ago

Also, if init system is the problem, try poweroff -n -f. This may damage your file system, but you can burn SD if it corrupts. I'm not really sure what is stopping module removal to go through.

dianjixz commented 3 months ago

@nnn112358 @MeemeeLab The g_serial module got stuck during the process of uninstallation, because it didn't function without a USB inserted, which prevented it from being unloaded. I'm considering whether to separately start a service for g_serial or simply not proceed with unloading the kernel module. Additionally, the shutdown process in OP-TEE has been completed, and I'm currently organizing the code and creating patches so that poweroff can be used soon. Could you provide me with some advice regarding the issue with g_serial?

dianjixz commented 3 months ago

The handling of AXP2101 is a very tricky issue. First of all, the I2C3 bus it resides on is an insecure bus with many devices hanging on it and has been reinitialized by the kernel. Therefore, it is difficult for OP-TEE to continuously use AXP2101 as a PMIC module driver. Secondly, it seems that the new kernel has a shutdown hook API, but I don't think this is a good choice because U-Boot and lower versions of the Linux kernel cannot use it. Thirdly, in the final stages of shutdown, it is often difficult to have access to the driver framework API for use, which is a very frustrating point. I used GPIO to simulate the I2C timing to complete the final power-off task of the PMIC, which may not be universal, but it can solve the power-off problem of CoreMP135 during shutdown.

MeemeeLab commented 3 months ago

@dianjixz I'm not sure how U-Boot and/or OP-TEE should handle poweroffs, but I'd rather implement it in kernel side. AXP2101 handles backlight and poweroff which backlight should be handled via kernel, so kernel should initialize AXP2101 IMO. Though it is fact that u-boot can't poweroff if we implement poweroff this way. For poweroff in Linux 5.15, I believe poweroff hooks are present from old days.

I didn't notice problem with g_serial, so I'll investigate what's happening there. Because module shouldn't disrupt unload whenever module is not used or initializing, I feel something is off here.

MeemeeLab commented 3 months ago

This is example poweroff driver found in linux 5.15: https://github.com/gregkh/linux/blob/v5.15.161/drivers/power/reset/gpio-poweroff.c It's implemented using pm_power_off global callback, not method callback way, so bit of axp2101-poweroff.c modifications are required.

dianjixz commented 3 months ago

@MeemeeLab I know about the pm_power_off method, and I have tried it before. When it gets to the pm_power_off callback, the kernel has almost no driver framework APIs available. I have decided to keep the OP-TEE implementation as a last resort. You can continue trying to implement this feature in the kernel; it will be an art.Please help me check the g_serial issue. It has a very low priority in my work schedule. Thank you very much.

MeemeeLab commented 3 months ago

@dianjixz Just performed quick test with my USB charger; it does not make any use of USB data line. Tried modprobe g_serial and then rmmod g_serial, it did not hang up. I'm not really sure g_serial is causing poweroff disrupt, as @nnn112358 poweroff logs does not mention about g_serial, and mine too.

MeemeeLab commented 3 months ago

AFAIK, kernel does not unload any drivers before poweroff, but init systems (BusyBox, systemd, etc.) may unload modules before shutdown. Because i2c drivers are not compiled as modules, it's probably not unloaded, but that's my guess.

I'll try backporting to 5.15 when I have enough time.

nnn112358 commented 3 months ago

@MeemeeLab First, let me and you confirm this problem.

Is there a difference between CoreMP135 and the firmware? Here is the firmware I built. https://github.com/nnn112358/CoreMP135_buildroot_firmware/

Can you store the firmware(sdcard.img) you built somewhere?

dianjixz commented 3 months ago

@nnn112358 m5stack_coremp135_515_defconfig shutdown patch has been pushed. You can test it. Once confirmed that it's working fine, we will plan to update it to our release image later.

MeemeeLab commented 3 months ago

@nnn112358 Just decompiled dtbs from your images and yes, it contains lines changed by my PR. I'll test with your image and see what happens. also I'll upload images build on my computer later.

MeemeeLab commented 3 months ago

@nnn112358 Poweroff works for me with your build. Can you try them again?

This is fingerprint of your build I just tried.

Linux CoreMP135 6.1.28 #1 SMP PREEMPT Tue Jun 25 02:11:25 JST 2024 armv7l GNU/Linux

MeemeeLab commented 3 months ago

Here's my build, but I don't think there is difference for drivers. My build contains libdrm and its utilities to test DRM devices. https://gofile.io/d/wueGV9

MeemeeLab commented 3 months ago

Also, can you try poweroff -n -f if you still can't poweroff? this helps identify init system problem or driver problem.

nnn112358 commented 3 months ago

@MeemeeLab @dianjixz

I have described the results of my test. The latest CoreMP135_buildroot-external-st allowed me to poweroff on my CoreMP135. MeemeeLab firmware poweroff Failed depending on USB-C connection status.

@MeemeeLab firmware → https://gofile.io/d/wueGV9

CoreMP135_buildroot-external-st(commit 6d7b28c , 2024/6/26)↓

git clone https://github.com/m5stack/CoreMP135_buildroot.git
git clone https://github.com/m5stack/CoreMP135_buildroot-external-st.git
cd CoreMP135_buildroot
make BR2_EXTERNAL=../CoreMP135_buildroot-external-st/ m5stack_coremp135_515_defconfig
make -j4

MeemeeLab Firmware non USB-C connect

# poweroff
# Stopping /etc/rc.localOK
Stopping dropbear sshd: OK
Stopping dhcpcd...
stopped /sbin/dhcpcd (pid 188)
Stopping network: OK
Stopping tee-supplicant: FAIL
Stopping mdev... stopped process in pidfile '/var/run/mdev.pid' (pid 140)
OK
Stopping modules: rmmod: can't unload module 'soundcore': Resource temporarily unavaila     ble
rmmod: can't unload module 'snd': Resource temporarily unavailable
rmmod: can't unload module 'snd_timer': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm_dmaengine': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_core': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_simple_card_utils': Resource temporarily unavailabl     e
rmmod: can't unload module 'cdc_acm': No such file or directory
rmmod: can't unload module 'libcomposite': Resource temporarily unavailable

MeemeeLab Firmware non USB-C connect

# poweroff -n -f
[   32.743654] reboot: Power down

MeemeeLab Firmware with USB-C connect

# poweroff
# Stopping /etc/rc.localOK
Stopping dropbear sshd: OK
Stopping dhcpcd...
stopped /sbin/dhcpcd (pid 188)
Stopping network: OK
Stopping tee-supplicant: FAIL
Stopping mdev... stopped process in pidfile '/var/run/mdev.pid' (pid 140)
OK
Stopping modules: rmmod: can't unload module 'soundcore': Resource temporarily unavailable
rmmod: can't unload module 'snd': Resource temporarily unavailable
rmmod: can't unload module 'snd_timer': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm_dmaengine': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_core': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_simple_card_utils': Resource temporarily unavailable
rmmod: can't unload module 'cdc_acm': No such file or directory
rmmod: can't unload module 'libcomposite': Resource temporarily unavailable
OK
Stopping klogd: OK
Stopping syslogd: OK
Seeding 256 bits and crediting
Saving 256 bits of creditable seed for next boot
umount: devtmpfs busy - remounted read-only
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[   87.433591] reboot: Power down

CoreMP135_non USB-C connect

# poweroff
# Stopping /etc/rc.localOK
Stopping dropbear sshd: OK
Stopping dhcpcd...
stopped /sbin/dhcpcd (pid 177)
Stopping network: OK
Stopping tee-supplicant: FAIL
Seeding 256 bits and crediting
Saving 256 bits of creditable seed for next boot
Stopping mdev... stopped process in pidfile '/var/run/mdev.pid' (pid 138)
OK
Stopping modules: OK
Stopping klogd: OK
Stopping syslogd: OK
umount: devtmpfs busy - remounted read-only
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[   81.623593] reboot: Po

One thing I noticed, Compared to MeemeeLab firmware connected with USB-C, the timing of poweroff seems to be a little earlier and shutdown occurs in the middle of displaying "reboot: Power down" in Port.C Serial. This may not be a problem.

MeemeeLab commented 3 months ago

I noticed that I had different kernel configuration when testing g_serial. this may explain why g_serial on my kernel worked correctly.

Will push PR if this was case.

MeemeeLab commented 3 months ago

@dianjixz Opened PR to fix module unloads. Do you mind if I open PR to re-implement poweroff for 5.15 on kernel side? (not doing backport for now, just a possibility, idk I can implement or not) I'm not sure hard-coding poweroffs in U-Boot is good idea, though it's patch only used in CoreMP135.

dianjixz commented 3 months ago

@MeemeeLab I am very happy that you can perform the task of cutting off the kernel. I am considering using optee as the underlying security layer, which should have a valid pmic shutdown procedure, so I will place the code for cutting off the pmic in optee. Go ahead, theoretically the kernel also needs a shutdown code.

nnn112358 commented 3 months ago

@MeemeeLab @dianjixz

In the MeemeeLab firmware, I edited /etc/init.d/S02modules & /etc/init.d/S50agetty & /etc/inittab to match MeemeeLab's pulls. Power-off succeeded. I understood that there are multiple solutions to the poweroff problem.

Great job, I am very happy with the results. Thanks for your help !!

# poweroff
# Stopping /etc/rc.localOK
Stopping dropbear sshd: OK
Stopping dhcpcd...
stopped /sbin/dhcpcd (pid 188)
Stopping network: OK
Stopping tee-supplicant: FAIL
Stopping mdev... stopped process in pidfile '/var/run/mdev.pid' (pid 140)
OK
Stopping modules: rmmod: can't unload module 'soundcore': Resource temporarily unavailable
rmmod: can't unload module 'snd': Resource temporarily unavailable
rmmod: can't unload module 'snd_timer': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm': Resource temporarily unavailable
rmmod: can't unload module 'snd_pcm_dmaengine': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_core': Resource temporarily unavailable
rmmod: can't unload module 'snd_soc_simple_card_utils': Resource temporarily unavailable
rmmod: can't unload module 'cdc_acm': No such file or directory
rmmod: can't unload module 'libcomposite': Resource temporarily unavailable
OK
Stopping klogd: OK
Stopping syslogd: OK
Seeding 256 bits and crediting
Saving 256 bits of creditable seed for next boot
umount: devtmpfs busy - remounted read-only
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[  266.883670] reboot: Power down

m5stack / CoreMP135_buildroot-external-st