Serial port not working properly after kernel upgrade

mariusmotea commented 1 month ago

Describe the bug

I have a PI5 used for my home stuff (HomeAssistant, Zigbee2MQTT, DiyHue, etc), all running in docker containers. Two days ago i notice more than 100 package updates so i decided to update all and reboot. After few minutes i discovered the lights where not working anymore so i begin to troubleshoot. I notice in zigbee2mqtt configuration the serial interface was pointing to /dev/ttyAMA1 and this was not present anymore on the OS so i replace it with the new name /dev/ttyAMA0. After this the zigbee2mqtt was not crashing anymore but it hang right after it start to connect to the zigbee controller. I performed the following test in order to isolate the problem:

replacing the zigbee controller (i have 4 in total) => no change
downgrade docker, containerd => no change
flashing the zigbee controller firmware using the same serial interface was successful => the serial port was correctly identified and at last is partially working
test the serial port using the loop method at the 115200 baud rate, the same used by zigbee2mqtt => test was successfully
connect one zigbee controller to an USB to serial adapter => zigbee2mqtt started and it work as expected.

I spent about 8 hours trying to find a solution to this problem and i know it sound wired but i cannot find an explanation why connecting the zigbee controller to an USB adapter (3.3v TTL) is working fine while when i connect it directly to the pi serial interface i can only flash the firmware but i cannot use it and this is happening exactly after i update the packages on the PI.

Steps to reproduce the behaviour

Connect a TI 2652P zigbee conectoller to Raspberry PI 5 serial interface and start Zigbe2MQTT

Device (s)

Raspberry Pi 5

System

cat /etc/rpi-issue Raspberry Pi reference 2023-12-05 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 70cd6f2a1e34d07f5cba7047aea5b92457372e05, stage4

uname -a Linux raspberrypi5 6.6.47+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.47-1+rpt1 (2024-09-02) aarch64 GNU/Linux

Logs

No response

Additional context

No response

mariusmotea commented 1 month ago

Today i made a test with uart3 and uart4, same result like with uart0, the application freeze. I also installed zigbee2mqtt on the host to exclude a problem with docker.

mariusmotea commented 1 month ago

I think i can confirm now that this is a kernel bug. I installed Raspberry PI OS on a new SD card using Imager tool and serial connection to zigbee controller was working fine. The interface rename was not the root issue since the new name (ttyAMA0) was present.

pi@raspberrypi:~ $ uname -r
6.6.31+rpt-rpi-2712

I made it to not work only by updating the kernel.

pi@raspberrypi:~ $ sudo apt install linux-image-rpi-v8
------------------------------
The following NEW packages will be installed:
  linux-headers-6.6.47+rpt-common-rpi linux-headers-6.6.47+rpt-rpi-2712 linux-headers-6.6.47+rpt-rpi-v8 linux-image-6.6.47+rpt-rpi-2712 linux-image-6.6.47+rpt-rpi-v8 linux-kbuild-6.6.47+rpt
The following packages will be upgraded:
  linux-headers-rpi-2712 linux-headers-rpi-v8 linux-image-rpi-2712 linux-image-rpi-v8 linux-libc-dev
--------------------------------------------
Setting up linux-kbuild-6.6.47+rpt (1:6.6.47-1+rpt1) ...
Setting up linux-headers-6.6.47+rpt-common-rpi (1:6.6.47-1+rpt1) ...
Setting up linux-libc-dev (1:6.6.47-1+rpt1) ...
Setting up linux-headers-6.6.47+rpt-rpi-v8 (1:6.6.47-1+rpt1) ...
Setting up linux-image-rpi-2712 (1:6.6.47-1+rpt1) ...
Setting up linux-headers-6.6.47+rpt-rpi-2712 (1:6.6.47-1+rpt1) ...
Setting up linux-headers-rpi-2712 (1:6.6.47-1+rpt1) ...
Setting up linux-image-rpi-v8 (1:6.6.47-1+rpt1) ...
Setting up linux-headers-rpi-v8 (1:6.6.47-1+rpt1) ...
---------------------------
pi@raspberrypi:~ $ sudo reboot

pelwell commented 1 month ago

Thanks - that's a useful bit of differential diagnosis which does implicate a recent kernel change. The problem is going to be assembling a system that can reproduce the problem without requiring significant investments of time or money.

Are there any bits of your system that aren't required? For example, is it necessary for you to containerise Zigbee2MQTT? In other words, what is the minimum (hardware and software) we would need to see the fault?

mariusmotea commented 1 month ago

Hi. Not sure if the issue is happening with all ZigBee2MQTT controllers from Texas Instruments, but mine is TI CC2652P flashed with Z-Stack firmware using this method (without --bootloader-sonoff-usb argument). There are some adapters available on the market for Raspberry pi, but i have a spare Raspberry PI 4B and i can connect this adapter and provide you SSH access to it if the issue is happening also on Raspberry PI 4B. If Raspberry PI 4B is not a good candidate i can provide access also to my rPI 5, but here we need to keep the running instance of zigbee2mqtt with USB - serial adapter and start a new instance on /dev/ttyAMA0 where i can plug the second zigbee controller.

ZigBee2MQTT is a nodejs based application and i tested both docker and host installation methods with same result. Both installation methods are provided here.

Regards!

qrp73 commented 1 month ago

try to use /dev/ttyS0 instead of /dev/ttyAMA0. I remember, previously onboard serial port was worked as /dev/ttyS0 but when I added dtoverlay=disable-bt line into config.txt, it appears that serial port is reported as /dev/ttyS0 but cannot work with this name anymore and requires to use /dev/ttyAMA0 instead. So, may be you have similar issue.

mariusmotea commented 1 month ago

With dtoverlay=disable-bt i now have /dev/ttyS0 but i don't know the GPIO pind used, for sure are not 14 and 15. Also no change on ttyAMA0 functionality after adding dtoverlay=disable-bt

pelwell commented 1 month ago

disable-bt is not useful on a Pi 5 other than to disable Bluetooth - the UART that is freed cannot be mapped to the 40-pin header, and it's not like Pi 5 is short of UARTs that can be used for other purposes.

pelwell commented 1 month ago

There are two test kernels you can try:

sudo rpi-update - this disables DMA to the UART
sudo rpi-update pulls/6377 - this re-enables DMA in a way which is more aware of the UART's needs

mariusmotea commented 1 month ago

Hi,

I tested both and are working. Currently i'm on rpi-update pulls/6377 so this will be much better tested in the next days.

Thanks.

pelwell commented 1 month ago

Thanks for letting me know. One of those two changes will be released for all users.

raspberrypi / linux