search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.03k stars 4.96k forks source link

Pi 5 UARTs broken in 6.6.44 build of rpi-6.6.y. #6365

Open pelwell opened 4 days ago

pelwell commented 4 days ago

Describe the bug

As detailed by @lgeitner in the thread starting at https://github.com/raspberrypi/linux/pull/6144#issuecomment-2349489287, the MeteoPi application which requires UART access seems to be broken in the 6.6.44 build of rpi-6.6.y. It has also been shown to have worked in the 6.6.42 build. The corresponding source commits for those two builds are:

"kernel: Bump to 6.6.42" (1969fd7): bfbd468e81defa70d4356a1a3e467cc5d67ca89b "kernel: Bump to 6.6.44" (44a287f): 2806f39528464ddcc56f77dd8ea4d81f71b7e619

Steps to reproduce the behaviour

Run MeteoPi. Other test cases have yet to be determined.

Device (s)

Raspberry Pi 5

System

Linux 6.6.44-v8-16k+ #1789 SMP PREEMPT Mon Aug 5 15:24:18 BST 2024 aarch64 GNU/Linux

Logs

N/A

Additional context

No response

pelwell commented 4 days ago

The current list of suspect commits is:

2806f39528464 Revert "mfd: Partial revert of Add rp1 driver"
d0ea54e1c941a mfd: Partial revert of Add rp1 driver
216df57950849 DTS: bcm2712: enable SD slot CQE by default on Pi 5
3c319a466a1c7 dtoverlays: Document display_[width|height] on hd44780-lcd overlay
e94e761305fa2 dtoverlays: Add overlay for HD44780 via I2C PCF8574 backpack
a4bf61fad9fe1 fixup! pinctrl: bcm2712 pinctrl/pinconf driver
16d0ee22d2c0b hwmon: (adt7410) Add DT compatible strings
05e3687c6c973 overlays: i2c-rtc: Correct bq32000 property name
e9294823cf020 spi: dw: Clamp the minimum clock speed
199e611183de0 spi: dw: Fix non-DMA transmit-only transfers
70636ad110715 ARM: dts: bcm2712: Fix invalid polling-delay-passive setting
485d11cfa7df2 drm/vc4: Disable the 2pixel/clock odd timings workaround for interlaced
062434ab3be76 dts: rp1: restrict i2s burst lengths to 4
b6b4260fa546d sound/soc: dwc-i2s: choose FIFO thresholds based on DMA burst constraints
5112fd8cce4f1 DT: bindings: add a dma-maxburst property to snps,designware-i2s
f3cb675102a2a tty/serial: pl011: restrict RX burst FIFO threshold
1c4c90982462e Revert "spi: dw-dma: Get the last DMA scoop out of the FIFO"
ce56098eb4dc2 drivers: dw-axi-dmac: make more sensible choices about memory accesses
4c1a665b465fa dts: rp1: hobble DMA AXI burst lengths
3af7822df36e3 spi: dw: don't immediately kill DMA transfers if an error occurs
cd9084ceb606a spi: dw: Save bandwidth with the TMOD_RO feature
6014649de765a spi: dw: Save bandwidth with the TMOD_TO feature
31b9871b8895d staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1

I'll try and create a PR from the 6.6.44 build head (2806f39528464) but with those commits reverted.

pelwell commented 4 days ago

6366 is the PR that will create the trial builds. After each update to it, wait about 45 minutes for the builds to complete then run sudo rpi-update pulls/6366.

pelwell commented 4 days ago

Do you want to reconsider any of the above patches, @P33M?

P33M commented 4 days ago

f3cb675102a2a tty/serial: pl011: restrict RX burst FIFO threshold may be causing side-effects. This isn't critical to UART operation with the restricted DMAC burst sizes - the RX timeout interrupt will gather any remaining bytes.

P33M commented 4 days ago

This might be the same bug - long TX bursts appear to drop data on Pi 5 - https://forums.raspberrypi.com/viewtopic.php?t=376532

pelwell commented 4 days ago

With f3cb675102a2a I see data loss at the end of transmission:

00009300: 72 69 61 6c 3a 20 6e 6f : 20 70 6c 61 74 66 6f 72
00009308: 20 44 4d 41 20 70 6c 61 : 6d 20 64 61 74 61 0a
00009310: 74 66 6f 72 6d 20 64 61 :
00009318: 74 61 0a                :

(where 931a is the last byte, and everything before 9300 matches) With it reverted, I've not seen any data loss yet.

I've updated #6366 to be top-of-tree rpi-6.6.y but with f3cb675102a2a reverted. As before, wait about 45 minutes then run sudo rpi-update pulls/6366.

lgeitner commented 4 days ago

ran sudo rpi-update pulls/6366

received the error "Invalid artifact specified. Response: 404." please see below

pi@cumulusmxkitchen:~ $ sudo rpi-update pulls/6366 Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom Performing self-update Relaunching after update Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom FW_REV: BOOTLOADER_REV:d05f05c94fd7a1916942e56d7f486c142ae5661e WANT_32BIT:0 WANT_64BIT:1 WANT_PI4:1 WANT_PI5:1 *** Downloading specific artifact revision (this will take a few minutes) curl -L https://builds.raspberrypi.com/github/linux/8aa0ad843d49a58d738177fd892ec547015dd231/bcmrpi | zcat | tar xf - -C //root/.rpi-firmware --strip-components=2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0

gzip: stdin: unexpected end of file tar: This does not look like a tar archive tar: Exiting with failure status due to previous errors % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 Invalid artifact specified. Response: 404. pi@cumulusmxkitchen:~ $

pelwell commented 4 days ago

I thought this might happen. I updated the PR to include a merge-ready commit message, and you came online and tried to update in the interval while it was being rebuilt. It should be good now.

lgeitner commented 4 days ago

ran sudo rpi-update pulls/6366 then sudo reboot

the meteopi is not working!

the command uname -a displays the following:

pi@cumulusmxkitchen:~ $ uname -a Linux cumulusmxkitchen 6.6.51-v8-16k+ #1 SMP PREEMPT Tue Sep 17 16:24:28 UTC 2024 aarch64 GNU/Linux

pelwell commented 4 days ago

Did you reboot after the install? Yes - I can see that you did.

There's another build with more reversions available via sudo rpi-update pulls/6369.

lgeitner commented 4 days ago

I ran sudo rpi-update pulls/6369 then sudo reboot

The meteopi is now working!

I would like to allow the meteopi to run overnight and check for any errors from the meteopi.
I will let you know if there are any errors or problems sometime tomorrow.

Thank You!

output from the command uname- a

pi@cumulusmxkitchen:~ $ uname -a Linux cumulusmxkitchen 6.6.51-v8-16k+ #1 SMP PREEMPT Tue Sep 17 19:44:50 UTC 2024 aarch64 GNU/Linux pi@cumulusmxkitchen:~ $

lgeitner commented 4 days ago

I just noticed when I ran the command dmesg there is the following error in red

[ 11.585942] pl011-axi 1f00030000.serial: unable to pause DMA transfer

pelwell commented 3 days ago

sudo rpi-update pulls/6371 will get you the release candidate patch. It simply disables DMA for UART0 until it can be shown to be reliable.

lgeitner commented 3 days ago

I let the meteopi run overnight, there were no errors! Its working as expected!

Thank you again for your help!

I will try sudo rpi-update pulls/6371 sometime in the next couple hours and will let you know if I have any issues with it.

Thanks, Lance

lgeitner commented 3 days ago

I ran the command sudo rpi-update pulls/6371 then sudo reboot.

The meteopi is working fine and the error "pl011-axi 1f00030000.serial: unable to pause DMA transfer" no longer appears in dmesg.

the output from uname -a is as follows:

pi@cumulusmxkitchen:~ $ uname -a Linux cumulusmxkitchen 6.6.51-v8-16k+ #1 SMP PREEMPT Wed Sep 18 16:24:48 UTC 2024 aarch64 GNU/Linux pi@cumulusmxkitchen:~ $

Thank you again for your help!

pelwell commented 3 days ago

Thank you for reporting it - shipping that kernel in an image would have been embarrassing.