Serial console broken on RPi 3

[diederikdehaas]

Using the raspbian-jessie-lite image dd 2016-02-26 (and performing all updates after that), I can't get the serial console working on a RPi 3. When I put the image in a RPi 2, everything is fine.

By not working, I mean that I mostly get garbled output and I am unable to use my keyboard to interact with the system.

There is a forum thread about it and (apparently) someone found a workaround ... by crippling the performance of the RPi 3 by adding core_freq=250 to /boot/cmdline.txt which also reduces the CPU speed to 600 Mhz.

I don't consider that an valid option.

[XECDesign]

You can also get around the issue by adding force_turbo=1 to config.txt

[diederikdehaas]

It does indeed work :smile:

Is there some side effect that causes that? Because logically I don't see the connection ...

[XECDesign]

Something to do with the source clock for the serial. I don't know the specifics.

[diederikdehaas]

Thanks :+1:

Is this the official solution or is there another one in the pipe-line? I'll leave it up to you to close the issue or not.

[XECDesign]

I think I saw someone say that it may have already been fixed in newer firmware, but I couldn't see that in the commit messages.

@pelwell Is this something that can be or has been fixed?

[pelwell]

There are some inescapable facts:

1) uart1, like the sdhost interface, uses the core clock. If the core clock changes then the baud rate will change. 2) Bluetooth uses uart0 because it has larger FIFOs and because it is immune to changes in core clock.

You can reduce clock rate fluctuations using force_turbo=1, but there is still the possibility of an enforced clock rate reduction because of an over-temperature situation. Between 80C and 85C the ARM overclock is disabled, then at 85C all other clocks (core, sdram, v3d, isp, h264) are also reduced to minimum.

It may be possible to ameliorate the issue by automatically reprogramming the uart clock divisor when the core clock changes, but that will depend on there being a suitable ratio between the overclock and the minimum. Therefore the only way to be sure that the baud rate doesn't ever change is to set core_freq to minimum to start with, which is 250MHz.

Setting core_freq to 250MHz does not limit the ARMs to 600MHz. Those clocks are all independent.

The main clocks are controlled by the config.txt settings arm_freq, core_freq, sdram_freq, v3d_freq, isp_freq and h264_freq. gpu_freq is an alias for the non-ARM clocks that haven't been set explicitly. Until the most recent firmware there was an oversight that meant that setting core_freq implicitly by setting gpu_freq wasn't being communicated to the Linux world, causing incorrect baud rates. The latest rpi-update firmware fixes that issue, but you can avoid it by using core_freq instead of gpu_freq.

[XECDesign]

Ah, thanks for the full explanation.

Related https://github.com/raspberrypi/firmware/issues/553

[Mwt-Anthony]

I have same problems... I can't use my RPI3... I used Razberry zwave card, and impossible to communicate with ! I try force-turbo but without success!

I put my sdcard into my RPI2 it work like a charm...

I hope problem can be solve quickly. Thanks

[XECDesign]

Careful, that's force_turbo, not force-turbo.

[pelwell]

A user on the other thread has found that gpu_freq=300 works for them.

[josn0]

Phil (pelwell) is right. I started to retrieve all frequency settings on a earlier Pi (using 'vcgencmd get_config int | grep freq'), copying all to config.txt to overrule the new defaults, and the problem was gone. Then, to get performance back, I stepwise increased the settings until I reached the new defaults (also found with get_config). After I reached all new default settings, everything kept working. Then I commented-out the __freq settings until the problem re-appeared; that was with gpufreq. So put that back and the problem was all over, without changing any settings. My theory is that, pre-kernel-boot, something needs to set some frequencies, reads them out of the config.txt, and if not present, uses the (old) defaults. I think that is done in the start.elf files. Alas, they are closed-source. But, by grep-ing them, I found the texts 'gpu_freq' and 'config.txt', which makes my assumption more plausible.

Another thing that is probably done there, is 'patching' the kernel's commandline parameters: file 'cmdline.txt' now contains 'console=serial,115200' instead of 'console=ttyAMA0,115200'. But whether the console is named 'serial' or 'ttyAMA0', if you check /proc/cmdline, both are changed to 'ttyS0'. I think that is also done in the start*.elf code.

If so, it is a pity that such non-transparent 'hacks' are buried in closed-source proprietary code. Please dont do that. I can understand that the start*.elf file need to be closed-source, because it needs to interact with the proprietary gpu, but then keep the code in there as small as possible and as transparant as possible. Put strange hacks somewhere else, like in the kernel code, that everyone can read.

[diederikdehaas]

'cmdline.txt' now contains 'console=serial,115200' instead of 'console=ttyAMA0,115200'. But whether the console is named 'serial' or 'ttyAMA0', if you check /proc/cmdline, both are changed to 'ttyS0'.

I haven't investigated it, but I noticed that too.

[ladyada]

verifying that the baud rate is ~72Kbps not 115Kbps - works great if I set my serial cable to 72000, note that prolific chipsets (PL230x) cannot do weird baud rates, try an FTDI or SiLabs chip :)

[pelwell]

@josn0 I was surprised that I couldn't find a standard DT mechanism for giving serial ports a chosen name. The PL011 driver attempts to renumber its own instances based on serial aliases, but it calls its instances ttyAMA instead of ttyS and other serial drivers don't seem to obey the same rule.

The reason for the cmdline rewriting by the firmware is to allow a single SD card image to work in any Pi until we have worked out a better mechanism in the kernel. I don't particularly like it but our options were limited and time was short.

@ladyada 72Kbps is 115200*(250/400), with 400 being the standard VPU overclock and 250 being the minimum used in overtemp or undervolt situations. I think you will see the power LED blinking off, indicating undervolt. I had to change my power supply to 2.5A model - I have any inline switch in the USB power cable and with the additional loss it adds the startup current for WiFi and Bluetooth is sufficient to drag the voltage down below the threshold.

[glynhudson]

I am trying to connect an Arduino @ 38400 baud to the Rpi3 via ttyAMA0 with no joy. Zero comms. Is this related this this thread or should I start another issue? I am using an SD card upgraded from Jessie that worked fine on RPi2. Serial console has been disconnected from terminal in the usual way

[pelwell]

It is related. ttyAMA0 on a Pi3 is configured to communicate with the Bluetooth modem. Try ttyS0.

[glynhudson]

How can I try ttyS0? I am connecting via Pi's UART pins on GPIO. It;s not possible for me to physically change the pins since the PCB connected directly onto the GPIO header.

On 4 March 2016 at 10:18, Phil Elwell notifications@github.com wrote:

It is related. ttyAMA0 on a Pi3 is configured to communicate with the Bluetooth modem. Try ttyS0.

— Reply to this email directly or view it on GitHub https://github.com/RPi-Distro/repo/issues/22#issuecomment-192221269.

Glyn Hudson

http://megni.co.uk http://adventuresplusnorthwales.blogspot.com

[pelwell]

UART1/MiniUART/ttyS0 is available as Alt function 5 on GPIOs 14 and 15. UART0/ttyAMA0 is Alt function 0. If you run:

sudo apt-get install raspi-gpio     # if you don't have it already
raspi-gpio get 14

on a Pi2 you will see that GPIO14 is mapped to a0 (TXD0), whereas on a Pi3 it is mapped to a5 (TXD1).

In other words, just open ttyS0 instead - the pin-muxing has already been configured to bring it out onto 14 and 15 (pins 8 and 10).

[diederikdehaas]

ttyAMA0 on a Pi3 is configured to communicate with the Bluetooth modem. Try ttyS0

Only on the Pi3 or should I use ttyS0 on all Pi's now?

[pelwell]

Only on Pi3. ttyAMA0 is a better serial port, immune from core clock changes, and it would be a shame to deny it to all Pi users just so we can use it for Bluetooth on the Pi3.

[diederikdehaas]

Ok, thanks for the info.

Right now we use the following as the start of our cmdline.txt: cmdline="dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 elevator=deadline" I can make the ttyAMA0/ttyS0 conditional based on the Pi version detected during installation, but AFAICS now that would make it so that you can't just swap the card to another Pi version and use the serial console. Ideas? (besides documenting it)

[grigorig]

How much does core_freq=250 affect performance, really? I wasn't able to measure any difference with some simple CPU benchmarks. It might be a suitable workaround if you absolutely need a stable Mini UART. Still, is there any way to change behaviour in low voltage or thermal event situations? It doesn't look like setting core_freq_min has any effect.

[pelwell]

That's what the "serial" aliases were created for. The firmware will look for console=serial0 and replace it with ttyS0 on a Pi3 or ttyAMA0 otherwise. It also replaces console=ttyAMA0 on a Pi3 with console=ttyS0 to help users with upgraded images. Unfortunately it doesn't detect kgdboc=serial<n>.

I can easily improve the mechanism to also work for kgdboc, but I also want to base the rewriting on the contents of the DTB rather than have it hard-coded for the Pi3, so that users who don't need Bluetooth can restore the previous behaviour with a DT overlay.

[diederikdehaas]

Thanks again :+1:

[glynhudson]

Thanks @pelwell that explains it.

Using ttyS0 I can see serial coms from my device (ATmega328 @ 38400 baud), however there also seems to be some other unexpected traffic on the same port. Serial messages from my device are slightly garbled and my device is reporting receiving serial traffic when minicom is open. I have tried adding core_freq=250 or force_turbo=1 or pi3-disable-bt to cmdline.txt, and nothing seem to have much of an effect. Does the order in cmdline.txt make a difference?

[pelwell]

The order in cmdline.txt and config.txt shouldn't make any differences. DT overlays are applied in order, but provided the overlays don't overlap then there shouldn't be a problem.

If you are seeing garbled serial port data when the core frequency is fixed then it suggests that the CPU is being throttled due to over-temperate or under-voltage (but that won't happen if you have pegged the core at 250MHz). A 2.5A power supply is recommended.

I've just completed a firmware mod that improves the serial port alias handling: 1) It will search for any existing "serial0" in the DT, allowing the pi3-disable-bt alias to also disable the remapping. 2) It now searches for "=serial0", "=ttyAMA0" etc. instead of "config=...", so it will work with "kgdboc=serial0,115200" etc.

[lurch]

That's what the "serial" aliases were created for. The firmware will look for console=serial0 and replace it with ttyS0 on a Pi3 or ttyAMA0 otherwise. It also replaces console=ttyAMA0 on a Pi3 with console=ttyS0 to help users with upgraded images.

Would it also be possible to do something similarly clever with udev rules, so that Linux-side applications could be configured to use e.g. /dev/serial0 or similar; which would be a symlink to /dev/ttyS0 on a Pi3 or a symlink to /dev/ttyAMA0 on other models? (thereby allowing the same application to work the same on all Pi models without reconfiguration)

[pelwell]

Either udev or, if we can't make that work, a systemd script - all it would take is sudo cp -pR /dev/ttyS0 /dev/serial0 with a bit of logic to work out which alias goes where.

[lurch]

...and then once it's included in Raspbian by default, maybe raspi-config could be modified to be take advantage of the /dev device aliasing/symlinking too?

I'm not familiar with udev / systemd / etc. but is there any reason that sudo cp -pR /dev/ttyS0 /dev/serial0 is better than sudo ln -s /dev/ttyS0 /dev/serial0 ?

[XECDesign]

I think I've come up with some rules.

KERNEL=="ttyS0", PROGRAM="/bin/sh -c 'if [ -d /proc/device-tree/soc/gpio@7e200000/bt_pins ]; then echo 0; else echo 1; fi;'", SYMLINK+="serial%c"
KERNEL=="ttyAMA0", PROGRAM="/bin/sh -c 'if [ -d /proc/device-tree/soc/gpio@7e200000/bt_pins ]; then echo 1; else echo 0; fi;'", SYMLINK+="serial%c"

%c is replaced with output of PROGRAM. Not sure if the check itself is good enough, but it seems to work for the simple/default case.

Edit: A bit confused about ownership though. I was under the impression that serial devices should belong to the dialout group, but ttyS0 belongs to tty.

lrwxrwxrwx 1 root root          5 Mar  9 02:44 /dev/serial0 -> ttyS0
lrwxrwxrwx 1 root root          7 Mar  9 02:44 /dev/serial1 -> ttyAMA0
crw-rw---- 1 root dialout 204, 64 Mar  9 02:44 /dev/ttyAMA0
crw--w---- 1 root tty       4, 64 Mar  9 02:44 /dev/ttyS0

[pelwell]

Thanks - that PROGRAM usage completes the puzzle for me. Leaving aside ownership, these rules work for me:

KERNEL=="ttyAMA[01]", PROGRAM="/bin/sh -c 'ALIASES=/proc/device-tree/aliases; if cmp -s $ALIASES/uart0 $ALIASES/serial0; then echo 0; elif cmp -s $ALIASES/uart0 $ALIASES/serial1; then echo 1; else exit 1; fi'", SYMLINK+="serial%c"
KERNEL=="ttyS0", PROGRAM="/bin/sh -c 'ALIASES=/proc/device-tree/aliases; if cmp -s $ALIASES/uart1 $ALIASES/serial0; then echo 0; elif cmp -s $ALIASES/uart1 $ALIASES/serial1; then echo 1; else exit 1; fi'", SYMLINK+="serial%c"

with a modified hciuart.service:

[Unit]
Description=Configure Bluetooth Modems connected by UART
ConditionPathIsDirectory=/proc/device-tree/soc/gpio@7e200000/bt_pins
Before=bluetooth.service
After=udev.service
ConditionPathExists=/dev/serial1

[Service]
Type=forking
ExecStart=/usr/bin/hciattach /dev/serial1 bcm43xx 921600 noflow -

[Install]
WantedBy=multi-user.target

The rule accepts ttyAMA1 as well as ttyAMA0 in case we drop the patch that prevents the port renumbering.

[grigorig]

There's another problem when it comes to Bluetooth: you have to reduce the baudrate to make it work with the Mini UART.

[pelwell]

Yes. That's why Bluetooth is configured to use ttyAMA0 by default, and why the pi3-miniuart-bt overlay documentation says:

Note that this may reduce the maximum usable baudrate.

[XECDesign]

Should we change cmdline.txt to use serial0 by default?

Regarding permissions, it seems like when getty runs, it changes the ownership group to tty, then login changes it to the logged in user.

[pelwell]

Yes we should - has that not happened yet? We did talk about it.

[XECDesign]

Wasn't sure, thanks.

[ghost]

Was directed here from the Adafruit forums

I tried connecting to the Pi 3b's serial console through the GPIO ports with this cable https://www.adafruit.com/products/954

This is what I saw on the screen upon attempting to connect with screen /dev/ttyUSB0 115200

[pelwell]

I suspect that if you add enable_uart=1 to config.txt and sudo rpi-update then the corruption will go away, but at the expense of fixing the core_freq to 250 (MHz). You can achieve the same effect by just adding core_freq=250 to your config.txt, but remember to remove it when you eventually update.

[ghost]

@pelwell I am unable to send reliable input to the Pi over the serial connection. Is there some other way to apply those changes?

[pelwell]

Either plug in a monitor and keyboard or take out the SD card and edit it in another computer.

[ghost]

@pelwell Great, thanks! I don't have a HDMI cable but I can edit on a different computer :)

[ghost]

@pelwell I tried making that change and booting again. My input & output are both still mostly garbled.

[pelwell]

That should have said core_freq=250. Now fixed.

[ghost]

@pelwell It works now, thanks for your help.

[ghost]

I have run rpi-update and it works:

Linux raspberrypi 4.4.7-v7+ #876 SMP Tue Apr 12 22:28:41 BST 2016 armv7l GNU/Linux

[Arun22]

Hi, i was using my Pi3 for last few weeks.my serial output was good after i fix the core_clock to 250. I just update my kernel/firmware using rpi-update and reboot the Pi3 yesterday. after that point i couldn't get any serial output. i thought i corrupted the firmware. so i tried the same SD card with my old Pi Model B one. i can get the output on the serial console in Pi Model B. i am not sure what happen to my Pi 3 after the rpi-update.

btw, I tired the following suggestion from this post: so far no luck so far, if you guys see anything obvious thing i miss here, please let me know.

1. force_turbo=1
2. gpu_freq=300
3. core_freq=300

[popcornmix]

Try enable_uart=1

[lurch]

https://github.com/raspberrypi/firmware/issues/553#issuecomment-199827489 has a bit more detail on how things work after an rpi-update.

[agherzan]

@grigorig in terms of performance I expect to impact on ARM but probably some impact on the GPU side. I would be interested on this too if you are/were able to benchmark anything. @pelwell correct me if i'm wrong.

[pelwell]

@agherzan Only the GPU core clock (which ironically doesn't really affect the graphics performance) is limited. It should have no impact on ARM performance except where it is waiting imperceptibly longer for the VPU to do something.