raspberry pi image on linuxcnc.org (Raspbian Buster/2.8.1) fails to boot after upgrade

[SebKuzminsky]

Download the "LinuxCNC 2.8.1 Raspberry Pi OS based on Debian 10 Buster" image from <http:/linuxcnc.org/downloads> (2021-01-20-linuxcnc-pi4.img).

Write to a micro SD card & boot.

Go through the first-boot system setup, join a Wifi network, but do not click "Update Software". Reboot (if you don't, the clock is wrong and apt refuses to update).

Run sudo apt update. Accept multiple warnings about the repository changing Suite:

E: Repository 'http://raspbian.raspberrypi.org/raspbian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
Do you want to accept these changes and continue updating from this repository? [y/N] y

Run sudo apt upgrade. This fails with errors about upgrading our linuxcnc.org kernel:

Get:1 http://linuxcnc.org buster/base armhf linux-image-4.19.71-rt24-v7l+ armhf 4.19.71-rt24-v7l+-4 [18.9 MB]
...
Preparing to unpack .../158-linux-image-4.19.71-rt24-v7l+_4.19.71-rt24-v7l+-4_armhf.deb ...
Leaving 'diversion of /boot/./bcm2709-rpi-2-b.dtb to /usr/share/rpikernelhack/./bcm2709-rpi-2-b.dtb by rpikernelhack'
Adding 'diversion of /boot/./vmlinuz-4.19.71-rt24-v7l+ to /usr/share/rpikernelhack/./vmlinuz-4.19.71-rt24-v7l+ by rpikernelhack'
Leaving 'diversion of /boot/./bcm2837-rpi-3-b.dtb to /usr/share/rpikernelhack/./bcm2837-rpi-3-b.dtb by rpikernelhack'
Leaving 'diversion of /boot/./bcm2708-rpi-zero-w.dtb to /usr/share/rpikernelhack/./bcm2708-rpi-zero-w.dtb by rpikernelhack'
Leaving 'diversion of /boot/./bcm2708-rpi-b-plus.dtb to /usr/share/rpikernelhack/./bcm2708-rpi-b-plus.dtb by rpikernelhack'
Leaving 'diversion of /boot/./bcm2835-rpi-cm1-io1.dtb to /usr/share/rpikernelhack/./bcm2835-rpi-cm1-io1.dtb by rpikernelhack'
Leaving 'diversion of /boot/./bcm2837-rpi-3-b-plus.dtb to /usr/share/rpikernelhack/./bcm2837-rpi-3-b-plus.dtb by rpikernelhack'
...
Unpacking linux-image-4.19.71-rt24-v7l+ (4.19.71-rt24-v7l+-4) over (4.19.71-rt24-v7l+-1) ...
dpkg: error processing archive /tmp/apt-dpkg-install-InsrxM/158-linux-image-4.19.71-rt24-v7l+_4.19.71-rt24-v7l+-4_armhf.deb (--unpack):
 unable to make backup link of './boot/System.map-4.19.71-rt24-v7l+' before installing new version: Operation not permitted
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
...
Errors were encountered while processing:
 /tmp/apt-dpkg-install-InsrxM/158-linux-image-4.19.71-rt24-v7l+_4.19.71-rt24-v7l+-4_armhf.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

If you reboot at this point, the machine fails to boot:

There are some alternative Raspberry Pi images floating around (e.g. https://forum.linuxcnc.org/9-installing-linuxcnc/45212-linuxcnc-on-raspberry-pi-4?start=70#261676).

We should fix the old image and/or make a new one, so that Raspberry Pi users have an easier time getting started with LinuxCNC. I don't know if there's a quick fix for the old Raspbian Buster/2.8 image, and with the momentum behind LinuxCNC in Debian Bookworm my preference would be for a new image. Then the choices are:

Raspberry Pi OS

Images here, image builder here, our fork is here.

Raspberry Pi OS has good support for device tree overlays (for enabling/disabling the various nifty hardware features of the Pi).

There is a Preempt-rt kernel in the Raspberry Pi OS package repo, so that's easy.

The latest image is based on Debian Bullseye, so we'd have to supply our own LinuxCNC debs, but that's not a showstopper.

Debian

Pre-built images here, image builder here, we don't have a fork of this yet.

Debian lacks support for Raspberry Pi OS's dynamic device tree overlays, but we have HAL drivers for the two most important pieces of hardware: SPI and GPIO (docs missing) so this is not a showstopper.

Latest image is based on Bookworm, so we get our Debian LinuxCNC packages and the Debian Preempt-RT kernel "for free".

Thoughts/comments?

[SebKuzminsky]

Here is a fork of the Debian Raspberry Pi Team's image builder "image-specs", with some customization for LinuxCNC: https://github.com/sebkuzminsky/image-specs

Here's the latest image: https://github.com/SebKuzminsky/image-specs/actions/runs/4217239851

That's Debian Bookworm arm64 with linux-image-rt-arm64 (6.1), xfce, and linuxcnc (2.9~pre from debian.org).

Log in as "pi" with password "raspberry" (just like our current/old image).

Works for me. Latency is pretty bad, spikes up above 200 µs, but good enough to run with Mesa hardware (SPI or Eth). The busywait branch of linuxcnc makes latency much better on this platform, but it's not merged yet.

[cakeslob]

Hey Seb, let me first say thank you for a renewed interest in the offical linuxcnc raspberrypi support.

Thoughts/comments?

Preface, I fall under the "Idiots" subcatagory of the user base The first thing I learned about the prebuilt images, is that upgrade breaks the image , thats been the case since the first one

I would say what ever is easiest for maintainers, so long as the end result isnt more work for the end user. By which I mean, if the end result doesnt involve searching for and installing some of the missing basic utilities for setup/usage. For me personally, this would be little things that make setup/usage more easy/standard. emphasis on easy and standard, where the most things work the easiest, little things for basic setup

• arandr for screen configuration
• a terminal that can use copy/paste commands, also things like
• correct graphics drivers which can now effect the physical display,
• correct qt library, there was something wrong with the raspberrypi buster qt libs calling the wrong gxf lib similar to this https://github.com/LinuxCNC/linuxcnc/pull/2314

Forgive my lack of true understanding as to what the dynamic device tree overlays is, but Im fairly sure it is how I configure my hardware without recompiling kernels or something of that nature? Would this have an effect on:

• non HDMI displays? eg DSI and DPI based hardware screens
• CM4 compute module based setups or Pi400 setups
• I2c related setups (i dont think it would be an issue but ill ask, Ive seen a few people using i2c hardware with lcnc/rpi)

There is also Remora, a newish SPI based controller for RPI not yet merged with linuxcnc. I dont think it would be an issue but it would be a dealbreaker if it is. If you have not yet aware of remora, it is a SPI based controller supporting multiple different mcu. https://forum.linuxcnc.org/18-computer/42276-remora-rpi-software-stepping-using-external-microcontroller-via-spi https://github.com/scottalford75/Remora

I just think that the largest percent of the userbase using a premade raspberry pi image, would benefit for an image being the most standard to what ever works across the range of hardware. Anyone capable enough is going to make a fancy custom install anyways, so I say cater to the lowest common denominator so it generates less rpi related support questions. Ill test this thing out see how it goes.

[cakeslob]

Hey Seb, this is looking pretty cool. For HDMI screens its working pretty good. For my non hdmi/compute module setups, it will take some figuring out compared to dtoverlay

[SebKuzminsky]

Hm, I can't get hm2_rpspi to work on this image. rtapi_app gets "permission denied", even after sudo make setuid. I think it's because the kernel is built with CONFIG_STRICT_DEVMEM. And the normal hm2_spi driver doesn't work, because the SPI hardware is disabled by the default DTB.

[SebKuzminsky]

The latest image built here fixes a bug with root partition resizing on the first boot: https://github.com/SebKuzminsky/image-specs/actions?query=branch%3Alinuxcnc

[samcoinc]

Side note. I downloaded a 'tested' bookworm image here https://raspi.debian.net/tested-images/ (boots to terminal) installed a desktop, rt preempt kernel and linuxcncn (apt-get install)

Everything went smooth except that the display has noise on it. https://photos.app.goo.gl/bQ21YgkBRjqDVCiN7

This would be awesome if it worked... I asked raspberry pie forum about it and it kinda sounds like it will never work..

https://forums.raspberrypi.com/viewtopic.php?p=2079760

this is as far as I got.. (tried multiple desktop environments - all act the same)

sam

On Fri, Feb 24, 2023 at 9:39 AM Sebastian Kuzminsky < @.***> wrote:

The latest image built here fixes a bug with root partition resizing on the first boot: https://github.com/SebKuzminsky/image-specs/actions?query=branch%3Alinuxcnc

— Reply to this email directly, view it on GitHub https://github.com/LinuxCNC/linuxcnc/issues/2349#issuecomment-1443865413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHRGQV3MTFYXOWW6LXFPKLWZDI3BANCNFSM6AAAAAAU72P5VY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[SebKuzminsky]

Yeah, i saw the noise on both the raspi.debian.net images and on the derived linuxcnc images linked above. It's ugly but doesn't seem to actually hurt anything.

[SebKuzminsky]

The latest LinuxCNC Raspberry Pi 4 Debian Bookworm image has working SPI using the hm2_rpspi driver only. The "normal" hm2_spi driver does not work because /dev/spidev* is not present, because the DTB disables SPI.

[SebKuzminsky]

Another alternative is the Ubuntu Raspberry Pi image. The 22.10 image comes with pre-built LinuxCNC debs in Ubuntu's package archive, but no Preempt-RT realtime kernel as far as i can tell. That image does have an updated DTB, so the normal SPI devices show up in /dev.

[SebKuzminsky]

And finally(?), the official Raspberry Pi OS Bullseye/arm64 image.

I had to download it via bit torrent, the https link kept giving me a corrupted image. Not sure what's up with that...

The image worked fine, and to my surprise and pleasure they're now using the debian.org package archive, plus their own overlay (very similar to how linuxcnc has been distributing our debs "on top of" mainline debian since forever). Normal debian.org packages install and run just fine (though of course, since this is based on Bullseye, there is no linuxcnc deb in the debian.org archive).

The awesome Raspberry Pi DTBs are available, and the runtime userspace device-tree overlay tools work fine.

And as usual, there's no realtime kernel. :-(

So we're kind of stuck. The only option with a realtime kernel is the official debian images (or our linuxcnc-specific fork of them), but that doesn't have good DTB support, so it's fairly limited in hardware support. We have working SPI (via hm2_rpspi) so maybe that's not a show-stopper, but for example mesaflash won't be able to talk to SPI boards. (It will only be able to talk to Ethernet boards.)

[samcoinc]

For some reason I get the impression Ubuntu is going to charge for a realtime kernel?

On Sun, Mar 12, 2023, 9:47 PM Sebastian Kuzminsky @.***> wrote:

And finally(?), the official Raspberry Pi OS Bullseye/arm64 image https://www.raspberrypi.com/software/operating-systems/.

I had to download it via bit torrent, the https link kept giving me a corrupted image. Not sure what's up with that...

The image worked fine, and to my surprise and pleasure they're now using the debian.org package archive, plus their own overlay (very similar to how linuxcnc has been distributing our debs "on top of" mainline debian since forever). Normal debian.org packages install and run just fine (though of course, since this is based on Bullseye, there is no linuxcnc deb in the debian.org archive).

The awesome Raspberry Pi DTBs are available, and the runtime userspace device-tree overlay tools work fine.

And as usual, there's no realtime kernel. :-(

So we're kind of stuck. The only option with a realtime kernel is the official debian images (or our linuxcnc-specific fork of them), but that doesn't have good DTB support, so it's fairly limited in hardware support. We have working SPI (via hm2_rpspi) so maybe that's not a show-stopper, but for example mesaflash won't be able to talk to SPI boards. (It will only be able to talk to Ethernet boards.)

— Reply to this email directly, view it on GitHub https://github.com/LinuxCNC/linuxcnc/issues/2349#issuecomment-1465428345, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHRGQR4QL7MMSNXDYI25C3W32KEJANCNFSM6AAAAAAU72P5VY . You are receiving this because you commented.Message ID: @.***>

[petterreinholdtsen]

[Sebastian Kuzminsky]

So we're kind of stuck. The only option with a realtime kernel is the official debian images (or our linuxcnc-specific fork of them), but that doesn't have good DTB support, so it's fairly limited in hardware support. We have working SPI (via hm2_rpspi) so maybe that's not a show-stopper, but for example mesaflash won't be able to talk to SPI boards. (It will only be able to talk to Ethernet boards.)

Where is the bugs.debian.org request asking for DTB improvement for the debian images?

-- Happy hacking Petter Reinholdtsen

[rodw-au]

The feedback I have had is that the static is caused by poor HDMI connection or insufficient power supply. a 5V USB is not enough. My Pi's have PoE Hats and I have not seen static. 53 volts seems to fix it!.

My Pull request #2496 documents how to install using the Debian packages. One tester had static but resolved it.

Latency is around 90k, One user on the forum has reported 60k using the 6.3 kernel. On AMD64, the 6.3 kernel has significant performance improvements so I suspect most of the improvements are due to the later kernel.

[rodw-au]

David Smith built a Pi image including the 6.0 RT kernel. I have built it. There is a matching forum thread somewhere. His repo is here https://github.com/djsftree/rpi-img-builder/tree/master The challenge at the time was to build VTK. I did this on the Pi and it took 5-6 hours. But the wheel can be precompiled. I think David offers one

[SebKuzminsky]

Does this kernel have both Preempt-RT and the Raspberry dynamic device tree overlays? If not, I think the regular debian.org linux-image-arm64-rt is a better option.

[rodw-au]

I don't know anything about the Raspberry device tree overlays but I suspect they are included from the starting upstream repo. From upstream, it seems it has advanced to the 6.3 kernel which is desirable But yes preempt_rt is included. You can see the RT patches under /userpatches You might be better off starting from upstream at https://github.com/pyavitz/rpi-img-builder and grab the RT patches Hopefully VTK is now available as a package.

[rodw-au]

Coming back to me. Here is the forum thread https://forum.linuxcnc.org/9-installing-linuxcnc/45912-rpi4-debian-linuxcnc-ethercat-xfce?start=0 Also, this file contains the scripting to install linuxcnc and other parts. https://github.com/djsftree/rpi-img-builder/blob/master/files/userscripts/uscripts We did get the latency down to about 50k and 107k overnight. Still useable https://forum.linuxcnc.org/9-installing-linuxcnc/45912-rpi4-debian-linuxcnc-ethercat-xfce?start=40#243333

Good luck!

[rodw-au]

I think this issue can be closed. I built a raspberry pi installer for linuxcnc on Debian 12 Bookworm. I forked Raspberry's installer build repo https://github.com/rodw-au/rpi-img-builder-lcnc/tree/linuxcnc So kernel source come from raspberry's repository. It would be good if this can be forked by Linuxcnc. It has 6.1.46 PREEMPT_RT kernel and Linuxcnc built and installed from the latest Buildbot2 source Device tree overlays etc.XFCE desktop. isolcpus set correctly. As its a crosscompiled environment and fully scripted (in three parts), It would lend itself to installing on a buildbot. You could save the debs Any questions, preferably, please raise an issue on my repo and ask them there as I am only a novice with git.

[andypugh]

@SebKuzminsky As you opened this, are you now happy with closing it?

[rodw-au]

It would be good if you guys could test my work. I seem to have had a regression and am rebuilding it again tonght. a forum member reported he did not get PREEMPT_RT and it was not booting for me.

I had set a custom defconfig to avoid opening menuconfig and electing the fully preemptible kernel all the time so I am building with menuconfig now. eg. set MENUCONFIG=1, CUSTOM_DEFCONFIG=0 in userdata.txt. Apparently you are then meant to be able to copy and rename the generated defconfig to the defconfig folder and invert those values to get a hands off build.

Anyway, teething problems and all, it definitely works if the environment is set right!

Settings for SPI etc are user configurable in /boot/broadcom/config.txt

[rodw-au]

Update: PREEMPT_RT and headers now here https://drive.google.com/drive/folders/187Nx4a4zQdHz61hzL3eIRZ59FEslnQ4B?usp=sharing

Onto ROOTFS build

[rodw-au]

My Pi Image builder for Debian Bookworm is now complete. https://github.com/rodw-au/rpi-img-builder-lcnc/tree/linuxcnc Linuxcnc debs are saved in output//linuxcnc Quite a few have tested this on the forum https://forum.linuxcnc.org/9-installing-linuxcnc/39779-rpi4-raspbian-64-bit-linuxcnc?start=290#280351

Note that pi-gen https://github.com/LinuxCNC/pi-gen used currently to generate linuxcnc images only supports up to Debian Buster so a new installer is required for a Bookworm image.

I am very grateful for all of the upstream support as significant changes were required to allow this to be so comprehensive and easy to use.

Note there is not enough room for the docs due to an 8 gb limit on the ROOTFS partition

[rodw-au]

As far as I can see my imager is now production ready and well tested. Ref: https://github.com/rodw-au/rpi-img-builder-lcnc/tree/linuxcnc I've added a number of linuxcnc specific flags in userdata.txt to control which branch to build, whether to include the docs etc. So when andy's now tested GPIO driver is back ported to 2.9, that is supported

Note my defaults use kernel version 6.1.46 but overnight, the moons aligned and raspberry pi advanced to 6.1.54 and the RT team released a patch for that version. I will update and rebuild in hte next couple of days.

My recommendation is that one of you guys fork my builder into the linxcnc repository and close this issue.

[andypugh]

OK, forked into LinuxCNC. I guess we will need to see if it works with the new Pi5. I have one on pre-order.

[rodw-au]

Thanks Andy, I am hoping upstream will solve the Pi5 support before we need to. I could not see how to raise an issue on your new fork, I wondered if the linuxcnc branch should be pushed to master. Linuxcnc does not need the plain pi image builder.