After updating 32 bit Raspi OS Pi 4 is on 64bit kernel

[edegroot-nl]

After updating Raspberry PI OS to the last kernel update, the kernel is switched to 64 bit.

$ uname -a Linux 5.15.84-v7l+ #1613 SMP Thu Jan 5 12:01:26 GMT 2023 armv7l GNU/Linux

$ sudo apt update .... $ sudo apt upgrade

The following packages will be upgraded: curl libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libcamera-apps-lite libcamera0 libcurl3-gnutls libcurl4 libnss3 libtiff5 linux-libc-dev raspberrypi-bootloader raspberrypi-kernel raspi-config vcdbg 16 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 108 MB/111 MB of archives. After this operation, 3,879 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://archive.raspberrypi.org/debian bullseye/main armhf raspberrypi-kernel armhf 1:1.20230317-1 [102 MB] Get:2 http://archive.raspberrypi.org/debian bullseye/main armhf linux-libc-dev armhf 1:1.20230317-1 [1,119 kB] Get:3 http://archive.raspberrypi.org/debian bullseye/main armhf raspberrypi-bootloader armhf 1:1.20230317-1 [4,542 kB] Get:4 http://archive.raspberrypi.org/debian bullseye/main armhf vcdbg armhf 1:1.20230317-1 [213 kB] ...... Setting up raspberrypi-kernel (1:1.20230317-1) .. .....

$ sudo reboot .... $ uname -a Linux 6.1.19-v8+ #1637 SMP PREEMPT Tue Mar 14 11:11:47 GMT 2023 aarch64 GNU/Linux

For me this is a problem while running some software that needs 32bit.

arm_64bit=0 added tot /boot/config.txt and everything fine now. Why not respect the 32bit kernel when already active and change the setting to arm_64bit=1 if needed?

[pelwell]

The switch to running a 64-bit kernel by default on Pi 4 is intentional. We believe it gives a better experience with few drawbacks. As you've found, you can revert to the 32-bit kernel by adding arm_64bit=0 to config.txt.

See https://forums.raspberrypi.com/viewtopic.php?p=2088935#p2088935 for more details.

Which software needs to be run on a 32-bit kernel?

[edegroot-nl]

Which software needs to be run on a 32-bit kernel?

Unifi Controller with MongoDB and Keepalived for example.

[pelwell]

What happens when you try to run them? Do you know the technical limitations?

[edegroot-nl]

What happens when you try to run them? Do you know the technical limitations?

Hi Phil, For MongoDB quite old version but needed by unifi:

It gets an bus error. I don't know the limitations.

[2023-03-18T14:54:20,673] INFO db - DbServer stopped [2023-03-18T14:54:24,850] INFO db - Sat Mar 18 14:54:24.849 [2023-03-18T14:54:24,853] INFO db - Sat Mar 18 14:54:24.850 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability. [2023-03-18T14:54:24,854] INFO db - Sat Mar 18 14:54:24.850 [2023-03-18T14:54:24,854] INFO db - Sat Mar 18 14:54:24.854 Invalid access at address: 0x1c076ee from thread: [2023-03-18T14:54:24,855] INFO db - [2023-03-18T14:54:24,855] INFO db - Sat Mar 18 14:54:24.854 Got signal: 7 (Bus error).

Keepalived is the default version from the repo:

Mar 18 12:29:15 edg-rpi03 Keepalived[4765]: pid 13817 exited due to signal 7 (Bus error) Mar 18 12:29:15 edg-rpi03 Keepalived[4765]: VRRP child process(13817) died: Respawning Mar 18 12:29:15 edg-rpi03 Keepalived[4765]: Restart of VRRP process delayed 60 seconds to limit respawn rate

II'm fine with this solution to go back to 32 bit right now.

[soren121]

@edegroot-nl You must be using an old version of UniFi Controller!

You can use current versions of UniFi Controller (7.3+ at least) with MongoDB 3.6+, which is available for aarch64. See this project for more info: https://github.com/ryansch/docker-unifi-rpi

[edegroot-nl]

@soren121

I'm using UniFi Network Application 7.3.83 stable released January 30th 2023, which is still supported with an older MongoDB version. The default version from the 32-bit repo.

I will install the standard 64-bit version of RaspBerry PI OS later if everything runs without issues with aarch64 packages.

[ralphrmartin]

The real problem is that people running production systems rely on updates to fix security problems, making unannounced changes of this kind which break things a real problem. Fortunately, there is a fallback, but it would have been better to let people know about the change and tell them how to stay on a 32 bit kernel.

[VirusPilot]

I am also not exited about this change. Espressif esp32 tools are only available for 32bit platforms and a usual apt upgrade just renders the setup unusable, unless this is fixed as mentioned above.

[pelwell]

It what way are the esp32 tools dependent on a 32-bit kernel? Very few 32-bit apps should care what mode the kernel is running in.

[Tylli]

After putting arm_64bit=0 into config.txt the whole system is unable to boot. I removed the statement and the Pi boots fine but PIVCCU3 is not able to work because of missing kernel modules which are not (and most likely will never be) compatible with a 64bit kernel. Any suggestions why the raspberry doesn't boot anymore? The kernel modules were automatically removed after apt-get upgrade.

[VirusPilot]

It what way are the esp32 tools dependent on a 32-bit kernel? Very few 32-bit apps should care what mode the kernel is running in.

The esp32 tools are simply parsing $arch and if the result is aarch64, they won't work.

[STMAndreas]

PortAudio isn't working properly after this update. With adding arm_64bit=0 into config.txt our application (using portaudio) is working but without it will crash. The problem we have ist that port audio will report different count of devices with Pa_GetDeviceCount(). In our logs we see after calling Pa_Initialize() several time this failure: Expression 'GetExactSampleRate( hwParams, &defaultSr )' failed in 'src/hostapi/alsa/pa_linux_alsa.c', line: 895 So it seem this change has more side effects than expected.

[fabiopuddu]

Another example: GlusterFS fails with SIGBUS - it took me three nights to figure out what was wrong - luckily I noticed something was wrong before updating all nodes. Definitely not a great idea to push this update without a "breaking changes" notice.

[AndyA]

Doesn't this change mean that uname -m is returning the wrong value? libc etc are still 32 bit so any build that attempts to detect the architecture using uname will fail, right?

[AndyA]

OK, it's returning the correct value for the kernel - but that doesn't seem super helpful in userspace.

[TheHoolio]

@edegroot-nl thanks a lot for posting the keepalived issue on here. I would've never figured out what broke it. Can anyone point me into the right direction on how to get it back running when I wanna stick with the 64-bit kernel?

[popcornmix]

Doesn't this change mean that uname -m is returning the wrong value? libc etc are still 32 bit so any build that attempts to detect the architecture using uname will fail, right?

uname asks about the kernel. It's the wrong tool for this job.

getconf LONG_BIT

would be a better way to query if userland is 32 or 64 bit.

[HinTak]

@pelwell does the 32-bit OS contains all the userspace tools to build out-of-tree kernel modules in 64-bit?

[pelwell]

You should just need sudo apt-get install gcc-aarch64-linux-gnu, but I don't build out-of-tree modules so there might be something else to install.

[HinTak]

@pelwell thanks. There are also hints that matching kernel header package isn't available via apt-get. I see it on archive.raspberrypi.org, but is there an oversight somewhere on this - ie the arm64 kernel headers package isn't accessible via apt-get?

[popcornmix]

@XECDesign are arm64 kernel headers available on 32-bit RPiOS through apt?

[XECDesign]

@XECDesign are arm64 kernel headers available on 32-bit RPiOS through apt?

Unfortunately not yet. https://github.com/raspberrypi/linux/issues/5408

[popcornmix]

Until that package is available it may be possible to use rpi-source

[ebaauw]

This change breaks installation scripts that rely on uname -m to decide whether to install 32-bit or 64-bit software. What's the supported way to determine whether you're running the 64-bit kernel on a 32-bit OS vs a 64-bit OS?

[HinTak]

@ebaauw I think one of apt- or deb- family has a --archtecture query option. It still surprises me somewhat, how a 64-bit kernel get passed that, but the header package does not.

[pelwell]

What's the supported way to determine whether you're running the 64-bit kernel on a 32-bit OS vs a 64-bit OS?

Already answered in this thread.

[ebaauw]

Sorry, missed that. Thanks!

[oilcan-productions]

homebridge fails as well

[popcornmix]

@oilcan-productions "fails" doesn't help much.

Describe exactly what you are doing, what should happen, what currently happens, and if setting arm_64bit=0 avoids the issue.

[oilcan-productions]

@oilcan-productions "fails" doesn't help much.

Describe exactly what you are doing, what should happen, what currently happens, and if setting arm_64bit=0 avoids the issue.

https://github.com/homebridge/homebridge/issues/3349

and yes it fixes the issue for homebridge. I do agree with the sentiment of others that a breaking change like that should not happen

[ebaauw]

homebridge fails as well

Please keep that discussion to the issue in the homebridge repo, linked above. No need to involve the good people here.

[diehummel]

@oilcan-productions "fails" doesn't help much.

Describe exactly what you are doing, what should happen, what currently happens, and if setting arm_64bit=0 avoids the issue.

Holy... setting it to 0 solved my issue with the meshcentral/meshagent. Glad i found this post.

[mbentley]

Also reporting that the TP-Link Omada Controller for armv7l relies on an ancient 32 bit MongoDB since they stopped supporting 32 bit MongoDB for arm and this change does have some unintended consequences for users of the controller since MongoDB will not start in both a Docker container and natively on the host as reported here. At least there is a fallback to forcing a 32 bit kernel but it isn't immediately obvious since people will only see it once they reboot to a new kernel which doesn't always happen immediately after updates.

[chrisovergaauw]

The real problem is that people running production systems rely on updates to fix security problems, making unannounced changes of this kind which break things a real problem. Fortunately, there is a fallback, but it would have been better to let people know about the change and tell them how to stay on a 32 bit kernel.

This cannot be emphasised enough. This change was a terrible, terrible decision and should be reverted ASAP. The havoc you're creating is immense. A breaking change like this should be opt-in, not opt-out.

[EDIflyer]

Thank you @diehummel for flagging this thread - I've spent hours trying to work out why MeshCentral had stopped working properly on my Pi 4! (https://github.com/Ylianst/MeshAgent/issues/183)

Applied the arm_64bit=0 flag to /boot/config.txt and rebooted and lo-and-behold all back and running again!

[jsiverskog]

for whatever reason RME ADI-2 Pro (soundcard) doesn't work without arm_64bit=0. it shows up on lsusb but alsa doesn't seem to find it. in addition to a number of other unexpected issues caused by this change.

[6by9]

for whatever reason RME ADI-2 Pro (soundcard) doesn't work without arm_64bit=0. it shows up on lsusb but alsa doesn't seem to find it. in addition to a number of other unexpected issues caused by this change.

What does lsusb -t report the driver to be on the two kernels? My initial guess is that the driver is missing on 64 bit, although most would use the standard USB audio driver (name escapes me).

[jsiverskog]

@6by9 : they look very similar, 32 bit:

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 2: Dev 100, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 2: Dev 101, If 0, Class=Hub, Driver=hub/4p, 480M
                |__ Port 4: Dev 103, If 0, Class=Communications, Driver=cdc_acm, 12M
                |__ Port 4: Dev 103, If 1, Class=CDC Data, Driver=cdc_acm, 12M
            |__ Port 3: Dev 112, If 0, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 112, If 1, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 112, If 2, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 112, If 3, Class=Vendor Specific Class, Driver=, 480M
            |__ Port 4: Dev 102, If 0, Class=Human Interface Device, Driver=usbhid, 12M

64 bit:

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 2: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
                |__ Port 4: Dev 7, If 0, Class=Communications, Driver=cdc_acm, 12M
                |__ Port 4: Dev 7, If 1, Class=CDC Data, Driver=cdc_acm, 12M
            |__ Port 3: Dev 5, If 0, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 5, If 1, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 5, If 2, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 5, If 3, Class=Vendor Specific Class, Driver=, 480M
            |__ Port 4: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M

[6by9]

@jsiverskog Seeing as both list snd-usb-audio as being loaded for the endpoints, that looks reasonable to me.

I don't have one of these soundcards, so can't debug further. For more extensive debugging it would be worth opening a new issue (under Linux please) or forum thread to cover it. First things to check would be for error messages in the kernel log (dmesg), and whether it works on the 64bit version of Raspberry Pi OS.

[2play]

The switch to running a 64-bit kernel by default on Pi 4 is intentional. We believe it gives a better experience with few drawbacks. As you've found, you can revert to the 32-bit kernel by adding arm_64bit=0 to config.txt.

See https://forums.raspberrypi.com/viewtopic.php?p=2088935#p2088935 for more details.

Which software needs to be run on a 32-bit kernel?

Is there a forced changed after a specific kernel version? Im testing update and full-upgrade on a buster 5.10.103-v7l+ and there are no upgrades or changes. Pi4 seems not finding even latest May firmware, eeprom, or few days ago kernels

sources are default. Any feedback?

[6by9]

Is there a forced changed after a specific kernel version? Im testing update and full-upgrade on a buster 5.10.103-v7l+ and there are no upgrades or changes. Pi4 seems not finding even latest May firmware, eeprom, or few days ago kernels

That presumably means your on Buster / Raspberry Pi OS (Legacy), as Bullseye is now on 6.1.21. Buster / Legacy does not get significant updates, as per the original blog post about it - https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/

This is a firmware change, not kernel, although the two are generally shipped together.

[jefwu72]

After putting arm_64bit=0 into config.txt the whole system is unable to boot. I removed the statement and the Pi boots fine but PIVCCU3 is not able to work because of missing kernel modules which are not (and most likely will never be) compatible with a 64bit kernel. Any suggestions why the raspberry doesn't boot anymore? The kernel modules were automatically removed after apt-get upgrade.

Any solution to this booting issue? I am a user of Omada controller. After meeting arm_64bit=0, the Pi doesn't boot at all. Any workaround to this?

[XECDesign]

@jefwu72, what image are you starting with? The first result I get when I look up "omada controller raspberry pi" says to start with the 64bit image. If you try to use that with arm_64bit=0, that just won't work.

[si458]

im just wondering if people are still having this issue? ive just done a complete FRESH install of raspbian bookworm 64bit on both my pi 3b+ and my pi4, and its now installing perfectly fine? i even checked the config.txt and arm_64bit=1 was set by default now? only issue is you MUST switch from wayland to x11 with raspi-config to get the Desktop tab to work/show

my mesh agent is showing as 'Linux ARM 64 bit (glibc/2.24 NOKVM)' as well rather than the normal 'Linux ARM 64 bit'

[mbentley]

If you're doing a fresh install starting with 64 bit Raspbian, you're going to get packages (binaries & libraries) that are compiled for 64 bit as well as a 64 bit kernel. If you started with a 32 bit system and had your kernel switch from 32 bit to 64 bit, you'd then have 32 bit packages with a 64 bit kernel so you wouldn't run into 32 bit binary with 64 bit kernel issues.

[si458]

thanks @mbentley! just realised i posted in the wrong repo haha my bad!

[dougle03]

I just followed your advice to amend to 'arm_64bit=0' and now my Pi won't boot.. Totally stuck...

[dougle03]

@mbentley Scratch that. Reverted arm_64bit=0 back to arm_64bit=1 and Pi will now boot, but still stuck with the original problem. Any other solutions?

[pelwell]

@Braum I hope you've realised that the 32-bit OS still comes with a 64-bit kernel, and that the unavailability of the 64-bit OS that you reported in the now-deleted comment should not be a blocker.

[artofcomputing]

@pelwell

If you check http://archive.raspberrypi.org/debian/dists you will notice the availability of aarch64 content files under the /debian/dists/bookworm

Using https://mirrors.tuna.tsinghua.edu.cn/raspbian/raspbian/ as an example we can clearly see that such aarch64 content files are not available under the same directory /dists/bookworm/

However, these content files are present on /raspbian/multiarch/dists/bookworm/, making it possible to download arm64

To solve it, it requires to explicitly inform which arch is available under which directory if you need to use an external mirror, requiring us to add two entries on raspi.list

deb [arch=arm64] http://mirrors.tuna.tsinghua.edu.cn/raspbian/multiarch/ bookworm main deb [arch=armhf] http://mirrors.tuna.tsinghua.edu.cn/raspbian/raspbian/ bookworm main

Anyways, users will require to understand mirror behaviour if using armhf or arm64 as their OS arch and in areas where the main repo is not reachable, but yeah not a blocker for this context.