[Bullseye] Chromium fails to start on ARMv6

[MichaIng]

The chromium-browser v92 package from the RPi Bullseye repository seems to not run on ARMv6, throwing an "illegal instruction" error, while it works on ARMv7/8 models. I tested as well the older -rpt1 build after recognising that with FFmpeg there was a new build -rpt2 uploaded just yesterday: The old one fails the same way, so it is not related to the last recent update.

The error matches quite well to what we see when a binary was build with ARMv7 (Debian armhf) toolchain 🤔.

[XECDesign]

Aside from just running, were you actually finding it to be usable in real world scenarios on armv6 pis?

I think due to limited RAM on armv6-only platforms, it might be time to drop support.

@ghollingworth Should I get it running, or should I just add something to the launcher script to notify the user that it's no longer supported?

[MichaIng]

We thought about the same question. IME especially the kiosk mode is something that is often used with small SBCs and performs pretty well. And in case of Raspberry Pi OS, Chromium is shipped OOTB with the RPi desktop, so it wouldn't be nice to ship an image with a broken core component on RPi 1 and Zero 🤔.

[ghollingworth]

We're definitely reaching the point where there isn't enough memory on RPi1 and Zero to use Chromium without it soon crashing...

It might be better to default to a different browser on those targets.

[MichaIng]

Has the Chromium memory consumption increased so that it is an issue now or becomes one while it wasn't before?

[ghollingworth]

The memory usage of Chromium is unrestricted and is specifically designed to work well in a system with a supply of swap space. Since we heavily restrict swap on SD cards this doesn't help and opening tabs will generally end up with an 'Oh snap' message which means one of the threads failed a memory allocation.

Things like Facebook, youtube etc. seem to continuously use more javascript and complex code which again increases memory allocations, plus extra functionalities in Chromium tend to have a higher basic footprint (although we've not measured this, I believe this is the case)

[MichaIng]

Well, but that was never different, so you mean Chromium should have never been made the default? It is btw trivial to attach a USB drive/stick and create a sufficiently large swap file there. It is also possible to configure Chromium to use less memory, finally it depends on the browsing behaviour as well, and as said, using it in kiosk or app mode is what works pretty well on RPi 1/Zero, and naturally keeps memory usage in borders since it's a single website opened only (usually).

Yeah, not sure, I mean it is clear that on RPi 1/Zero one won't have an awesome browsing experience, with any web browser, but which browser shall be the alternative? Shipping a dedicated RPi OS desktop image with a different pre-installed browser is not an option for maintenance reasons I'd say.

[XECDesign]

It wasn't always as bad as it is now. Web sites do get heavier over time. I'd like to believe that browsers are getting more optimised and should run better, but that doesn't seem to be the case. I remember testing builds on a pi zero, being able to open youtube, play videos and browse a bit. Last time I tried this, it could barely load google.com by the time I finished my coffee.

My approach wouldn't be to ship with an alternative browser. Most don't launch at all and the ones that do are not much better than chromium. I'd only provide a message saying chromium is not supported on this platform. Maybe offer to install Web instead, which IIRC is the only other one that ran and was somewhat usable.

For kiosk applications, there should be plenty of distros dedicated for that exact purpose.

Sure, maybe some people can optimise it to run better, but most of posts I've seen is just stuff people have cargo-culted together by fiddling with flags that don't do anything or make things worse. If you have some tweaks that make things measurably better and actually usable, we could take a look at integrating them.

The thing to consider is how much time should we be spending on armv6 browser support given all of this. It doesn't come for free and if the only benefit is that 5 people can have an awful browser experience and somebody else doesn't need to switch to a dedicated kiosk distro, I don't know if it's worth it.

That's why I asked, aside from hypothetical use cases, how many people are actually actively using chromium on 512MB devices. Maybe when we release bullseye we'll have a lot more reports and will need to reconsider, but right now it doesn't look like it.

[Farhankhosravi]

Is there really NO way to use chromium on RP0w bullseye? I was able to use it on buster, just few weeks ago! it wasn't fast, but working... just want same experience on bullseye... please change your decision; a slow Browser is better than nothing.

[JamesH65]

I suggest you remain on Buster, this will continue to have security updates so should still be good to use for a year or two. Chrome on Bullseye on Zero is pretty much unusable due to low memory and CPU performance.

[MichaIng]

I cannot imagine that the hardware requirements increased so much from Chromium 92 (currently shipped on Buster) to v95 (currently shipped with Bullseye). It was always slow on RPi 1/Zero models, but whether it is sufficient or not obviously lies in the eye of the individual user:

it wasn't fast, but working...

So I would vote against using it as argument here. Chromium (based) browsers are still the no. 1 choice for most users, even when other browsers may run quicker. Not sure what the underlying issue is, so of course, so if it is too difficult/impossible to get it running on ARMv6, that we need to live with it.

@Farhankhosravi Btw, generally Firefox runs better on those models, never was different 😉:

apt install firefox-esr

[JamesH65]

AIUI, there are extra loads on the system from Bullseye itself, the move the KMS, libcamera etc etc. When you combine that with the loads created by modern webbrowsers, Chrome on Zero on Bullseye is unusable - pages crash with out of memory errors all the time and its horribly slow. When it comes down to it, the Zero just doesn't have enough grunt to run Bullseye and Chrome together.

[MichaIng]

Bullseye alone does not imply additional system load, at least not on our tests with a bare system, but KMS and libcamera/V4L2 decoders are probably an argument.

Are there plans or ideas to address this package or RPi OS desktop image wise? I mean it does now will cause future user reports when Chromium is the shipped default browser with the desktop image or installing the package and trying to run it causes failures, and RPi Zero (1) is still a very much used RPi model: After RPi 3 (non-plus) and 4 the it is the most used one with DietPi, ~7% of all DietPi systems run on RPi Zero W, another percent on RPi Zero (non-W), and I guess it is similar with RPi OS.

[Farhankhosravi]

I cannot imagine that the hardware requirements increased so much from Chromium 92 (currently shipped on Buster) to v95 (currently shipped with Bullseye). It was always slow on RPi 1/Zero models, but whether it is sufficient or not obviously lies in the eye of the individual user:

it wasn't fast, but working...

So I would vote against using it as argument here. Chromium (based) browsers are still the no. 1 choice for most users, even when other browsers may run quicker. Not sure what the underlying issue is, so of course, so if it is too difficult/impossible to get it running on ARMv6, that we need to live with it.

@Farhankhosravi Btw, generally Firefox runs better on those models, never was different 😉:

apt install firefox-esr

Just to let you know: Firefox says "illegal instruction" on RP0, too.. 🥲

[MichaIng]

Can you try:

apt install libatomic1

That was missing in another case already. Background is that Raspbian uses armhf package architecture, which on Debian refers to ARMv7 at least, where libatomic1 is often not required. ARMv6 however requires it in cases, which is addressed on Debian via armel/mipsel package arch dependencies, but again, Raspbian is armhf. If this turns out to be the issue, we can report to Raspbian bug tracker: https://bugs.launchpad.net/raspbian

[Farhankhosravi]

AIUI, there are extra loads on the system from Bullseye itself, the move the KMS, libcamera etc etc. When you combine that with the loads created by modern webbrowsers, Chrome on Zero on Bullseye is unusable - pages crash with out of memory errors all the time and its horribly slow. When it comes down to it, the Zero just doesn't have enough grunt to run Bullseye and Chrome together.

I don't think so; in my RP0W, bullseye is almost same with buster, they both use same amount of cpu and ram in idle, and make my pi warm equally.. even with some apps, i experience faster/smoother running on bullseye! And، your branded new pi (raspberry pi zero 2) has exactly same amount of ram (512mb) and has no problem with chrome on bullseye! This shows that our problem is ARMv6/v7, not ram. I think you can fix this issue easily, if you want to.

[Farhankhosravi]

Can you try:

apt install libatomic1

I tried that. firefox still says "illegal instruction" .

[MichaIng]

Nasty, I'll try to replicate on my Pi0W. Interesting that the errors are the same, as if it was the same underlying issue 🤔.

[MichaIng]

Bug report about Firefox submitted to Raspbian: https://bugs.launchpad.net/raspbian/+bug/1956152

Another alternative is the old Raspbian default browser:

apt install epiphany-browser

[XECDesign]

Well, I've uploaded a build that will run on armv6 (as long as you don't have libwidevinecdm0 installed). It's completely unusable, won't be supported and is likely to disappear if it becomes any more of a pain to build, but it's there.

[MichaIng]

Wow, that is nice, many thanks for doing this! I'll give it a shot on RPi Zero W (1) quickly, rotation through KMS/fKMS/legacy and Chromium renderers with full overclocking. Let's see whether there is any joy possible.

[MichaIng]

Okay I tested thoroughly on RPi Zero W (1) with all combinations of KMS, fKMS and legacy driver, with and without --use-gl=egl (or other value), where it falls back to EGL in case of KMS/fKMS and SwiftShader in case of legacy driver. In call cases, web pages behave extremely slowly, shuttering simple animations and scrolling. Also Chromium startup is extremely slow, typing in an address takes a second or two for every character, keeping CPU usage at 100%. Starting in kiosk mode with proper flags starts faster, and as long as the shown page does not have larger animations, it may be acceptable, e.g. a slowly updated monitoring page or so, but any "moving" animation and scrolling stays nearly as bad as from desktop.

I gave up to test video playback as loading related websites would have taken ages.

I tested with 1920x1200 resolution, so with a much lower resolution on a small LCD panel it should be much better, but also I used max stable overclocking with 1100/450/450 MHz, i.e. especially on RPi 1 it will be worse. Also I used all flags I know for minimal resource usage, low-end hardware and skipping compositing/antialiasing.

So while there may be a minimal use case with kiosk mode on LCD panels, maintaining Chromium for this doesn't seem reasonable. Interesting would be an alternative faster browser with a kiosk mode feature, or one which is designed for this explicitly without the overhead of a fully fledged GUI and all its features.

---

Just to assure, the RPi 1 does not have any additional GPU/SoC features, hardware or firmware/device tree wise, over RPi Zero (1), i.e. lower performance in any case due to lower (default) ARM frequency, right?

[popcornmix]

A Pi1 overclocked to match Pi0 will have the same performance. There is no hardware difference that affects performance (just pi0 gets factory tested at a higher clock, so we are happier enabling that by default).

[popcornmix]

Thanks for testing - it matched what we saw, that chromium on Pi0/Pi1 is not really feasible in the general case. Sure, a very basic web site may be usable, but they are becoming increasing rare.

[MichaIng]

I wonder whether it makes sense to add e.g. a preinst check for armv6l to warn/abort or do an interactive debconf/whiptail prompt/choice noting that it won't run (nicely), to inform and prevent end user reports in the first place. A generic prompt could be added to other such cases, like Firefox and OpenJDK 11+ (maintained by Raspbian, not RPi repo, I know), which I guess will increase over time. On the other hand there seems to be no too large number of reports here, and we disabled Chromium and Firefox for those models on DietPi end already.

[rxhfcy]

Also I used all flags I know for minimal resource usage, low-end hardware and skipping compositing/antialiasing.

I hope this isn't too off topic, but which flags are you talking about specifically? I want to continue being able to show very simple and static web pages on a Raspi 1, and I'm currently using chromium-browser --kiosk --incognito (url), any other concrete suggestions? Thanks!

[MichaIng]

--no-sandbox --disable-smooth-scrolling --disable-low-res-tiling --enable-low-end-device-mode --disable-composited-antialiasing

[whitpa]

FYI our (ThinLinX's) customers have encountered this issue too, but in Raspbian Buster rather than Raspbian Bullseye. Testing using debs provided by both Ubuntu and Raspbian strongly suggests that something broke after Chromium 72 (72 works OK, 73 and later encounters an illegal instruction on ARMv6 hardware). The problem is clearly in Chromium itself and not in its shared library dependencies (e.g. an Ubuntu build of Chromium 73 will get the same error when transplanted into a context in which Chromium 72 is known to run OK, e.g. Raspbian Stretch).

As far as we can determine, this is not a RAM exhaustion issue in disguise. Changing the RAM/VRAM split via gpu_mem in config.txt to provide more application RAM at the expense of VRAM - or vice versa - makes no difference (although no ARMv6-based Pi with RAM > 512 MiB exists, so our ability to test this is limited). Adding swap space also makes no difference.

We suspect that either there is an assembly source file somewhere in the Chromium source with an inappropriate #ifdef guard that is causing ARMv7 instructions to be added, or that there is some erroneous runtime code selection behaviour that amounts to the same thing.

[MichaIng]

@whitpa Does the new build not work for you? Obviously the issue has been identified and solved already. But as of my tests above, it is way too slow to be practically usable on any ARMv6 RPi model.

[whitpa]

@MichaIng,

It wasn't obvious to me from above comments that the issue had actually been identified and resolved (there does not seem to be any clear information on exactly what the fix entailed). I commented here because this forum is one of the only places this issue has been mentioned. If I'm not adding any new information, never mind, and thanks for fixing it.

Right now we're still using Buster, so any fix to the current Chromium version that nobody has ever tested in a userspace older than Bullseye is of little use to us. But we'll be upgrading soon, and if this fix trickles down into future packaged Debian/Raspbian releases, that's great.

[MichaIng]

It wasn't obvious to me from above comments that the issue had actually been identified and resolved

At least @XECDesign wrote about the fixed build (and may be able to give some details about what the issue was/how it was fixed) and I wrote about my successful tests with it, but bad performance experience 😄.

Right now we're still using Buster

With Chromium 92 pushed to the Buster suite, I guess the same issue that previously affected Bullseye only was introduced on Buster as well. And the fixed rebuild was probably only uploaded to Bullseye.

However, IMHO it is not worth it to waste any further though on this topic. As said, I tested with the fastest and maximum overclocked ARMv6 RPi available, and it is a horrible, unusable experience. If you watch a single static page in kiosk mode, with some changing characters at best (a counter or so), it may be acceptable, but for anything more it doesn't make sense.

[XECDesign]

(and may be able to give some details about what the issue was/how it was fixed

There are multiple builds included in the .deb. One is armv6 and and other is armv7. The launcher wrapper determines which one to actualy launch. There are additional patches to make the armv6 version work and load the correct armv6-specific files.