search

Ylianst / MeshCentral

A complete web-based remote monitoring and management web site. Once setup you can install agents and perform remote desktop session to devices on the local network or over the Internet.
https://meshcentral.com
Apache License 2.0
4.32k stars 575 forks source link

Raspberry Pi 4, 64 Bit kernel and 32 Bit Userland #5108

Open diehummel opened 1 year ago

diehummel commented 1 year ago

Describe the bug Not a bug itself with Meshcentral/Meshagent, but will hit many raspberry pi users. On Raspberry Pi 4 with an update the kernel changed to aarch64. The meshagent then gets killed with a "BUS" error when it starts. After around 8 retries when it starts, the console is not really connected. No console commads work. Not Desktop and Terminal Tab is available.

Programs using uname -m to detect the architecture get a problem. The download script is loading the 64 bit binary, which will not work. Source: https://github.com/raspberrypi/firmware/issues/1795

Using arm_64bit=0 in /boot/config.txt will fix it, when using the 32bit OS.

dinger1986 commented 1 year ago

Can you submit a PR to fix?

diehummel commented 1 year ago

Not sure if this can be fixed from Meshcentral. Because even if the install script will detect aarch64 and use getconf LONG_BIT to check if it is really 32 or 64 bit, the 32 bit agent will not work on the aarch64 kernel.

Not sure if the agent can be compiled as 32 bit able to run correct on aarch64 and all packages installed as armhf.

EDIflyer commented 1 year ago

Oooh thank you @diehummel - this explains why I'm seeing problems on my Pi 4 that's running bullseye and is now aarch64 but not on my Pi 400 that's still on buster armv7l! It had been working fine - I hadn't realised there'd been a kernel update that changed things. I was finding I could still run console/terminal/files but desktop wouldn't work. I tried uninstalling/reinstalling but it just made things worse with the agent recurrently restarting itself and showing the bus error if run manually! (https://github.com/Ylianst/MeshAgent/issues/183)

Did you have any luck finding a fix?

EDIflyer commented 1 year ago

Ah - don't worry, just saw your post in the Raspberry Pi repo and confirm adding the arm_64bit=0 flag to /boot/config.txt and rebooting did the trick for me too. Thanks for hunting that workaround down! Agree would be good if there was a recompiled version of MeshAgent to solve this but reassured to have it back and working for now at least :)

pelwell commented 1 year ago

Hi @diehummel, I'm one of the Raspberry Pi maintainers.

the 32 bit agent will not work on the aarch64 kernel

Are you certain about this? Very few apps care that the kernel is 64-bit. If you have examples of incompatibilities I'd be interested in investigating why.

diehummel commented 1 year ago

@pelwell For me it only worked, when the agent was installed with the 32 bit kernel before the "painful" update. After os patching and switching to the aarch64 kernel it worked, i think, just until the agent got updated. Since then the agent will be displayed as online, but there is no RDP or terminal nor a console function anymore. Reinstall manually does not solve it. Same behavier.

So yes, i am sure thew 32 bit version of the agent did not work correctly with the aarch64 kernel and 32 bit userland. I don't know of other apps, but i have read in the raspberry pi repo (link in first post) from other users reporting issues with other apps.

EDIflyer commented 1 year ago

Hi @pelwell - definitely didn't work for me either, see the linked bug on here for the variety of errors I experienced. As soon as I added the flag that @diehummel mentioned it went back to normal. This was after me trying reinstallation too.

pelwell commented 1 year ago

From a clean installation (with arm_64bit=0), what's the least I need to do to demonstrate the problem (with arm_64bit=1)? Bear in mind that I've never even heard of meshcentral before now.

pelwell commented 1 year ago

The web server is up and I've created an account, but that's all.

EDIflyer commented 1 year ago

Hi @pelwell haha - I appreciate the extent you're going to 😄 If you spin up Mesh Central on a server (Windows or Linux) then you want to add the agent to a Raspberry Pi. You click the 'Add Agent' and select the relevant options below: image

Without the arm_64bit=0 flag added then it installs OK but each time it tries to start if falls over again. Once the flag is added and you reboot it works OK.

Happy to jump onto a Microsoft Teams session, etc to show you or discuss further if at all helpful. MeshCentral itself is a really nice bit of software so would recommend it more generally, I'm using it to manage a variety of Windows and Raspberry Pi units (including a Pi20W).

pelwell commented 1 year ago

After downloading meshagent and installing it (sudo Downloads/meshagent), I used the GUI to wake it up. Then I rebooted into 64-bit mode, and the agent still wakes up, and I can open a terminal onto it in the GUI.

pelwell commented 1 year ago

But the remote desktop only works on the 32-bit kernel - that's something to work with.

EDIflyer commented 1 year ago

Thanks @pelwell for looking at this. Interesting to hear your experience. I had MeshAgent running OK on my Pi 4 for about 6 months and it used to work fine. It stopped working for desktop connections a couple of months ago (in retrospect when this kernel update was applied) but still worked OK for terminal/files. If I tried to connect to desktop the agent would crash and then reload.

In an attempt to fix the issue I uninstalled and reinstalled the Agent again from scratch- however I then found the service was continually restarting (you get the notifications at the top right of the MeshCentral page). Adding the flag above and rebooting solved it but I find it interesting that you were still able to access it in some way via 64-bit as on my fresh install it was resolutely not working (I suppose there could have been something residual left behind but I thought I cleared it out well).

pelwell commented 1 year ago

I posted a possible fix - it seems to work for me, but I'm not claiming this is the best solution - on the MeshAgent repo here: https://github.com/Ylianst/MeshAgent/issues/183#issuecomment-1569076318

In short, the agent code is doing unaligned accesses, and the arm64 kernel is less forgiving that the arm32 kernel. It isn't clear why this hasn't been an issue with an arm64 userspace.

si458 commented 1 year ago

TEMP FIX HERE, UNTIL NEW AGENT RELEASE: add arm_64bit=0 flag to /boot/config.txt ON YOUR Pi 32Bit OS

si458 commented 10 months ago

im just wondering if people are still having this issue? ive just done a complete FRESH install of raspbian bookworm 64bit on both my pi 3b+ and my pi4, and its now installing perfectly fine? i even checked the config.txt and arm_64bit=1 was set by default? only issue is you MUST switch from wayland to x11 with raspi-config to get the Desktop tab to work/show

EDIT: my mesh agent is showing as 'Linux ARM 64 bit (glibc/2.24 NOKVM)'