[BUG] Octoprint fails to start up on arm64 based Raspberry Pi because of "Janus quit! This should not happen"

[werdnum]

Describe the bug At some point, Octoprint stopped starting up correctly. When I looked at the logs, I found this:

2021-04-05 23:30:42,877 - octoprint.plugins.thespaghettidetective - ERROR - Backing off 4.631189 seconds: Janus quit! This should not happen. Exit code: 127
2021-04-05 23:30:45,750 - backoff - INFO - Backing off wait_for_janus(...) for 6.9s (ConnectionRefusedError: [Errno 111] Connection refused)

When I disabled the Spaghetti Detective plugin, this stopped happening.

Expected behaviour: This problem doesn't occur, or if it does, the plugin fails gracefully and does not block the startup of the whole Octoprint instance (and instead displays a helpful message explaining what "Janus" is, why it's important, and how one might go about debugging its failure).

To Reproduce I don't really know how to reproduce this, it just started happening. But I would start by running the Octoprint plugin on an aarch64 based Raspberry Pi.

In the last 24 hours, I have backed up my Octoprint installation and restored the backup to a new image based on arm64 -- but Octoprint + Spaghetti Detective were working fine after this. The immediate trigger was when I installed a new version of PSU Control (I had to bypass some GCC-10 compatibility issues by running CFLAGS="-fcommon" sudo pip install rpi.gpio to get that to install but this seems super unrelated).

Seems more likely to have been something latent in my configuration that inserted this bug, and restarting Octoprint 'activated' the bug. It seems likely that thing is enabling premium webcam streaming based on my investigation below.

When I dig into the code, it looks like you're running bin/janus/bin/run_janus.sh, and when I try running that myself, I get a mysterious "no such file or directory" error.

However, file tells me that this file is a ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=c01ab17e8cb14e9c24438b5f7d39e1084f86efbf, with debug_info, not stripped.

... Which makes sense, because it seems you've shipped a 32-bit armhf binary and assumed it will run on any device where pi_version() returns a truthy value - which seems to mean if you can find "Raspberry Pi" somewhere in /sys/firmware/devicetree/base/model.

It seems like you have a couple of options here:

1. Fix your logic to check for an armhf architecture instead of a raspberry pi specifically.
2. Ship janus binaries for a couple of architectures, or better yet download the appropriate release, or better yet build it somehow as part of installing the pip package (I'm kinda fuzzy on what that actually means in practice).
3. Add documentation for how to set up Spaghetti Detective without this Janus dependency, and what the implications of that choice would be.

Desktop (please complete the following information): N/A

Additional context I got somewhere nearby with this workaround:

sudo dpkg --add-architecture armhf
sudo apt update
sudo apt install libc6:armhf libnice10:armhf libconfig9:armhf

(there are probably more missing libs)

I also tried just installing regular old Janus from apt and symlinking it into the tree, but this didn't do the trick because there's a bunch of .so lib files shipped with the extension too, and those need to be rebuilt as well - though I'm not quite sure why this Janus thing can't bring its own libraries instead of you shipping them in an ostensibly Python pip package.

[werdnum]

(FWIW, manually setting disable_webcam_streaming in config.yaml under thespaghettidetective does not do the trick)

[Dids]

@werdnum Did you ever figure out a fix or workaround for this?

As far as I can tell, this is still an issue, unfortunately.

[werdnum]

Nope, just uninstalled it.

[kennethjiang]

@Dids @werdnum Not sure why I missed this issue.

When OctoPi 0.18 tried to support the 64bit version of raspbian (now called Pi OS) in their RC, they ran into too many issues and ended up only releasing with the 32-bit version. One of the compatibility issues was this.

We will try to fix it when the 64-bit Pi OS is more mature and more adopted. For now, you will need to install the official octopi 0.18 (32-bit) if you want the premium streaming to work. The plugin will still work you want to stick with the 64-bit version, but the premium streaming won't work.

[rerickson1]

@kennethjiang any chance you are going to look into 64-bit support soon? Octopi just released 1.0.0 RC1 and has a 64-bit image based on Ubuntu.

[kennethjiang]

@rerickson1 Thank you for alerting us on the availability of octopi 1.0 rc. We will definitely start testing it at some point. We may wait a little while - last time we jumped in testing 0.18 64-bit too soon and the time we spent (and arguably wasted) wasn't non-trivial given we are a tiny team... :(

[rerickson1]

We may wait a little while - last time we jumped in testing 0.18 64-bit too soon and the time we spent (and arguably wasted) wasn't non-trivial given we are a tiny team... :(

I can appreciate that. I did some initial testing and it seems to work pretty good. I had to uninstall the TSD plugin. I then noticed some issues with the webcam stream freezing when viewing it in Safari via the OctoPrint UI. I am streaming at 1080p 30 FPS. That issue doesn't happen on the 32-bit build.

[Dids]

@Dids @werdnum Not sure why I missed this issue.

When OctoPi 0.18 tried to support the 64bit version of raspbian (now called Pi OS) in their RC, they ran into too many issues and ended up only releasing with the 32-bit version. One of the compatibility issues was this.

We will try to fix it when the 64-bit Pi OS is more mature and more adopted. For now, you will need to install the official octopi 0.18 (32-bit) if you want the premium streaming to work. The plugin will still work you want to stick with the 64-bit version, but the premium streaming won't work.

When using arm64 and specifically disabling premiums treaming, the "Janus quit!" messages still keep filling the logs. Shouldn't TSD not attempt to connect to Janus if premium streaming is explicitly disabled?

EDIT: Just to provide more context, in my case I've containerized the entire setup on a Raspberry Pi 4, however I've also offloaded the primary webcam streaming to a more powerful machine, which is still on the same network, so both OctoPrint's webcam and Octolapse work surprisingly well with it, however of course TSD does not, as it can only in compatibility mode in this case. Still, I'd rather not have TSD keep trying to reconnect to Janus when effectively told not to (with premium streaming disabled), however I'm also working on hacking together a similarly "offloaded" streaming solution for TSD. Additionally, I also plan on adding a Raspberry Pi camera back to the machine, at which point I'd likely dedicate it for TSD 100%, so it could provide more accurate issue detection, while keeping the higher quality camera and stream setup separate, as it is right now. However, I of course couldn't do this without TSD supporting 64-bit first.

EDIT 2: In terms of TSD development and testing for arm64, please let us know if the community can be of any help, as I'm sure many of us would gladly contribute, myself included. :)

[kennethjiang]

@Dids Good point about Janus is still trying to launch when the premium streaming is disabled. I opened a separate ticket https://github.com/TheSpaghettiDetective/OctoPrint-TheSpaghettiDetective/issues/160 so that we can fix it sooner than supporting the 64-bit OS.

[leandrotoledo]

Following up on this for the support of Premium Streaming on aarch64 builds. Any updates? Thanks!

[wieluk]

Also getting the same error with moonraker-obico on a raspberry pi4 with raspbian os 64. Is there anything I can do? The stream is showing in Obico(selfhosted) but it just says:

Failure Detection is Off
Printer is not actively printing. Learn more.
Looking Good
Obico for OctoPrint is Offline. Why? 

[kchristensen]

Is there any chance this is going to get fixed?

[kennethjiang]

This is not fixed yet because of the slow adoption of the 64-bit bullseye for Raspberry Pi OS. 64-bit bullseye introduced a lot of compatibility issues but provided very little advantage on a weak CPU like Raspberry Pi's. With a very limited adoption, and huge amount of effort to figure out these compatibility issues, it just doesn't make sense for us to prioritize this issue very high. :(

[kennethjiang]

Of course, if any of you feel strongly about it, a PR will be very welcome. This is the beauty of open source: whoever is impacted can always take issues in own hands and fix it.