Closed zimmertr closed 5 days ago
That's some interesting bug. But as you already noticed, Crowsnest is basically doing nothing, that could create this problem. I recommend to get Nginx, Mainsail and Crowsnest out of the equation and test it only with ustreamer. So run the following command:
/home/tj/crowsnest/bin/ustreamer/ustreamer --host 0.0.0.0 -p 8080 -d /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0 --device-timeout=2 --encoder=HW -r 1280x720 -f 10 --allow-origin=* --format=yuyv
Then connect to http://<ip.of.your.pi>:8080/?action=stream
and see if you can reproduce the issue like that.
Hm, yeah. You're totally right. It does seem like an upstream issue with ustreamer
. I can replicate the bug by launching it directly instead of through Crownest.
Here's another screen recording: https://www.youtube.com/watch?v=3VNb6MwUPKQ
Can you try iperf
? Just to see, if it's really ustreamer or just some problem at higher bandwidth usage
I cannot bring the network interface down withiperf
, so it does seem unique to ustreamer
. Here's another screen recording: https://www.youtube.com/watch?v=lghwhJxn3MU
I'll try and test against the newest version as you suggested on Discord and report back after lunch.
What's also interesting, is that I am not observing the same behavior when I run curl
locally against the endpoint. Instead of using my browser. Nothing breaks. However, if I run curl
against the endpoint from outside the Pi, for example from my Macbook, it breaks networking. So this bug only occurs when the traffic originates outside of localhost?
curl hektor.sol.milkyway:8080/stream --output /dev/null
Unfortunately it occurs with version 6.16 as well. Here is another screen recording showing that as well as the note I made above about it only failing when the traffic originates from outside of the device itself. https://www.youtube.com/watch?v=QSUJxR5HU0g
I'll go ahead and file a different issue with ustreamer
. You can close this one if you want. Or leave it open to track the status of the other. In either case, I'll comment a link to it here once I've opened it.
Issue posted: https://github.com/pikvm/ustreamer/issues/293
What's also interesting, is that I am not observing the same behavior when I run
curl
locally against the endpoint.
Do you use localhost or 127.0.0.1 for that? These addresses point to loopback apdapter which is a virtual interface. So you don't have the same way the packages go as through your external wifi card.
I used the FQDN which resolves to the 192.168.30.170 IP Address. I just repeated the experiment and I can see that ustreamer
logs the connection attempt as originating from that IP too. But I wouldn't be surprised if Linux is somehow redirecting it to go through the loopback interface.
Thanks for your information. Well I wouldn't surprised either if Linux does something like that.
Anyways did you check the syslog for entries of the usb device disconnecting or something related?
No, but I did notice that dmesg
was continuously dumping the following error message:
brcmfmac: brcmf_set_channel: set chanspec fail, reason -52
This is interesting, because brcmfmac
is the driver used by the Raspberry Pi's internal wlan0
NIC
I set up a second NetworkManager connection profile for wlan0
and enabled it and rebooted. Then those errors stopped getting logged.
Afterwards, I went ahead and started up a ustreamer server again and connected to it over 192.168.30.170 via wlan1
as usual and it did not crash!
However, using bmon
, I can see that wlan0
is the one transferring data continuously, NOT wlan1
. Despite the fact that I am hitting the ustreamer
server over wlan1
's IP Address.
Like, I'm happy things are working right now. But something is still wrong. ustreamer
/crowsnest
is only seemingly functional when it can use wlan0
. And even when I connect to the Raspberry Pi over wlan1
, I still see that wlan0
is being used to transfer data. This is problematic because the wireless reception of that NIC is poor, and I can only get 7-11FPS over the 720p stream. Which is what originally led to me purchasing the USB wireless adapter to begin with.
Right above the brcmfmac spam you can see your wifi stick disconnecting and reconnecting to the usb bus. Also you can see a tx timeout. When did you check dmesg? After a crash?
I think that dmesg
output was from immediately after startup. I don't see the log indicating a disconnection nor a tx
timeout. I do see it mention limiting tx
power. I presumed that had to do with my Ubiquiti AP's Power settings but I could be wrong.
It was previously difficult to get crash information. The crash only occurs if wlan0
is down. In these cases, once wlan1
also goes down the device is no longer accessible on the network. Additionally, the Raspberry Pi 5 does not have an HDMI output, it just has a DSI header.
However, I've just attached a USB keyboard to the Pi, set a password for my user, turned off the KlipperScreen
SystemD unit, and now I can access a TTY and log in without networking.
Give me a while and I will trigger a crash and record of a video of the screen to gather more data.
Okay, I've gotten to the bottom of this. Mostly.
I recorded a long video of the DSI screen and then when I started to watch it over I realized none of this was necessary at all I could simply use journalctl --dmesg -b -1
to get the last boot's log message. Then I remembered that not even that was necessary, because the networking starts to work again after ~20 minutes. So, with wlan0
down, I SSHd into the device, started a ustreamer
log, hit it with my browser and caused a crash. Then I waited until it started to respond to pings again and pulled a log from the device.
In this, I saw various messages repeating like the following examples:
[ 129.573465] usb usb1-port1: over-current change #1 [ 129.708999] usb 1-1: USB disconnect, device number 2 [ 131.309504] usb 4-1: reset SuperSpeed USB device number 10 using xhci-hcd
This led to me trying a few things.
I removed the Wireless adapter from the extension cable it was using in case the cable was causing a power/bandwidth problem. That didn't fix anything. Then I removed one unnecessary USB device from my Pi (A Noctua fan) to see if that might help with reducing power consumption. That's didn't fix anything either.
It was at that point I was starting to worry that I messed something up between when I wired my printer's power supply to a buck converter and then eventually the Raspberry Pi. And I was over/under volting the device. I'm a software engineer, but do not have any real experience with electrical engineering.
On a whim, I decided to change to a new USB Wireless Adapter that just came in the mail. And the problem went away entirely. To rule out any power issues, I ran stress --cpu 4
and maxxed out all four cores on the device for a minute or two until lm-sensors
started showing the temps creeping towards 80°C when I backed it off. None of the USB devices or anything ever disconnected, and the Pi never turned off.
So I'm inclined to believe it's an issue with the mt7921u
driver. So, anyone landing here from Google in the future. Don't use anything with that driver. Including, in this case, a NETGEAR Nighthawk (A8000) - AXE3000. The random, and significantly cheaper, Nineplus adapter works without issue. And is actually performing BETTER than the NETGEAR one was.
A huge thank you @mryel00 and @rackrick for patiently trying to help me troubleshoot this over the past day. Now I can finally finish up this guide and put the printer back together :)
Still only getting like 7-11fps at 720p though. So I still need to figure that out. Not sure why it's been capped at 10 by ustreamer
.
... [10/25/24 17:37:17] crowsnest: [cam 1] [10/25/24 17:37:17] crowsnest: mode: mjpg [10/25/24 17:37:17] crowsnest: port: 8080 [10/25/24 17:37:17] crowsnest: device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0 [10/25/24 17:37:17] crowsnest: resolution: 1280x720 [10/25/24 17:37:17] crowsnest: max_fps: 30 ... [10/25/24 17:37:19] crowsnest: Size: Discrete 1280x720 [10/25/24 17:37:19] crowsnest: Interval: Discrete 0.033s (30.000 fps) [10/25/24 17:37:19] crowsnest: Interval: Discrete 0.042s (24.000 fps) [10/25/24 17:37:19] crowsnest: Interval: Discrete 0.050s (20.000 fps) [10/25/24 17:37:19] crowsnest: Interval: Discrete 0.067s (15.000 fps) [10/25/24 17:37:19] crowsnest: Interval: Discrete 0.100s (10.000 fps) [10/25/24 17:37:19] crowsnest: Interval: Discrete 0.133s (7.500 fps) [10/25/24 17:37:19] crowsnest: Interval: Discrete 0.200s (5.000 fps) .... [10/25/24 17:37:20] crowsnest: DEBUG: ustreamer [cam 1]: -- INFO [55.348 ] -- CAP: Using HW FPS: 30 -> 10 (coerced)
Okay, managed to get up to about 15fps for my Logitech C920 with this configuration:
[crowsnest]
log_path: ~/klipper_logs/crowsnest.log
log_level: verbose
delete_log: false
no_proxy: false
[cam 1]
mode: ustreamer
port: 8080
device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920_809E7D1F-video-index0
resolution: 800x600
max_fps: 30
v4l2ctl: focus_automatic_continuous=0,focus_absolute=70,brightness=100,contrast=100,saturation=100,sharpness=200,backlight_compensation=1
custom_flags: --format=yuyv
I gather from the Discord forums that the recommended solution to further improve FPS would be to use WebRTC which requires camera-streamer
. However, these issues indicate to me that's not supported on the Raspberry Pi 5 do to a lack of hardware encoders.
So I suspect this is the best possible outcome at this time. I don't exactly understand why I cannot reach higher resolutions/fps though as this does not appear to be a bandwidth constraint. bmon
currently shows a TX
of about 360kb/s while streaming to Mainsail at 800x600@15fps. But iperf
shows my Macbook Pro can talk to the Pi at over 5MiB/s.
Higher resolutions and MJPEG (normal ustreamer format) needs a lot more bandwidth to transfer the images (its just multiple jpeg per second. nothing advanced). WebRTC is much more bandwidth optimiced then MJPEG.
About your fps. You should not use --format=YUYV
unless you really need it to run the cam. It's possible that your cam itself doesn't produce more pictures with YUYV. YUYV is not compressed therefore it uses more USB bandwidth. With 800x600
you only get a maximum of 24 fps for your cam. At 1280x720
you only get up to 10 fps with YUYV. Those informations can also be found in the log.
Furthermore you should use the MJPG service in Mainsail, not the Adaptive MJPG.
If I remove the --format=yuyv
flag, FPS drops to 1-3 at both 800x600
and 1280x720
. So I'm afraid this is the best I can do with a Raspberry Pi 5 and a Logitech C920.
Also, I'm starting to think my original issue was not necessarily driver related so much as power related.
Most Raspberry Pis provide enough current to USB peripherals to power most USB devices, including keyboards, mice, and adapters. However, some devices require additional current, including modems, external disks, and high-powered antenna. To connect a USB device with power requirements that exceed the values specified in the table above, connect it using an externally-powered USB hub. The power requirements of the Raspberry Pi increase as you make use of the various interfaces on the Raspberry Pi. Combined, the GPIO pins can draw 50mA safely; each pin can individually draw up to 16mA. The HDMI port uses 50mA. The Camera Module requires 250mA. USB keyboards and mice can take as little as 100mA or as much as 1000mA. Check the power rating of the devices you plan to connect to the Raspberry Pi and purchase a power supply accordingly. If you’re not sure, use an externally-powered USB hub.
https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#typical-power-requirements
Okay, so I've come to learn that the Raspberry Pi 5 requests the input current from the USB-C port when it boots. If no response is received, or it is less than 3A, the USB-C devices are limited to consuming 600mA. However, if it's greater than 3A, they can use up to 1.6A. Since my device is powered by the GPIO pins I was being limited.
This can be fixed by turning this feature off and manually defining the input amperage.
sudo raspi-config
# 4 Performance Options
# P4 USB Current
# Would you like the USB current limit to be disabled? -> Yes
sudo rpi-eeprom-config --edit
# Add PSU_MAX_CURRENT=5000
# Save and exit the file: Ctrl+X -> Return
sudo reboot
After this, the NETGEAR wireless USB adapter works just fine.
What happened
I recently flashed my printer to run Klipper which is driven by a Raspberry Pi 5. The Pi is installed beneath the printer in it's electronics housing which results in poor wireless reception. So I purchased a USB wireless adapter with an antenna that sits outside of the printer. This works great, however, I've been chasing down a bug where the Pi would randomly disconnect from my network when I would interact with Mainsail, Fluidd, or KlipperScreen. I believe the ultimate catalyst if the bug in all of these cases is Crowsnest.
The Internal wireless chip is
wlan0
and the USB wireless adapter iswlan1
. I run Raspberry Pi OS Lite which uses NetworkManager. I've defined a connection profile forwlan1
but notwlan0
. So, on boot,wlan1
comes up and gets an IP from my router through a static ip reservation.This all works great, and I can use Mainsail, KlipperScreen, etc just fine. But as soon as I try and load any UI element that loads the webcam, the
wlan1
interface gets setDOWN
and networking breaks. Locally on KlipperScreen, Klipper continues to function just fine. So it's not crashing or anything. The networking just breaks.I poked through the code and I'm not sure where this could be happening, but I suspect it has to do with how ustreamer binds to a NIC. I don't see any configuration options documented that allow this to be configurable.
It does eventually recover and wlan0 comes back up. I'll share some STDOUT from ping showing this in the Additional information section below. But it takes minutes.
Here is a screen recording demos the bug occurring around 1:40: https://www.youtube.com/watch?v=WBQlUoP5Dfc
What did you expect to happen
Crowsnest to support multiple network interfaces, not default to the first one, and to work with any of them
How to reproduce
Run Klipper, Mainsail, Crowsnest on a device with multiple wireless NICs and a USB Webcam. Have the first NIC
DOWN
and the second NICUP
. Load the webcam and observe the second NIC be set toDOWN
.Additional information
Crownest Config
Network Manager Connection Config
ip addr
Crowsnest Debug Log
STDOUT from Ping During Failure and Eventual Recovery