Dashboard video streaming - erratic behavior

[jsimpso81]

My architecture:

• Raspberry PI 3b - PhotonVision image 1.4
• Latested 2022 updated FRC game tools
• 1 USB camera
• 3 pipelines (almost identical - orange ball, red ball, blue ball detect) Robot can switch between pipelines.

Driver mode is off (For out purposes, I think driver mode would always be off.) I want to stream the "*-input" stream, but I think this happens with the "output" steam as well.

Often after rebooting the robot, restarting the dashboard, and cycling power on the raspberry PI, the "-input" and "-output" streams are reported to network tables, but the streams can be be used. Opening the web interface and selecting the stream for display there also causes the dashboard stream to start working.

Often when closing the web interface (Firefox or Edge) the dashboard stream also stops.

Sometimes when programmatically switching between streams, the dashboard stream stops.

Sometimes cycling power on the Raspberry PI changes the behavior.

This can always be fixed by starting the web interface and showing or toggling the stream on the web interface. However since this doubles the wifi bandwidth, this isn't an acceptable workaround.

I didn't look at the code, but my assumption (and preference) is that the web interface buttons are "show the stream in the web browser" and not "set the existence of the stream on the server". Behavior seems to suggest this assumption isn't correct.

To me it seems that the ability to connect to the "input" and/or "output" stream should always be available and that this should be a "camera" level thing, not a pipeline level thing (unless it is or will be possible to run parallel pipelines on the same camera). If for some reason the input stream isn't available, a static picture a changing time could be sent to the dashboard stream.

If for some reason, the server can't serve up a stream any longer, such as a camera is deleted or renamed, I'd suggest deleting the network table variable (if that works...).

I tried this with two types FRC dashboards to make sure it wasn't a dashboard issue.

[jsimpso81]

I haven't looked at the code, but one potential fix would be to have the server perform the same action as server performs on behalf of the web interface when the button is pressed to start the stream when the server detects that a new stream socket connection request arrives and while there are more than zero active socket connections. When the number of stream socket connections for a particular port goes to zero, the stream could be disabled, similar to the user unselecting the stream on the web interface.

[jsimpso81]

I looked at the code and did some testing. It seems like it would be easy to add a check to VisionModule.java to check if there are active dashboard connections in the fpsLimitedResultConsumers.add section of code. This would require a new method in MJPGFrameConsumer.java to return this indication. However I did not find the ability to retrieve the number of connections in the cscore MjpegServer. It appear that this value may exist internally but isn't exposed. My suggestions would be: 1) Ask WPILIB to add a method to MjpegServer to return the number of connected clients. Then add this as described above to determine what to do in fpsLimitedResultsConsumers.

2) Add an option web interface to "Enable Dashboard" for the input and output streams. This option could then be used to determine what to do in fpsimitedResultsConsumers.

I have source code that has the PhotonVision portion of 1) coded. Right now it just defaults to indicating that there are clients. This does keep the dashboard alive and avoid the "disabledTick" screen. If anyone wants this, let me know. I could post a fork on my github.

[mcm001]

Adding that to wpilib sounds clean to me. Unsure how big of an internal change that is. I just want to make sure that for the picam, people don't enable it and inadvertently hurt performance. (We can document that as much as possible but we still see people who miss that part of the docs). Or adding a network tables "keep alive" entry might be the fastest fix? Imo, totally open to a PR for either of those. Or we could limit disabling behavior to picams?

[jsimpso81]

It turns out that there is already a function that indicates if there are connected MjpegServer clients. I implemented this and it works okay. If you like, I can implement these changes to the current development release and do a pull request.

Also there is a bug in the latest dev source. The call to disabledTick uses dashboardInputStreamer when it should use dashboardOutputStreamer.

[mcm001]

Yeah a pull request would be lovely. We can bump the version once this is merged in too.