MoffKalast
commented
1 year ago I've been thinking about this problem yeah, and it's an annoying one for sure because it's filled with tradeoffs. The standard way images are sent through rosbridge is by encoding them as base64, making them about 30% larger which completely clogs web tcp sockets at any reasonable rate. Could work with heavy throttling I suppose, but even transferring OccupancyGrid type images freezes other topics for a split second.
Second option is the web_video_server, with mjpeg streaming which is workable over wifi at low resolutions and low frame rates and goes through http requests so it doesn't directly impact socket transfers (probably the most sane option). webrtc_ros used to be a better option, but building webrtc is such an undertaking that it hasn't been kept up to date at all. I think another option would be using aiortc through python, since they handle that part but one would need to build a web_video_server type node around it.
As for displaying the images client side, it's also not ideal since there's very little screen real estate on mobile. Could work on desktop with a movable scalable pop-out window, but if a camera position is defined in the TF tree then maybe something like a floor reprojection that assumes the floor is at Z=0 (scroll all the way down) would be more practical, though if there's no localization it would only be fixed in the camera frame and if the camera isn't seeing any of the floor then it's not useful either.
So no silver bullets unfortunately, but I'd like to hear what everyone thinks.
Please enhance the code to display the camera images on web view