amalnanavati
commented
8 months ago After a deep dive on this yesterday, here are some options:
Option A
The way HCRLab has done it is by using a "standard" WebRTC approach, e.g., both the peers are pages in a web browser. For example, in stretch_teleop_interface, they have a signalling server, a browser page to run on the robot's computer, and a browser page to run on the client computer. As part of their startup script, in addition to starting the React web server and the signalling server, they also launch Chromium in headless mode and open the robot page on it.
Overall, their pipeline is ROS -> rosbridge -> browser running on the robot's computer -> WebRTC -> browser running on the client's computer.
The amaln/webrtc branch has the start of code for this. As opposed to using a fully P2P signalling server as they did in Stretch teleop, I implemented an SFU server (a broadcast server) following this tutorial. Other relevant tutorials include this and this.
Option B
Although the above setup works, it seems unnecessary to have a headless browser be the client. In theory, the server itself should be able to subscribe to the ROS topic and broadcast it out to clients over WebRTC. In other words, one end of the peer connection would not be in the browser, but would be in C++/Python code.
That is what webrtc_ros provides. It is written in C++, and runs a signaling server. When the server gets a connection (which includes the ROS topic to subscribe to in the URI), it launches a client that subscribes to that topic (via image transport) and sends the stream to the WebRTC server.
In theory, this should work. However, I tried it for a few hours and found: (a) a frustrating lack of documentation. The best I could find was this and this. (b) Challenges building on by Ubuntu 22.04 aarch64 Ubuntu VM due to dependency issues, coupled with the fact that no binaries exist for ROS2, gave me the impression that it's not well-maintained.
It is still worth looking into webrtc_ros more to see if there is a way we can get it to work. An alternative approach is to use aiortc in Python, which has an example client and an example server, and is well-maintained.
As documented in #106 , #111 , and this online article, WebRTC is the correct way to implement video streaming from the robot to a web app. What we currently have is okay but far from optimal in terms of data efficiency, lag, video clarity, etc.
This issue is to implement WebRTC streaming on the robot and web app.