I noticed adding more vCPUs to a video receiving VM scales decently but after running the numbers (tallying up CPU usage differences between 2 and 4 vCPUs) I think there is indeed some amount of multithreading going on and suspect there's still room for improvement.
I found an interesting article that said it was able to accomplish 3.2x speed up to raw video conversion with a ParallelizedTaskRunner they patched into the videoconvert element of GStreamer. [1] We use videoconvert on the screen sharing video sender because the default X11 format (BGRx) is incompatible with anything the v4l2sink for v4l2loopack accepts. Also, to make sure there is only one homogeneous attack surface across both webcam streaming and screen sharing. So, first of all we could speed up that using this patch.
I wonder if we could implement the same work for the rawvideoparse GStreamer element on the side of the video receiving VM to achieve dramatic performance improvements.
I also found some GStreamer documentation for GstTask in general which maybe we could use to (possibly, I'm not sure) implement this even without patching GStreamer if we wanted to. [2]
Of course, to take advantage of any of this we would have to rewrite the GStreamer components in a real programming language such as Python or C (GStreamer has binding for both of these languages through GI/GObjects) as opposed to shell. Note that we wouldn't necessarily have to do away with all the shell but essentially we would at least have to replace any instance of gst-launch-1.0 with a Python/C program utilizing the GStreamer library.
I noticed adding more vCPUs to a video receiving VM scales decently but after running the numbers (tallying up CPU usage differences between 2 and 4 vCPUs) I think there is indeed some amount of multithreading going on and suspect there's still room for improvement.
I found an interesting article that said it was able to accomplish 3.2x speed up to raw video conversion with a
ParallelizedTaskRunnerthey patched into thevideoconvertelement of GStreamer. [1] We usevideoconverton the screen sharing video sender because the default X11 format (BGRx) is incompatible with anything thev4l2sinkfor v4l2loopack accepts. Also, to make sure there is only one homogeneous attack surface across both webcam streaming and screen sharing. So, first of all we could speed up that using this patch.I wonder if we could implement the same work for the
rawvideoparseGStreamer element on the side of the video receiving VM to achieve dramatic performance improvements.I also found some GStreamer documentation for
GstTaskin general which maybe we could use to (possibly, I'm not sure) implement this even without patching GStreamer if we wanted to. [2]Of course, to take advantage of any of this we would have to rewrite the GStreamer components in a real programming language such as Python or C (GStreamer has binding for both of these languages through GI/GObjects) as opposed to shell. Note that we wouldn't necessarily have to do away with all the shell but essentially we would at least have to replace any instance of
gst-launch-1.0with a Python/C program utilizing the GStreamer library.[1] https://coaxion.net/blog/2017/10/multi-threaded-raw-video-conversion-and-scaling-in-gstreamer/ [2] https://gstreamer.freedesktop.org/documentation/gstreamer/gsttask.html