search

ethand91 / mediasoup3-record-demo

Simple Record Demo using Mediasoup 3 and GStreamer
198 stars 93 forks source link

Tiling several feeds? #40

Closed ldenoue closed 3 years ago

ldenoue commented 3 years ago

Not a bug but a question: would it be feasible to send ffmpeg or gstreamer not one but several incoming feeds and let them tile the result (mixing the audio, and tiling visually the video tracks)? This way we could produce recordings to file that contain video of several browser webcams!

useful for educational purposes.

anteeek commented 3 years ago

You can use node-canvas for that. Maybe combined with https://github.com/muaz-khan/MultiStreamsMixer

ethand91 commented 3 years ago

@anteeek Thanks!

ldenoue commented 3 years ago

@anteeek and @ethand91 Muaz's solution is client-side; with node-canvas, how would we decode the frames of the individual streams? Does mediasoup offer such a thing? node-canvas would be simpler for me (JS programmer), but it feels that the better route would be to build on @ethand91 work and extend the gstreamer pipeline to dynamically add new elements (new WebRTC producers coming into mediasoup).

In Pion/Ion (a golang SFU), someone wrote an example: https://github.com/pion/ion-sdk-go/pull/31/files

Notice the file compositor.go that calls the C function gstreamer_compositor_add_input_track which itself calls internally gstreamer_compositor_relayout_videos to actually layout each incoming track.

I suspect something similar is possible. Perhaps using https://github.com/dturing/node-gstreamer-superficial or the python bindings of gstreamer, similar to mediasoupbin https://github.com/vpalmisano/mediasoupbin