SurfaceCast is a super-flexible streaming toolkit that lets you create shared interactive surfaces from a huge variety of devices, merging up to four separate locations into one shared mixed-reality space.
Some examples for the usage scenarios that you can build with SurfaceCast include a hybrid physical-virtual whiteboard (a), remote piano teaching (b), "dining at a distance" (c), or distributed physical board games and puzzles (d).
You can use SurfaceCast with just about anything: a tablet, a plain old PC/laptop, a VR or AR headset, a projector-camera setup, or a dedicated tabletop like the SUR40.
SurfaceCast consists of a mixing server, and one or more clients. Each clients sends one audiostream and two video streams: a plain old webcam feed of the user called the front stream, and a second feed of a rectified interactive surface called the surface stream. The surface stream is expected to have any background removed and chroma-keyed with 100% bright green.
The mixing server then composes a new surface stream for each client, consisting of the layered surface streams of the other clients, and streams that back to each client (along with a single combined front stream of all individual front streams arranged side-by-side).
Here's an example walkthrough of how to connect an interactive surface with a browser client:
./webrtc_server.py
https://${SERVER_HOST}:8080/stream.html
/dev/video20
(see Usage - Example 2)./webrtc_client.py -t ${SERVER_HOST} -s /dev/video20 -f /dev/video0
(or whatever device your plain webcam is)surface
window as fullscreen on the surface display, and the front
window on the front display --fake use fake sources (desc. from -f/-s)
Mostly useful for testing, will create default outgoing streams with fake test data (TV test image, moving ball, tick sounds).
If any of -f/-s/-a
are also given, the string will be interpreted as a GStreamer bin description.
-m, --main flag this client as main (lowest z)
If a client does not have background filtering (i.e. a plain webcam), then you can use this flag to make sure that the surface stream from this client is always placed below all others. Note you can only have one "main" client per session, otherwise the surface mixing will get messed up.
-a AUDIO, --audio AUDIO
-f FRONT, --front FRONT
-s SURFACE, --surface SURFACE
audio/front/surface source (device name or pipeline)
If any of these are given without --fake
, they will be interpreted as a device name (e.g. /dev/video0
). Otherwise, they will be interpreted as a GStreamer bin description (e.g. "videotestsrc ! timeoverlay"
). Note that in the second case the whole string needs to be quoted.
-p PORT, --port PORT server HTTPS listening port (8080)
-t TARGET, --target TARGET
server to connect to (127.0.0.1)
Used to give the hostname or IP address of the server, and optionally a non-default port to connect to.
-u STUN, --stun STUN STUN server
If you want to use a different STUN server than the default (stun://stun.l.google.com:19302), specify here.
-n NICK, --nick NICK client nickname
Can be used to give a label (e.g. "Alice" or "Bob") to the frontstream.
--persp PERSPECTIVE perspective transformation
Can be used to "outsource" the perspective transformation of the surface feed to the server. PERSPECTIVE needs to be a transformation matrix as nine comma-separated float values in row-major order. Note: when your perspective transform happens to start with a minus, the argument parser gets confused. Use it with an explicit equals sign as in --persp="-1,0,0,1,0,0,0,0,1"
.
-s, --sink save all streams to MP4 file (default: False)
-o OUT, --out OUT MP4 output filename (default: surfacecast-20220327-125732.mp4)
If -s/--sink
is given, write the combined front, surface, and audio streams to a MP4 file. Optional target filename can be set via -o/--out
. Note that the file contains OPUS audio inside an MP4 container, which is not supported by all players. If necessary, use scripts/playback.sh
to recode to MP3 and play all streams simultaneously in VLC.
-p PORT, --port PORT server HTTPS listening port (default: 8080)
-u STUN, --stun STUN STUN server (default: stun://stun.l.google.com:19302)
If you want to use a different STUN server than the default (stun://stun.l.google.com:19302), or a different listening port, specify here.
Mixing server & standalone client
sudo apt install gstreamer1.0-libav gir1.2-soup-2.4 gir1.2-gstreamer-1.0 gir1.2-gst-plugins-bad-1.0 gir1.2-gst-plugins-base-1.0 gir1.2-nice-0.1 libnice10 gstreamer1.0-nice gstreamer1.0-plugins-bad gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly
HTML5 client
about:plugins
)chromium-codecs-ffmpeg-extra
, see issue #8)If you use SurfaceCast in an academic context, please cite our main publication:
F. Echtler, V. Maierhöfer, N. B. Hansen, and R. Wimmer. SurfaceCast: Cross-Device Ubiquitous Surface Sharing. In Proceedings of the ACM on Human-Computer Interaction, ISS '23 (to appear).
[...]: loop detected in the graph of bin 'pipeline0'!!
, which can be safely ignored.GST_V4L2_USE_LIBV4L2=1
can sometimes be used to fix format mismatch issues.--persp
parameter.