search

pxi-gmbh / pipfy

a floating webcam image on your desktop
9 stars 1 forks source link

MASSIVE FEATURE: add minimalist WebRTC-functionality to invite/broadcast video+audio to different #4

Closed jochmann closed 3 years ago

jochmann commented 3 years ago

can we add a video/audio ingestion to the web-app and invite guests to put on our desktops, just like our own floating videos?

this would enable adding several video feeds to a desktop that can be recorded or shared (via Zoom, Teams, Jitsi etc) without the need for additional software (like OBS). many users at universities cannot install software on their work computers, so a completely browser based solution to interactive group discussions or interview while screen sharing would create immensely powerful workflows.

perhaps to add this feature utilizing an existing code base like https://github.com/peers/peerjs could speed up the process. or use their implementation at https://glitch.com/~peerjs-video as inspiration for minimalist functionality and create a handshake server and code base of our own, if time permits

gaenseklein commented 3 years ago

you mean more then one PiP? (one from local webcam, others from streams over webRTC)

jochmann commented 3 years ago

you can create more than one PiP by opening up several browser windows and have each trigger a PiP even currently.

So yes, in one window I would run hovercam to access my local webcam -> PiP(1). In another window I would run hovercam(WebRTCedition) to ingest a stream from another hovercam-session -> PiP(2)

gaenseklein commented 3 years ago

okay, the video should be possible, as you render it as PiP(2) to the shared screen. but how would you transfer the sound? as i understood person a talks over jitsi with other people, now person a shares the screen to all participants in the chat and can include him or herself and also one or more other persons video via PiP. but the sound is only trasmitted from the microphone of person a, not what his speaker would sound like. so the sound gained over webrtc from person b would have to be linked to person a's microphone. of course you could setup something like this with for example jack in linux, but this is far more advanced and complicated and happens outside the browser, therefore outside of our control.

jochmann commented 3 years ago

yes, we would need to add ingesting audio via webRTC as well.

the audio would play on the desktop - still useful for recording or streaming. one way only. it's the users job to capture or re-route desktop audio. for the back channel people could simply call each other on the phone. for more advanced controls people will have to turn to obs.ninja and work out how to connect virtual cables to avoid feedback loops.

I would like to keep the app super simplistic - and this feature borders on pushing the "single task application" idea too far. but I can see it working:

default of the app is "put your video on your screen"

advanced option could be "put remote video (and audio) on screen" with either "send your video" or "invite a friend" as the interaction. an auto-generated link enables handshake to send audio+video over webRTC. the video and audio are displayed at the "invite a friend" screen. video resolution and bitrate are fixed to optimize for packet loss. resolution at 720p (at most, perhaps even less).

jochmann commented 3 years ago

out of scope. other tools exist that serve the use case better.