toomim
commented
3 years ago Hey Ben! First, the idea of a scalable video wall is also what inspired us when building https://tawk.space (https://github.com/invisible-college/tawk.space). We wanted to create a user experience where you could fluidly join and leave groups of people once the wall of humans got huge. However, this was a university research project, and we never implemented the back-end aggregation (what you're referring to as "sprite" state) that would allow videochat to scale to 100s of people.
Second, I want to make clear that Statebus isn't actually handling the network or encode/decoding of video in the demo you saw. Statebus is only drawing the UI, and putting WebRTC <video> elements into divs. It would be awesome if Statebus could transmit video streams, but it's only been optimized for JSON.
Ok, all that said, let's talk infinite scrolling! Indeed, I've thought about this problem before, too. One would like to disable the animation, updating, and even network traffic for UI elements that are offscreen. We haven't implemented such an optimization yet, but you could totally do it. You'd add some scroll event handlers to the browser, and whenever an element goes offline, you'd have it forget() all the state it's subscribing to. This forget() would chain down from the UI to the network, and cease sending updates for that state over the network, and then go to the server, and the server could relinquish holds on the database objects, etc.
You also seem to be asking if we could handle the coalescing of multiple video feeds with Statebus itself. I could envision this in the programming model... let's imagine we wanted to break up 100,000 videos into 50-video chunks. Then as you scroll the page, it would load only some chunks at a time. So perhaps at the top of the page, the client would issue:
fetch("/video-block/0")
fetch("/video-block/1")
fetch("/video-block/2")
fetch("/video-block/3")
...etc. Then as you scroll down, you'd start seeing:
forget("/video-block/0")
fetch("/video-block/4")
...as it stops subscribing to the top-most chunk, and starts subscribing to a chunk down below.
The server that does transcoding would then start transcoding chunk 4, and stop transcoding chunk 0. Unless, of course, there are other users that are still looking at chunk 0 -- in which case it would continue transcoding chunk 0 until it's not needed anymore.
But again, the Statebus library currently only transmits JSON. So if you wanted to put a video stream in there, you'd need to base64-encode it into a string, and that wouldn't be very fun or performant.
In 2013 I set out to build a project called Interconnect however the tech wasn't available for an indie developer to do it at the scale necessary at the time: notably, a live face wall of thousands-to-millions of live webcam feeds of users. Statebus seems possible to make that happen (akin to that webcam demo video on the homepage, but x1000).
I am left wondering though, if can it scale from say one webcam feed to a thousand. Typically with that many elements, a page would need to do infinite scrolling as well as purging of elements that are off screen while keeping a semblance of scroll position, as otherwise the browser gets overwhelmed as the machine runs out of compute and memory resources.
I imagine that statebus allows us to connect a specific DOM tree, or say a variable to a remote state to be synchronised, however the entire DOM tree, nor the entire variable state needs to be synced. One would just use whatever typical infinite-scroll + purge library they are use to using, and then just remote sync each webcam feed, a collection of coalesced feeds.
Performance could further be enhanced by coalescing webcam feeds into say a "sprite" state, which with statebus's magic, wouldn't just be a typical image sprite, but also a DOM sprite. So say a hundred live feeds and their DOM can be coalesced into shared sync state, assembled from dozens of server-to-server shared sync states of individual feeds. So say 100,000 video feeds, would reduce into say 1,000 feed segments (and how it is reduced can be up to say a search filter and sort order, of which popular ones are done).
Is this a somewhat accurate understanding of how one would implement statebus?