Floating webcam view above avatars that follows each avatar

[Coda-Coda]

It would be helpful to have the option to have a floating video of each person's webcam above their head, so that you could look around the room and see everyone else's real faces.

Further Detail

One of the options for sharing your webcam should be to "pin the video above your avatar", and the video should follow you around. It would be also worth investigating whether this would work best implemented as videos currently are displayed, or if they should be visible to anyone regardless of what angle they are facing you at (e.g. if they are directly to your left they should still see the full video clearly).

Existing Alternatives

Existing alternatives are people placing their webcam videos within the 3d space, however this doesn't allow them to move with the person they belong to. You can also "inspect media" to lock other people's videos one at a time to be visible to you as you move around, but this is only one at a time and doesn't create the experience of looking around at each other.

Context

This would be useful for allowing people to see each other as they walk around and interact.

Thanks!

(By the way I'm so very impressed by hubs and think that it is an absolutely amazing project!! - thanks for all the work you all have put into it.)

[gfodor]

Thanks for this submission! This is a common request. One reason we've been hesitant to implement it is that it would radically change the bandwidth/CPU requirements of hubs if, for example, it was common for people to just use their webcam as their avatar. It's certainly worth exploring, but in general it may open up a whole new category of problems due to the huge increase in server and client resource utilization. If you had a room of 30 people all using their webcams, the requirements for hubs would rapidly approach that of a videoconferencing tool, and a lot would need to be done to start selectively sending video data to clients based upon proximity, etc, which currently would be a challenge to do.

[Coda-Coda]

Fair enough, perhaps this feature would be better suited for certain non-public Hubs Cloud instances where the issue of bandwidth/CPU requirements could be better managed than the public hubs.mozilla.com ? Even then though, I imagine there would be challenges to work through. But it would be really cool to use :)

[luser]

but in general it may open up a whole new category of problems due to the huge increase in server and client resource utilization

I'm curious about the server side—does hubs have its own media server or are you using something else? I can definitely understand "we don't want to deal with the overhead of hosting a videoconferencing server" as a rationale here! In that case it might be more worthwhile for someone to look into an optional integration with Jitsi Meet since Jitsi runs a free-to-use server and provides both high and low-level APIs for conferencing (also Jitsi is all open source).

I don't know that I buy the "client resource utilization" argument as big stumbling block though, since people are participating in multiparty Google Hangouts in browsers every day currently. That's what sparked my interest in this actually—I'd love to have some better options for large group meetings that are better than the Zoom-style Brady Bunch view with everyone talking over each other.

[Coda-Coda]

This was what sparked my interest in hubs too. A solution in hubs something like iSee would be great for this use case.

[gfodor]

Yup, so we do already run a centralized webrtc called Janus gateway that is similar to Jitsi. So we would be on the hook for the bandwidth.

However, the client load is a legit issue for two reasons:

• first, we have no way to limit which videos are streamed to whom. Video conferencing does not stream all video to all people, but down scales and does other tricks to minimize bandwidth and end user decoding costs. You don’t need to receive or decode video you won’t see (eg when the video is small, it should be at a smaller bitrate.) we would need to implement this stuff to some degree if video streams were usually enabled for avatars.

• we target mobile VR hardware which is already heavily resource constrained given we are also rendering a 3D room, avatars, etc in addition to videos. Typically on lower end devices you only want a single video playing since as things slow down you get frame loss which can cause discomfort. If you tried playing a video on each avatar, in most cases mobile VR devices and other low end hardware would fail to keep up.

On Fri, May 1, 2020 at 11:21 PM Daniel Britten notifications@github.com wrote:

This was also what sparked my interest in hubs too. A solution something like iSee https://www.iseevc.com.au/ would be great for this use case.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/mozilla/hubs/issues/2327#issuecomment-622708191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABVW5FYYJ77K7YJAWBALLLRPO3VLANCNFSM4L4R4XSQ .

[vOwl]

i agree with @gfodor 's concerns about activation videostream for all users as default. The current state as activating the video-stream on demand is fine, it's just having a "snap-point" at the avatars (head) for the camera picture (and following). That would improve the user-experience a lot in the same way, preconfigured Snap-Points for media/assets in the scene would help a lot. Everyone who prepared a room for presentation in the past knows what i'm talking about.

Let the scene+avatar creators define Snap-points (with dimension+rotation) for their scenes and avatars where to put/sap media to, because they know about that best. It's kind of the same as the Way-point system in Spoke. e.g. Scene has several presentation-displays -> scene creator defines those with Snap-Points. Users of the scene spawns media/assets, drags it to snap-point -> done. e.g. Avatar is a robot with screen on belly -> put snap-point there and let user attach webcam-stream (or static image) there.

I know last part about Snap-points doesn't solve the video-stream-server-client-load issues, but there is always a part in every app or game, you can't prevent that users use them in a way that's not fitting your way they should do.

For cost/performance optimization: what about an option to limit number of parallel video-/webcam-streams of a hub room. That option could be helpful for Mozilla Hubs and Hub Cloud installations.

[GrahamTheCoder]

This would be awesome for allowing a bigger group to naturally drift into smaller groups and have side conversations while hearing the main conversation quieter in the background and rejoining later.

Setting some limit on the free one sounds sensible (bitrate or number of streams)

[lue]

Just the ability to attach an image or a link (to LinkedIn for instance) would be great! It would match the functionality of Multiverse (app for Quest and Rift) https://youtu.be/ZZfTKF4PE6o?t=41

And here is the implementation by @utopiah for those who want to try it right now: https://twitter.com/utopiah/status/1257956376376197120

[emclaren]

Another request for this came in via youtube today.

[krobertisaksen]

I work at a university and we'd be very interested in something like this for our off-campus students (even setting aside the current situation). I understand it could require some overhaul to the architecture but making this change would be very valuable. The biggest problem for off-campus students isn't online learning as they and the teachers understand the pro's and con's of this type of teaching and learning, but the important social aspect is something we would very much like to improve upon, and something like this cross-platform looks very very promising to that end!

[arpu]

@krobertisaksen you can test https://vrinq.pw

[krobertisaksen]

@krobertisaksen you can test https://vrinq.pw

Thanks for the tip!

[CahootsMalone]

I agree that adding the option to parent (attach) a webcam object to one's avatar (in addition to, not as a replacement for, the current behaviour where webcam objects behave like other objects) would be of great value, particularly for people wishing to use Hubs for virtual improv performances or other purposes for which facial expressions are crucial.

My preference would be for the webcam quad to be placed at eye level (ideally as part of specific avatar models created with this purpose in mind), more closely mimicking in-person interaction. Hopefully this crude illustration can provide inspiration:

Regarding concerns that this would encourage webcam use and the associated bandwidth/processing requirements, perhaps giving the room administrator the option to disable (selectively or globally) webcam use by other participants would help?

[nickovaras]

Tech challenges aside, this is the killer feature that Hubs is missing and the one that would appeal to a much larger audience beyond us geeks. Hopefully, with some incredible ingenuity thrown in, it can be made to work.

[emclaren]

We were asked about this multiple times last week (including from an a11y community who wants to do live sign language in Hubs that can move around with an interpreter).

This also came in via twitter this week as well: https://twitter.com/tartanski/status/1324865503291174913

[emclaren]

Another request from Twitter https://twitter.com/raktorxr/status/1325530172255211520

[yajo]

I guess this could just be a common feature for any kind of objects.

I'd like to be able to attach a headset to my head, or a hat or a rabbit. Then, I'd be able to do the same with my webcam feed.

[djay]

given that eye contact would never work properly with a webcam stream I think the end results would be better doing facial recognition and eye tracking and syncing to the avatar similar to suggestions in https://github.com/mozilla/hubs/issues/3285. It would require much less bandwidth and could feel more natural. Eventually headsets would have eye and mouth tracking built in but for now a powerful browser in a laptop or phone app should be able to achieve this. Checkout the video here to see how natural it could be - https://blog.tobii.com/why-eye-tracking-will-be-standard-on-all-vr-headsets-f1b0e5326f3c. I don't think we need to wait for headset support for this to work now.

[ilmatematico]

I agree that adding the option to parent (attach) a webcam object to one's avatar (in addition to, not as a replacement for, the current behaviour where webcam objects behave like other objects) would be of great value, particularly for people wishing to use Hubs for virtual improv performances or other purposes for which facial expressions are crucial.

My preference would be for the webcam quad to be placed at eye level (ideally as part of specific avatar models created with this purpose in mind), more closely mimicking in-person interaction. Hopefully this crude illustration can provide inspiration:

Regarding concerns that this would encourage webcam use and the associated bandwidth/processing requirements, perhaps giving the room administrator the option to disable (selectively or globally) webcam use by other participants would help?

I totally agree with you. I would add an automatic crop to the face to fit the reserved space on the avatar head