Multi User Testing

[MR-Alex42]

While developing a multi user game for Neos the topic of multi user testing was becoming an issue. Currently you always need human testers. So for testing multi user experiences some kind of simulated users would be required.

[Frooxius]

Hmm this is something I considered at some point, but I'm still not 100 % sure about what exact form it would take. The easiest approach would be to let the host connect to their own experience as another user and pretend like they're both present at the same time (otherwise others would always be away).

The biggest difficulty will be just controlling the different avatars. If toggling between them as if toggling between worlds is sufficient it could work, although it would also have significant performance overhead of running the same world and all of its updates multiple times on a single computer, which could make this perform poorly, perhaps enough to be prohibitive.

However there are other things to consider, like simulating network latency as well, which could be perhaps added as extra module.

Another option if there's a computer available is to allow it to run in some sort of "puppet" mode, where you could invite it to your world and command it to do things in some way. Essentially sort of controllable Neos Bot. This could even be offered as a service in a future, where you could invite one into your world for testing. This would be significantly more work, but wouldn't have the performance overhead.

What kinds of features would be required for this to be useful?

[MR-Alex42]

I like the Neos Bot approach. Maybe it could be using a headless Neos server running in the could. Regarding avatar control I have the following idea.: You could record all the movements and actions (see undo/redo) of a user and later play those back as a different users in the same world would enable multi user testing.

[Frooxius]

Yeah the Neos bot might be most practical. Could run it on another computer as well (or even local as headless instance if it can take the performance hit) so would be most flexible in that regard with a single implementation.

I don't think recording the actions would work well though, as often the testing scenarios are more interactive in nature and can depend on a lot of divergent behavior (timing, latency, pseudo-randomness, differences in the human user behavior), I think having a tool to puppeteer them would work the best.

Recording some input playback on those could be done on top of this, as prerecorded puppeteer sequences.

[sirkitree]

I'd like to point out that a few other applications have this idea of 'recorded avatars' which tend to be either like a movie that plays out in the world, or is scripted. EngageVR has a way that you can record a session, which records exactly what the avatars are doing and their audio. This can then be played back.

JanusVR has a means of recording your movements as a 'ghost' and then those avatar position are then replayed by a user when they enter a world. In this manner you can have avatars populating a world, doing different things, and the space can seem populated, even though they're scripted.

Mindshow is entirely based on this, and is intended as a means of creating videos and avatar actors, recording and playing back. It's a pretty killer feature IMHO.

So to scope this a bit:

Have a way to record an avatar's actions in a world (starting with position and audio), save this as an asset, and play them back.

[Medravr]

I second sirkitree's request. This feature will be necessary for my future vr cinematic projects.

[Frooxius]

It's interesting approach, but like I mentioned in my previous post, I don't think recording the actions would work really well for Neos, as it would only cover a subset of testing scenarios.

Problem is that things in Neos are significantly more dynamic and varied in other applications, so pre-recorded animations could often drift or not be relevant to the testing scenario, so being able to dynamically adapt would be crucial.

[Anomalous]

It sounds like Sirkitree's and Medra's requests are more for a general use avatar animation system rather than anything to do with testing in specific. Maybe as part of the timeline animation system #48 ?

[sirkitree]

yeah, testing isn't really the scenario I have in mind, rather creating animated avatars which you could then trigger the recorded animation, creating NPCs and such, or just recording your interaction with someone in a scene.