Setting camera position (multi-display, multi-client)

[yuvadm]

I'm working on a multiple-display-multiple-client setup where I have the same scene running on several different machines (Google's Liquid Galaxy).

In this setup I have one machine which is master, and broadcasts it's data to all other slave machines (WebSockets running on an external synchronization server, essentially).

The scene has one PerspectiveCamera attached to FlyControls. The basic behavior I need is to broadcast the right data from the master and set it properly on all slaves. I have something working but the behavior seems wrong.

Master - I set up a hook in the FlyControls keydown() method which sends out a broadcast of the current camera.position and camera.quaternion.

Slave - on each broadcast message received I simply set camera.position (x,y and z) and camera.quaternion (w,x,y and z) according to the data received. So right now I have all displays showing the exact same thing.

How should I go about syncing the displays with the proper deltas?

Can I use a better hook to send the broadcast, other than using keydown()? I can send out a broadcast from the render() method, but I prefer to send it only when the camera actually moves, not all the time.

Is there a better way to do what I'm doing?

I've seen the code at http://mrdoob.github.com/three.js/examples/webgl_multiple_canvases_circle.html but I'm kind of at loss as to which events I should propagate to all the clients, and how exactly the deltas between the different displays should be defined.

Any advice, much appreciated :) Thanks!

[mrdoob]

Yeah, I guess FlyControls (and all the other controls) would benefit of a update event like TrackballControl has. I think the best option right now is using the keydown() for broadcasting the positions as you're already doing.

[yuvadm]

OK. What about the camera deltas?

I've tried adding a camera with camera.setY(someDelta) and then adding that camera to a virtualCamera, but not quite sure to which camera I should reference any other code.

Should the controls be attached to the camera or the virtualCamera? What about scene rendering?

Alternatively, I prefer a method that I can set a permanent shift (on the Y axis) on a camera, without having to use a second virtualCamera.

[mrdoob]

What's the benefit of broadcasting deltas rather than absolute positions?

[yuvadm]

I'm broadcasting absolute positions :)

I just don't know how to apply the camera shift delta on each slave.

[mrdoob]

As far as I undertand, you just need a FlyControls in the master. Then for every change, broadcast [position.x, position.y, position.z, rotation.x, rotation.y, rotation.z]. And then for every slave camera.position.x = data[0], ... is this what you're asking?

[yuvadm]

OK I figured out where my problem is.

The thing is that my PerspectiveCamera combined with FlyControls do not allow me to set the camera.rotation vector. It's always (0,0,0) with no way of changing that. camera.lookAt() also has no affect. I assume this is because FlyControls sets camera.useQuaternion = true.

What would be the best way to set the camera rotation in this case?

[mrdoob]

I don't understand. Why you you need to modify the camera that the FlyControls is modifying?

[yuvadm]

Like you mentioned correctly, I only have one set of FlyControls - on the master machine.

The naive approach of broadcasting the master's camera.position and camera.rotation to all the slaves doesn't work due to camera.useQuaternion = true. Therefore I need a different method that allows me to broadcast the master camera data, and ensure all the slaves are synchronized to that data (in addition to applying the proper rotation on the Y axis for each different slave).

[yuvadm]

OK, I think I got the basic setup working. I'll leave the gist of it here for future reference for anyone doing multi-display and/or multi-client setups.

Master - has a world with a PerspectiveCamera with FlyControls attached. (We also have a 3Dconexxion Space Navigator that maps 3D controls to keyboard presses that affect the controls.) The FlyControls sets camera.useQuaternion = true which means that the camera uses position and quaternion (and not rotation). The camera data is broadcast on each render() (for stability and consistency). Data is broadcast over WebSockets to a synchronization server - a LAN is probably required for reasonable latency.

Slaves - each slave has an broadcast message handler that receives the camera data. The position is set directly, no conversion is needed. The rotation requires setting a proper delta. We use several screens in a horizontal setup, so we need to apply a rotation delta upon the Y-axis. We use the code as proposed in #187:

// in this example we have a 30 degree offset screen, to the right of the master
// 0.45 is a damping factor needed for some reason
var ydelta = (Math.PI / 6) * 0.45 * -1;
var deltaQuaternion = new THREE.Quaternion();
deltaQuaternion.set(0,ydelta,0,1).normalize();
broadcastQuaternion.normalize().multiplySelf(deltaQuaternion);
// now you can set the camera.quaternion properly

This is the gist of it, and it works pretty good. Thanks everyone for all the assistance (@mrdoob in particular!). Anyone doing this setup is more than welcome to contact me for advice :)

[mrdoob]

What if all the slaves had this setup:

var dummy = new THREE.Object3D();
dummy.useQuaternions = true;
scene.add( dummy );

var camera = new THREE.PerspectiveCamera( 50, width / height, 1, 1000 );
camera.rotation.y = deltaY;
dummy.add( camera );

Then, you would be broadcasting position and quaternion data from master and applying to dummy in all the slaves.

[yuvadm]

Interesting idea. In this case, would the camera be able to adjust the rotation according to the dummy quaternion?

[mrdoob]

Yep, dummy matrix would multiply its children's matrices.

[WestLangley]

@yuvadm

// 0.45 is a damping factor needed for some reason

You are not setting your quarternion correctly. Do this:

var deltaQuaternion.setFromAxisAngle( new THREE.Vector3( 0, 1, 0 ), -Math.PI / 6 );

This should work for any angle -- without a "factor".

@mrdoob's idea also sound's good, but in that case, you also want to omit the "factor".

[yuvadm]

@WestLangley - thanks for the bug fix.

We might want to fix the example at http://mrdoob.github.com/three.js/examples/webgl_multiple_canvases_circle.html as that's where I got the idea for the "factor" ;)