Undesired skew when objects gets moved outside the main ProCam unit

[DBFBlackbull]

The Problem: In our current setup, we have 3 ProCam units, that all face flat white walls. We have calibrated the room successfully using big white boxes for non planar surfaces, but once we start to move the objects around the walls it didn't behave as expected.

From our understanding, a setup as ours would translate to a very wide desktop in the x direction, with a normal hight in the y direction. Instead it seems like the perspektive is all wrong as the objects move up at a 45 degree angle once they leave the center projector. We wonder if this is due to the location of our ProCam units during the calibration or our Kinect used for user tracking.

Here is a video of out setup with objects being moved around, showcasing the problem. https://youtu.be/qOLY7pav-8E The problem is showed about halfway through the video and the Kinect used for hand tracking is visible and lit up in when I start to move the object around.

And here is a link to our calibration so you can take a look at the generated data http://cs.au.dk/~peterbm/skew/3_new_pos2/

The Desired Feature: For my final setup I would prefer a "wide desktop" where I can move objects around, solely by translation in the x and y coordinate, without having to rotate anything. If this is not possible, then any hints as to how to achieve this would be much appreciated, even if the solutions turns to hard coding some rotation based on coordiantes

[thundercarrot]

Thanks for posting the video. It very clearly illustrates what you are talking about.

In fact everything looks like it is working very well for you. What you might be missing is that the projection mapping techniques that are demonstrated in the sample are really all about viewpoint dependent 3D rendering; i.e., the "user view" is the view that you want the user to see given a particular head position in the room, regardless of the actual physical geometry of the room. You can simulate, for example, a huge space with objects moving around, and if the lighting in the room is right (turn off all ambient light) the user may really have the impression that they are no longer in the small room they started out with. The fact that the object appears to dramatically foreshorten as it moves over to the left wall is exactly correct, for a given viewpoint.

The reason the object has a bit of skew initially is because you probably haven't determined the head pose with respect to the ensemble. What you need is some means of bringing whatever head pose you set into the ensemble's coordinate system (which is based on the first camera in the .xml file). The fact that there is a bit of skew there is probably due to the fact that that Kinect camera is pointing down a bit in the room, and you are not accounting for this orientation when imputing a head position. It's fine by the way to use a static head position, but you need to get the relative pose of the head with respect to that first camera. The hardest part is getting the orientation. Try the SDK's FloorClipPlane to get the orientation; if that doesn't work, you can click on the wall in the depth image, convert to world coordinates and take the cross product a couple of times to form up a rotation matrix.

Given all that, it sounds like the effect you are looking for is not a 3D viewpoint dependent rendering, but more of a "decal" or "picture frame" on the wall. (btw there is a group at Georgia Tech doing something similar-- I can put you in touch with them if you like). You can use the same set of techniques to create this effect, but you must place your image as you would hang a picture in the real world. Try orienting and positioning your image so that it is just above the geometry of the wall. Try doing this for each of the walls in your room.

You could make a panoramic "desktop" by placing 3 quads, each placed precisely as you would hang picture frames in the real world, and then set their texture coordinates so that each takes a chunk of the some input texture.

Make sense?

[DBFBlackbull]

About the initial skew, you are exactly right about us not taking the orientation of the users head. We tried to look at the quads from different angels, and by using a step ladder, and placing the users head very close to the first Kinect camera (the one in the center of the coordinate system) the perspektive actually looked okay, even as the quads moved to the left and right, as they seemed to be turning around the users head if the user was tilted forward about 45 degree (Michael Jackson style, just like the Kinects are) To counteract this we simply tried to staticly move the head position down (editing the headPosition vector's y coordinate down), however this never worked, as the orientation did not change. I believe adding another rotation matrix to the transformation matrix, rotating around the x axis in order to make the "user view" "look up", would solve this?

Instead of doing this manually (through trial and error), how I understand you we should use the first Kinect (in our example the one viewing the center wall), and find its orientation, in order to determine how much it is "looking down" and counter that in our head tracking / static head position to input a "look up" matrix through rotation.

You are exactly right, about the "picture frame" on the wall. That is exactly what I want to do. What I ment about the panoramic desktop is that I would like to freely move the pictures around all the walls, not combine all three walls into showing a single picture by texture mapping. That is a beyond my current focus.

About placing the pictures "just as in the real world" is not really a big consern, I think, as a future feature that I would like to implement is either zooming or scaling each picture, which I plan on doing either through matrix scaling or simply by increasing the quads Z coordinate, moving it closer to the user.

Bonus: A little bonus question. Currently the quads are lit by a single pointed light, meaning their surfaces are not all lit up when moved further away from the light. I could counter act this by creating and moving around a light source for each quad that would shine directly at it, but that seems like overkill. Is there a way in the shaders, to "remove" or "by pass" the lightsource completely and making each quad light up at "max" or some specific value like in desktop duplicate? Currently I'm looking at the MeshWithTexturePS.hlsl file, and it seems like giving color variable a static number instead of the saturate methods output would do the trick, or am I looking in the wrong place?

[thundercarrot]

My suggestion would be to find the rotation/translation that brings the first depth camera into some reasonable coordinate system for your room (e.g., pointing straight at the wall, y = 0 corresponding to the floor, etc).

Re: lighting, if you want to defeat the lighting you are looking in the right place.

[DBFBlackbull]

I realised I've forgotten to give an update on how this ended up. We decided to take the first Kinect camera down from the ceiling, and place it on a tall object, having it being completely level looking straight at the wall, at roughly user head height. This resolved the issue of the skew just as you said. Now everything handles as expected since the coordinate system now is as expected.

This isn't quite the Computer Science solution to the problem, but more often than not, the simplest solution is the best.

About placement of the quads, you were right, as in I had to position them exactly in the virtual world related to where I want them in the real world. Pure translation does not cut it. By making a simple rotation function that rotates the quads based on their x position I am able to move them around as desired. It now looks more convincing.

Thanks for all the help