LiveDepthEnabled in ProjectionMappingSample

[zhaoyh630]

Hi Andy,

I'm working on a project, which uses Kinect+projector to detect staircases and highlight them with projection. I tried to use RoomAlive and also used the ProjectionMappingSample to see how depth information can be mapped to the environment in real time. The callibration seemed to work fine and I can generate a calibration.xml file and import it into the sample. However, when I turned on the "LiveDepthEnabled" option, I came across some problems:

(1) It seems that this option cannot work by itself. If I only enabled this option, disabling all the others, such as "ThreeDObjectEnabled" or "WobbleEffectEnabled", the program will only generate a black image. I can only make it work when I enabled another option (e.g., WobbelEffectEnabled) at the same time. Is this the case or I actually did something wrong?

(2) When I stream the depth image with the "LiveDepthEnabled" turned on, the image seemed to be duplicated. Here's the projection image I got at the beginning (seems fine):

But if I move the paper bag, there are two images of my arm and the bag rendered at the same time, like this:

Could you please help me figure out what is wrong with this?

I also tested the ColorCameraLoop function (which is commented in the source code), and that worked well. Could you please provide some information why only the depth image was not rendered correctly? Thank you so much!

Best, Yuhang

[thundercarrot]

Hi Yuhang,

I'll tackle both issues separately.

1. The program is behaving as it should, since in essence you didn't ask it to render anything. "ThreeDObjectEnabled" renders a 3D object as a view dependent "hologram" using projection mapping, taking into account the shape of the projection surface. Adding "LiveDepthEnabled" causes it to update the shape of the projection surface at video rates, from the depth cameras. Otherwise it just uses whatever depth image is collected during calibration-- in other words, its geometric model of the projection model is fixed. "WobbleEffectEnabled" causes an IllumiRoom-like effect where the RGB image collected during calibration is projected, and is distorted upon triggering. One approach for your application would be to follow the wobble effect shader, but change it so that it projects only on the stairs (and probably does not have "LiveDepthEnabled", since the stairs are static).

2. Coloring occluded surfaces is not handled particularly well by the current set of shaders. Considering just the depth and color cameras, when there is a bit of geometry from the depth camera that is not seen by the color camera (because they are separated), the depth mesh will pick up the wrong color. The example you have is pretty extreme. I wonder what it looks like in the real world, with the projector?

btw. be careful with shiny walls. Verify that the depth image looks correct there. Depth cameras like Kinect tend to be confused by specular surfaces.

[zhaoyh630]

Hi Andy,

Thank you so much for your quick reply. This is super helpful! One thing that I want to verify is, I actually want to create a mobile device, so that people can walk with it and the system can highlight staircases in real time. According to your explanation, should I recognize stairs based on the depth information, and then conduct corresponding modification to the wobble effect shader in real time to highlight each stair? In this case, it's more like that I want to conduct the projection mapping in a mobile situation (the geometric model is changing all the time), then should I keep "LiveDepthEnabled"?

Best, Yuhang

[thundercarrot]

Yes if you are moving the camera and projector together, you want "LiveDepthEnabled". How you texture that live mesh is up to you. The Wobble effect shader uses the color image captured during calibration.

Using the live color image while projecting has the problem that you create a feedback loop between color camera and the projector which usually is undesirable (think about it, its like when you bring a microphone too close to the speaker when giving a talk). That is why the live color loop is disabled in the sample.

You might consider computing some version of the depth image and using that instead. In which case the wobble sample may still be useful, but instead of computing texture coordinates for the color image, you use depth image coordinates.

[zhaoyh630]

Hi Andy,

Thanks for the information! I tried to generate color texture with depth coordinates (straight yellow lines along the edge of each stair step), but the line was distorted, not straight any more. I guess it's because of the mapping to the real time 3D structure?

So I'm thinking whether I can use the calibration data in the xml to map between the depth camera and the projector directly. I've already recognized the edges of each stair on the depth image. For example, for one stair, I've got the edge on the depth image, which is a line segment with the two end points P1(x1, y1), P2(x2, y2). Can I project these two points to the projector image directly and generate a new line? I've read a prior post, where you mentioned that if i have a 3D point in the depth camera, I can use the projector pose matrix(4x4), and the CameraMath.Project() function to get the 2D point in the projector image. Can I use this method (I'm not using Unity for now)? Say for the line I want to project P1P2, I can get the depth info of the two points on the depth image z1, z2. However, the x and y axis is in pixel, but z1 is in meter, would that be a problem? Should I use Kinect's camera space to get the 3D point ((0,0,0) is at the center of the IR sensor and 1 unit = 1 meter)? Also, if I project the depth 3D point to 2D projector image with Project function, is the 2D projector image be the same size with the projector resolution?

Best, Yuhang

[thundercarrot]

If you have x and y in pixels and z in meters, you can convert to (x, y, z) in meters easily with the focal length f of the depth camera. If you are in the CPU you can use the Kinect SDK's conversion functions. If you are in the GPU, look at some of the shaders in the toolkit read from the depth texture and convert to 'depthCamera' coordinates via the stored values in the vertex buffer. Then using CameraMath.Project will get you to image coordinates.

If you want to do everything in shaders, I would probably try something simple initially, such as projecting the depth image back into the scene. That way you will know when you got it right, since features in the depth image will line up with real world features. To do this, sample from the depth image rather than the color image (duh) and rework the calculation of texture coordinates (which is actually simpler, since you don't need to transform to the color camera). Maybe you already tried this?