Render Camera from projector view for Projection Mapping

[masterofshapes]

Maybe Im missing something stupid, I've got everything setup like the tutorial for unity and working. But for some reason when I press play its always locked to the camera perspective from the Kinect. In my case Im trying to do projection mapping so I want to render the camera from the view of the projector. I read in your doc that it defaults to kinect view when no user is detected but is there a way to easily change to the projector view camera.

Thanks

[hbenko]

Do you still have MainCamera somewhere in the scene? It might be a leftover and should be removed as it will likely render on top of the projector.

If not... please provide a bit more details about your scene (images of your scene graph as well as User Views inspectors would be helpful). Have you completed both Tutorial 1 and 2?

[masterofshapes]

Thanks for the quick response. I'm using Unity 5.6.0f3 I've followed both tutorial 1 and 2 and I've made sure the main camera is deleted from my scene. Everything Appears functional, I can see where my projector and kinect are in the scene and they line up accurately to there real world position. I've got the layers setup like your tutorial but when I press play it always defaults to what seems like the view from the kinect camera.

Scene Hierarchy

My Room Object

Projector Object

User Object

Depth Mesh

Assets Folder

[masterofshapes]

Whats weird is even the example scenes default to kinect view. Is there an option I need to change in one of the scripts to default to the projector view instead? Maybe its a unity 5.6 specific issue but Im not getting any errors so thats probably unlikely.

Thanks in advance for your help

[hbenko]

Right now it seems that your user is not actually tracked from Kinect or any other camera (i.e., there is no specified RAT Skeleton Provider linked in RATUser script). This means that the User will have the exact same view as your Kinect camera (since both of those are set at the origin). If you position the user somewhere else (e.g., by dragging that object around), then their views should be different from Kinect.

This still does not explain why the projected view in Game window is the same as Kinect, which is what I believe you are experiencing, correct? Could you share a side-by-side view of your Scene and Game windows?

[hbenko]

Your projector and Kinect are not all that far apart (~0.5m)... so if you are looking at far-away surface and your user's view is in the same location as your Kinect, I don't suspect that there will many significant distortions in the projector image that you can observe. Moving the User's position, should drastically affect the rendered image though.

[masterofshapes]

Thanks for the response, if I understand correctly it sounds like if I just move the user to the location of the projector manually (stupid me for not trying this) it will line up? Will it preserve all the FOV and camera intrinsics ? My main use at the moment is to test some projection mapping on 3d objects so getting the precise view from the projector is important.

[hbenko]

I would not move the projector. If you did the calibration step correctly, the pose of the projector should be correct. I'd move the User... since that dictates the perspective of the view that is the image that is then projection-mapped onto the world.

[masterofshapes]

Sorry my wording was poor I meant move the user. Sounds like that should help I will keep you posted thanks again for the help

[hbenko]

Ok... let me know how it goes.

[reillydonovan]

I'm interested in the same thing and am experiencing the same issue... whenever I play the application in Unity it renders from the kinect perspective. For my purposes I only want projection mapping; view dependent projection is great too but not a feature I want to activate currently. Even so, I have followed tutorials 1 & 2 and cannot seem to get the view dependent projection to work. The kinectserver.exe is running with skeletal tracking and I can see the depth stream in unity. I have added a user object and assigned all game objects to the layers described but I can't seem to change the camera view away from the kinect. My preference would be to only render the projection view. Thanks for any suggestions you might have and for making a very powerful toolkit! Running Unity 5.6.0f3

[reillydonovan]

[masterofshapes]

@reillydonovan were you ever able to resolve?

[hbenko]

I believe I am seeing the same behavior on Unity 5.6. Has either one of you tried this in Unity 5.5? It seems that Unity 5.6 has changed how camera rendering is ordered which will likely impact how projection mapping is performed. I need to investigate this further, but in the meantime, I'd recommend trying it out on Unity 5.5 which is the latest version where the code was tested.

https://unity3d.com/unity/whats-new/unity-5.6.0 Graphics: Refactored camera render ordering code. When a Scene is rendered, it now figures out which cameras can share the same render target. The rules for this are: Cameras must use the same display to share the same render target. Cameras must share the same viewport. All cameras must have a depth buffer, or none of them must have a depth buffer. If the Scene is being viewed in VR, the cameras implicitly share the same render target.

[hbenko]

Since upgrading to Unity 5.6 it seems that the camera rendering order is broken. I have cameras with depth -100 rendering on top of cameras with depth 1.

I'm working on a fix, but it won't be posted until next week. In the meantime, I'd recommend using 5.5.

Others have also noticed this: http://answers.unity3d.com/questions/1345923/camera-rendering-order-no-longer-works-upon-upgrad.html

[masterofshapes]

Ah makes sense, I had a feeling it could be cuz of unity version. Thanks for the update I will switch to 5.5 thanks!

[reillydonovan]

Thanks benko! Checking 5.5 now... rolling back to 5.5 works and corrects the render order for the cameras

[masterofshapes]

@reillydonovan Glad to hear its working but Im getting some even weirder results. No matter what version of 5.5 I am getting just a black screen even on the example scenes (I can see the green virtual screen in TestRATScene1x1) but Im not seeing the room geometry. Whats weird is I go to version 5.6 and I can see the room geometry but I have the render order issue. Curious if you have 5.6 and 5.5 both installed or what was your process on getting success.

[reillydonovan]

Do you have any items flagged as a virtual texture? If you put your kinect scan onto the virtual texture layer you should be able to see it projected back onto the scanned surface.

[hbenko]

Folks, I've just pushed new updates for Unity 5.6 which incorporate the fix for messed up camera z-ordering. I've explained more in the markdown (see FAQ) for the project. Until I find a more permanent fix, this will have to do. Please try it out and report back if you have more issues.