search

microsoft / RoomAliveToolkit

Other
715 stars 191 forks source link

Planar surfaces #16

Open pyrochlore opened 8 years ago

pyrochlore commented 8 years ago

Hi, First thanks for the kind sharing of this good project.

I notice the sentence in README: The current release of the calibration tool does not address the case where the projection surface is flat (such as a bare wall). If the projection surface is flat, place some sizable objects, such as a couch, a few boxes, whatever you have handy, to create a non-flat projection surface.

I also notice there are codes in class ProCamEnsemble to check if points are in plane.

I am curious to know the reason why we should have that constrain. If I want to make an improvement to enable plane solving, what should I do? Any suggestion?

Thanks!

thundercarrot commented 8 years ago

Thanks for asking. Basically solving for camera pose and intrinsics requires non-coplanar points (Hartley and Zisserman is a good reference on this). Intuitively, if you have a planar calibration pattern, a change in focal length is confounded by translating in z. So solving for projector post and intrinsics simultaneously will fail (try it).

I've spent a few days working out a separate calibration path (see CameraMath.cs in the develop branch) that will handle the planar case. In this case you'll have to fix the projector intrinsics (i.e., they will need to be computed separately on a non-planar pattern, and fixed in subsequent calibration on the planar pattern).

The existing code that detects planar calibration points is actually flawed in at least one way: what matters is whether the entire set of all points (from all cameras that contribute to the projector) is planar, not the points coming from each camera. However, considering the case where a camera's set is planar, even then this will require a different version of the DLT algorithm to obtain initial estimates of the pose; see CameraMath.cs for the PlanarDLT).

Anyway, it's in process; I think I need just a few more hours of quality coding time to finish this off.

pyrochlore commented 8 years ago

Good to hear the planar calibration is in progress. I just got the book you mentioned (Multiple View Geometry in computer vision by Hartley and Zisserman, is it right one?) and try to understand the part of code. Maybe it will be more clear for me after few days reading. Thanks.

AlanDZhang commented 8 years ago

I also noticed the requirement that "each of these overlapping regions must be not be planar" and it was validated in our experiment (it would not work if we put objects in non-overlapping area). I wonder whether it is the same reason you mentioned here, or is it something different?

thundercarrot commented 8 years ago

Yes, it's the same reasoning.

thundercarrot commented 8 years ago

I checked in bunch of code to handle planar surfaces (to both branches). See the README for some comments on how to deal with planar surfaces. In a nutshell, as long as you have performed some calibration on a nonplanar surface, so that calibration can pick up projector focal length and principal point, thereafter planar configurations will be detected and handled appropriately.