search

NVlabs / planercnn

PlaneRCNN detects and reconstructs piece-wise planar surfaces from a single RGB image
Other
556 stars 123 forks source link

Predicted plane parameters #12

Open vevenom opened 5 years ago

vevenom commented 5 years ago

Hi,

I find your plane segmentations very impressive. However, I have encountered some problems when trying to use the predicted plane parameters that are exported to "*_plane_parameters_0.npy". I have tried to reconstruct the depth map from these parameters but without any luck yet.

Is there some specific preprocessing step that needs to be applied to the plane parameter values before proceeding to the depth deconstruction step?

More specifically, given predicted plane normal n, depth map, plane segmentation mask and camera intrinsics, I calculate the plane offset parameter d as described in your work. Than for the individual plane, the depth reconstruction Z is given as:

Z = ((- d (K^(-1) u ) [3]) / (n^T ((K^(-1) u ))

where u is the homogeneous pixel representation of image pixel locations, n is the plane normal.

Am I assuming something wrong? Can you provide more information on the process of depth reconstruction from the plane parameters?

art-programmer commented 5 years ago

Your equation seems correct. But I might have used a different coordinate system. Y faces the front of the camera, X faces right of the camera, and Z faces downwards.

The depth maps for all planes can be computed using this function. https://github.com/NVlabs/planercnn/blob/e773eb3c7199f1d88b108ea8cefeecd9a9579401/utils.py#L439

You can then compute the final depth map based on the plane depth maps with plane masks.

vevenom commented 5 years ago

Oh, I didn't consider you might be using this exact coordinate system, you are right. So my coordinate system is:

X faces right, Y faces up, Z faces front

Anyways it works after transforming your plane parameters to my coordinate system like this: [X, -Z, Y]

Than you can use the formula from the original post (without 'minus') to get quality depth map:

Z = ((d (K^(-1) u ) [3]) / (n^T ((K^(-1) u ))

Thanks for your assistance, I have one more question. One alternative is to generate plane parameters from depth predictions and plane masks with SVD, similar to what you did for generating ground truth planes. Have you tried this experiment and if yes, have you observed any benefits of using plane predictions instead of this 'direct' approach?