Does FCGF work for small objects captured by RGBD camera?

[WilliamKRobert]

Thanks for publishing this great work! I made an initial experiment in which resunet model is trained on my custom dataset. Comparing to the 3DMatch and KITTI dataset, objects are pretty small in this dataset. For example, say the input is the point cloud of a mug, I would like to register the (partial) point cloud with corresponding CAD (complete) model points.

I was not able to get reasonable results. I noticed that the evaluation metrics used in your paper is too large for my dataset, e.g. \tau_1 = 0.1m (0.01m is suitable for my application). Am I doing something wrong or the FCGF model is actually not accurate enough to do object registration?

[chrischoy]

Hmm, I also did some experiments with small CAD models, ModelNet40 dataset, and it works reasonably well.

You have to adjust the quantization size (voxel downsampling) accordingly. 2.5cm is not good enough for a cup. It is a good quantization size for scenes that are at least 2.5m wide. Try using 2.5mm or 1mm. You have to guarantee that after quantization, it has at least 2k points. A low-resolution image 480x640=307200 ~300k data points. In 3D, things are a lot easier, but you still need a good resolution to figure out the accurate geometry.

Also, change your metric accordingly as well. I usually recommend x4 quantization for hit ratio test.

On a different note, make sure that the data is accurate. The hardest-contrastive loss is sensitive to noise as it picks up the hardest negative. If you don't have the accurate ground truth, the hardest negative mining would pick up the positive and force it to be at least margin away. If you can't control the ground truth, then try expanding the positive-pair-search-voxel-size-multiplier. https://github.com/chrischoy/FCGF/blob/master/lib/data_loaders.py#L112

[chrischoy]

Also, I think using the pretrained weights without training them with various quantization sizes [1mm ~1cm] would give you some estimate of the performance. If you think you've found a sweet spot, fine-tune the weights from the provided weights.

[WilliamKRobert]

Thank you for the suggestions. I did down scale the voxel size to 1mm as otherwise I got a assertion (exists_iter == coords_hashmap.map.end()) faild. A duplicate key found error.

Unfortunately, the observed point cloud in my dataset typically only has around 1k~1.5k points (since the object is small in the field of view of camera).

As for the quality of data, I am not sure how accurate they should be? If I use ground truth correspondences (calculated using ground truth pose), I still get RRE around 0.002 (RTE is 0.00) on my dataset. Is it good enough? If not, what value of positive-pair-search-voxel-size-multiplier would you suggest trying?

The pretrained models (3DMatch 2.5cm) do not give reasonable results on my dataset. I will train a model on 3DMatch with smaller voxel sizes.

[chrischoy]

I see. That somewhat explains the poor performance.

The error happens when you feed the same coordinate twice. You have to follow the procedure https://github.com/chrischoy/FCGF/blob/master/lib/data_loaders.py#L25 for correct downsampling.

Please review the 4D-SpatioTemporal ConvNets, CVPR'19 first to go over the hashing and voxelization process.

Also, resample mesh https://github.com/StanfordVL/MinkowskiEngine/blob/master/examples/modelnet40.py#L108 if you do not have enough points. Most CAD models are represented using sparse vertices and faces. If you don't sample points from the face, this information is lost.

[WilliamKRobert]

I was able to get good results via training the model on an easier custom dataset, i.e. the input two point clouds are both complete and have at least 2K points. The unsuccessful training on previous dataset is likely due to few points observed or overlapped. Since it is not the problem of the model, I am closing this thread.

Thank you so much for the clarification.

[chrischoy]

Great! Let me know if you have more problems with FCGF.