Attempt to submit for ScanNet?

[NicksonYap]

Hi Yan Yan, @traveller59

Awesome repo! (the Docker image really helps)

Currently SparseConvNet tops the Scan-Net 3D Semantic label benchmark http://kaldir.vc.in.tum.de/scannet_benchmark/ code: https://github.com/facebookresearch/SparseConvNet/tree/master/examples/ScanNet

Since SpConv is an improvement of SparseConvNet, could probably try to top it :P

SECOND is more suitable for the Scan-Net 3D Semantic instance benchmark The current winner (3D-SIS) uses multiple 2D RGB features + 3D to obtain semantic instance

Any intentions to extend SECOND to run ScanNet data? (contains ply rgb pointclouds and images)

Regards

[traveller59]

There are some works and limits to do if I provide support for semantic Instance (mask rcnn):

1. add point roi pooling and simple rcnn architecture (to do)
2. unlike fixed-size detection in LiDAR data, the multi-scale detection in scannet need multi-scale architectures such as FPN. I'm not familiar with them. the SECOND is a simple single-stage fixed-size detector.
3. SECOND is lidar-only detector. the image isn't used. only concatenate rgb features to point cloud is supported.
4. I don't have access to scannet data. web-based visualization tools are also needed.

In summary, the major problem is multi-scale RCNN architecture and data access/processing/visualization. It's not easy, I have no plan to develop the scannet data process/visualization, but a simple rcnn will be added in future.

[NicksonYap]

@traveller59

I've limited experience with this but I believe ScanNet data is similar of that to KITTI such that:

• It is also fixed-size (or at least, scale-corrected)
• Point cloud has RGB
• Accessing the point clouds may be simpler as the files are standard .ply files
• Using Open3D python lib, I manage to export out semantic labelled .ply and open using (MeshLab / CloudCompare)
• Open3D also provides visualization for point clouds http://www.open3d.org/docs/tutorial/Basic/pointcloud.html
• It has multiple images for a scene that could be extracted from .sens file using https://github.com/ScanNet/ScanNet/tree/master/SensReader/python (I'm not sure but the pixels should be mapped onto point cloud)

ScanNet can be downloaded by sending an agreement PDF to an email (It took me a day for a reply) https://github.com/ScanNet/ScanNet#scannet-data You'll be given a .py file to select what to download (it's a file called download-scannet.py :wink: )

By simple RCNN do u mean combining 2D Image data with the point clouds? If we're able to do that then I believe most of the work is already done :sweat_smile: