PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

[joapolarbear]

PointNet, a novel type of neural network that directly consumes point clouds,

• which well respects the permutation invariance of points in the input.
• A unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing.

Github. PDF.

Challenge

• Typical CNN require highly regular input data
• The irregular format of the point cloud, previous work, convert it to regular 3D voxel grids or collections of images
• Point cloud is a set of points, invariant to permutations of its members, (unordered)

Design

• Each point is represented by (x, y, z), plus extra feature channels such as color, normal, etc.
• Use a single symmetric function, max pooling for Unordered Input
• concatenating the global feature with each of the point features, per point feature is aware of both the local and global information, used for shape part segmentation and scene segmentation.

• [ ] Effectively the network learns a set of optimization functions/criteria that select interesting or informative points of the point cloud and encode the reason for their selection.
  • [ ] More interestingly, it turns out that our network learns to summarize an input point cloud by a sparse set of key points, which roughly corresponds to the skeleton of objects according to visualization.