gkioxari
commented
4 years ago Hi @bryanhpchiang
We prefer not to comment on other efforts but focus on our own. Thus, I will answer the question of "why PyTorch3d?".
PyTorch3D is a library of reusable 3D components that was built to facilitate research in the intersection of 3D understanding and deep learning. As part of our efforts to push our research agenda forward we found we needed to implement 3D operators and data structures that can efficiently handle
-
heterogeneous batches of 3D data. We provide data structures for creating batches of 3D data (meshes or pointclouds) of different sizes. Our data structures allow to efficiently switch between representations based on the input requirements of existing PyTorch operators, effectively bringing together PyTorch and 3D data. More details here.
-
batched and differentiable 3D operations. We provide 3D operators that can efficiently process batches of heterogeneous data, thus avoiding time consuming
for loops. This was achieved by writing custom CUDA functions for 3D operators. Note that this is crucial for maintaining a healthy training cycle when training deep learning models on batches. For example, in Mesh R-CNN, in a single forward pass we need to process batches which contain hundreds of 3D meshes of various sizes. Efficient batched operators allowed us to train a model in 2-3 hours. Alternatively, without handling batches and without our efficient 3D operators, our training time would have increased approximately by20xto40x! -
modular differentiable rendering. We implemented our own mesh and pointcloud differentiable renderer in an effort to modularize the different subcomponents in the rendering pipeline. This required some re-thinking of the pipeline which led to a very efficient and modular implementation that allows users to swap in and out any custom component they wish to experiment with without breaking the implementation. Our modular renderer was used in SynSin to render 3D pointclouds of learned features for the task of novel-view synthesis from a single image. You can read more details here.
This is a succinct description of PyTorch3D and also provides some insight into our motivation behind building such a library. If your work has similar needs and could benefit from the above features, then I think PyTorch3D is a good fit for you.
https://github.com/NVIDIAGameWorks/kaolin
Would love to hear any design/goal differences -- thanks so much!