Given a 2D image with multiple objects, how can I convert these objects to (textureless) meshes?

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data

https://pytorch3d.org/

Other

8.85k stars 1.32k forks source link

Given a 2D image with multiple objects, how can I convert these objects to (textureless) meshes? #606

Closed monacv closed 3 years ago

monacv commented 3 years ago

Is PyTorch3D able to do what the neural renderer in https://github.com/JiangWenPL/multiperson/tree/master/neural_renderer is doing in converting a 2D object in a 2D image to 3D mesh? If so, could you please provide pointers to an example code?

nikhilaravi commented 3 years ago

@monacv please have a look at:

The PyTorch3D tech report which has details of how to use PyTorch3D for single image 3D reconstruction using differentiable rendering. We will release the code for this soon.
Mesh R-CNN, which also reconstructs objects in 3D and uses PyTorch3D but doesn't use differentiable rendering.

Are you looking for a pretrained model or to train a model yourself?

monacv commented 3 years ago

@nikhilaravi
Thanks a lot for your time responding me :)

yes I am looking for a pretrained model. In your video you mentioned you are closest to SoftRasterizer (if I am not making a mistake) so I was hoping I could run your pretrained models on images. What classes of objects do you cover and how can I run your pretrained model on an RGB image? Does it perform better than Neural Mesh Renderer (NMR) (CVPR'18) and what metric are you using for comparison?

gkioxari commented 3 years ago

@monacv Please read our tech report that has a comprehensive analysis (quantitative and qualitative) to SoftRas. The comparison with NMS vs. SoftRas has been provided by the SoftRas tech paper and the differences in the two rendering pipelines are quite stark (NMR renders one face per pixel - SoftRas renders all faces per pixel - PyTorch3D renders K faces per pixel).

As a side note, I want to mention that the github issue page is meant to be for reporting erros or bugs in the library. For any discussions or questions on modern literature, Github Discussions is the more appropriate place.