search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.78k stars 1.31k forks source link

How do I use this framework for single image multi objects reconstruction? #890

Closed sainatarajan closed 2 years ago

sainatarajan commented 3 years ago

Firstly, thank you all for this great library.

I have already explored a few models (Pixel2Mesh, Sphere-FC, Sphere-GCN) using Pytorch3D to reconstruct a single object from a single image. However, I'm not sure how to proceed with reconstructing many objects from a single image.

I have a pre-trained Object Detector that will give me the number of objects in the image. What I'm trying to do now, is to create multiple initial spherical meshes and wrap them in a Meshes() object, and pass it on to the network. For example: if my Obj. Det tool detects 4 objects, I create 4 spheres and input them into the network for training. I also have corresponding 4 .obj files that I use for the loss calculation.

Can someone let me know if this procedure is the right way to approach the problem?

I'm happy to provide further information if needed.

nikhilaravi commented 2 years ago

Hi @sainatarajan apologies for the late response. This would be a good question for the Discussions section of the repo: https://github.com/facebookresearch/pytorch3d/discussions.

Are you trying to use only 2D supervision (i.e. train the model using differentiable rendering) or use full 3D supervision as you have the obj files for each object?