search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.8k stars 1.32k forks source link

How do I create an object file to read into pytorch3d? #465

Closed mikeyEcology closed 3 years ago

mikeyEcology commented 3 years ago

❓ Questions on how to create an object file to read into pytorch3d

Hi, Thank you for creating pytroch3D. I'm just beginning and I don't understand how I would use this on new images/videos. In all of the tutorials, .obj files are downloaded. But I want to use my own images and video. Do you have suggestions for how to create .obj files from image annotation software, like AIDE or cvat? Sorry if this is an obvious question, but I haven't been able to find the answer. Thanks in advance.

gkioxari commented 3 years ago

I think you are not fully aware what the library offers and provides. Please go through our tutorials and videos that we have posted on the https://github.com/facebookresearch/pytorch3d/blob/master/README.md

Closing this, as it's not an issue with the library.

mikeyEcology commented 3 years ago

Thank you but none of these tutorials show how to create a .obj file. I can't find any explanation of how you would take an image with annotations and create a .obj file that could be used in pytorch3d? All of the examples use already-created .obj files. What would you do if you were starting a project with images from multiple cameras that you want to train 3D models on?

jcjohnson commented 3 years ago

An .obj file stores 3D meshes, which is a fundamentally different kind of data than images. An image is essentially a 2D grid of pixel values; in contrast a mesh is a set of vertices in 3D, with faces connecting vertices, and potentially with other information like textures, texture coordinates, surface normals, etc. Since .obj files store a fundamentally different kind of data than images, asking how to convert images into .obj files doesn't really make sense -- it's kind of like asking how to convert a .jpg image file into a .mp3 music file.

If you have a set of mesh data, you can create an .obj file using pytorch3d.io.save_obj -- but getting some mesh data in some format to start from is up to you.

There are algorithms which input a set of images and try to reconstruct 3D world geometry from them; this is essentially the task of multiview reconstruction. But this is not a simple file format conversion; instead it's a general problem in computer vision, and there are a wide variety of different algorithms that might make sense under different circumstances. We don't currently implement any classical multiview reconstruction algorithms in PyTorch3D, but this is something we might consider adding in the future.

gkioxari commented 3 years ago

This tutorial gives an example of you could reconstruct a 3D mesh (which then can be stored as an obj file) from a collection of 2D images from different viewpoints of the object.

https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/fit_textured_mesh.ipynb

mikeyEcology commented 3 years ago

Thank you @jcjohnson and @gkioxari !