search

facebookresearch / MCC

Multiview Compressive Coding for 3D Reconstruction
Other
631 stars 48 forks source link

Get *.obj file of images generated by DALLE-2 #5

Closed Dandelionym closed 1 year ago

Dandelionym commented 1 year ago

MCC works on images generated from DALL·E 2. Since the generated images do not include depth, we use an off-the-shelf depth prediction model to estimate depth for these images before feeding to MCC.

Hi, may I ask, how to get the *.obj file corresponding to the image generated by DALLE-2?

I've tried:

Question: Could you please add another demo to help solve this problem?

Thanks a lot. :-)

chaoyuaw commented 1 year ago

Hi @Dandelionym,

Thanks for trying out! Yes, let me provide an outline first:

  1. Create a camera, e.g., PyTorch3D's FoVPerspectiveCameras
  2. Unproject depth into XYZ coordinates using, e.g. https://github.com/facebookresearch/pytorch3d/blob/main/pytorch3d/implicitron/tools/point_cloud_utils.py#L23 or https://github.com/facebookresearch/pytorch3d/blob/main/pytorch3d/renderer/cameras.py#L1385 This will give you the xyz points!

Please let me know if you have any questions!

zhangfuyang commented 1 year ago

Hi. I have one related question.

If I understand correctly, DPT model can only predict relative (normalized) depth from rgb image. Do you project this relative depth to get xyz points? Will this be a problem for the Pointcloud encoder part? Thank you!

chaoyuaw commented 1 year ago

Hi @zhangfuyang ,

As shown in demo.py, we normalize the input xyz points to be zero-mean and unit-variance, so the scale won't be an issue.