How to use Pytorch3D to render objects from Objaverse?

❓ Questions on how to use PyTorch3D

objaverse is a Universe of 10M+ 3D Objects, and its rendering script is based on bpy which is not differentiable. So I want to use Pytorch3D to render objaverse. However, I encounter some problems during this process. The results from the renderer are often blank images or have unusual camera angles.

I’m not sure how to adjust the parameters of PerspectiveCameras to enable the renderer to work well. Below is the code to reproduce my issue, along with some analyses I've made.

Code Demo

from pytorch3d.io import IO
from pytorch3d.io.experimental_gltf_io import MeshGlbFormat
from pytorch3d.renderer import RasterizationSettings, MeshRasterizer, PerspectiveCameras, look_at_view_transform, PointLights, MeshRenderer, SoftPhongShader
import matplotlib.pyplot as plt

'''
https://huggingface.co/datasets/allenai/objaverse/blob/main/glbs/000-000/0025c5e2333949feb1db259d4ff08dbe.glb
'''

device = "cuda"
image_size = (512, 512)
glb_path = "0025c5e2333949feb1db259d4ff08dbe.glb"

io = IO()
io.register_meshes_format(MeshGlbFormat())
with open(glb_path, "rb") as f:
    mesh = io.load_mesh(f, include_textures=True).to(device)

# Select the viewpoint using spherical angles  
distance = 3   # distance from camera to the object
elevation = 0.0   # angle of elevation in degrees
azimuth = 0.0  # No rotation so the camera is positioned on the +Z axis. 

# Get the position of the camera based on the spherical angles
R, T = look_at_view_transform(distance, elevation, azimuth, device=device)
cameras = PerspectiveCameras(device=device, R=R, T=T, image_size=(image_size, ), in_ndc=False)

cameras.to(device)
raster_settings = RasterizationSettings(
    image_size=image_size, 
    blur_radius=0.0, 
    faces_per_pixel=1, 
)
lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])
renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras, 
        raster_settings=raster_settings
    ),
    shader=SoftPhongShader(
        device=device, 
        cameras=cameras,
        lights=lights
    )
)

images = renderer(mesh)
plt.imshow(images[0, ..., :3].cpu())
plt.axis("off")

Problem Analysis

Texture is accurately loaded

plt.figure(figsize=(7,7))
texture_image=mesh.textures.maps_padded()
plt.imshow(texture_image.squeeze().cpu().numpy())
plt.axis("off")

Camera and mesh


from pytorch3d.vis.plotly_vis import plot_batch_individually

plot_batch_individually([mesh, cameras])


![image](https://github.com/user-attachments/assets/7a8a8505-196a-4a4d-97e7-a8a2ca596d27)

I would greatly appreciate any guidance or suggestions on how to resolve this issue. Thank you in advance for your help!

facebookresearch / pytorch3d