Unexpected white pixels observed when rendering certain objects from different viewpoints.

[kaustubh-sadekar]

🐛 Bugs / Unexpected behaviors

Unexpected white pixels are observed when rendering certain objects from different viewpoints. I changed the gamma parameter to verify if it is because of the opacity factory but the white pixels persist for any value of gamma.

Images for different values of gamma are shared here:

If I change the camera distance, the number of unexpected white pixels reduce, as shown in the following image:

I am unable to understand why this is happening and how to solve the problem.

I am sharing the code and 3D models to reproduce the issue.

Instructions To Reproduce the Issue:

import os
import torch
from skimage.io import imread
import cv2
import random
import numpy as np
# Util function for loading meshes
from pytorch3d.io import load_objs_as_meshes, load_obj
import time
# Data structures and functions for rendering
from pytorch3d.structures import Meshes
from pytorch3d.vis.plotly_vis import AxisArgs, plot_batch_individually, plot_scene
from pytorch3d.vis.texture_vis import texturesuv_image_matplotlib
from pytorch3d.renderer import (
    look_at_view_transform,
    FoVPerspectiveCameras,
    OpenGLPerspectiveCameras,
    PointLights,
    diffuse,
    Materials, 
    RasterizationSettings, 
    MeshRenderer, 
    MeshRasterizer,  
    SoftPhongShader,
    TexturesAtlas,
    BlendParams,
)

BlendParams.background_color = (1.,1.,1.)
BlendParams.gamma = 0.001

if torch.cuda.is_available():
    device = torch.device("cuda:0")
    torch.cuda.set_device(device)
else:
    device = torch.device("cpu")

random.seed(43)

# Set paths
DATA_DIR = "./data_folder"
obj_filename = os.path.join(DATA_DIR, "obj5/models/model_normalized.obj")

# Get the rotation and translation matrix for the desired camera pose
R, T = look_at_view_transform(0.9, 35, 180) 
# For less unexpected white pixels set dist = 1.5

# Applying the rotation and translation to update the camera mtrix
cameras = FoVPerspectiveCameras(device=device, R=R, T=T)

# Settings for rasterization
raster_settings = RasterizationSettings(
    image_size=512, 
    blur_radius=0.0, 
    faces_per_pixel=1,
    max_faces_per_bin = 50000,
)

# Setting the lights
lights = PointLights(
    ambient_color=((0.7, 0.7, 0.7), ),
    diffuse_color=((0.1, 0.1, 0.1), ),
    specular_color=((0.1, 0.1, 0.1), ),
    device=device,
    location=[[0.0, 0.0, 1.5]])

# Setting the renderer
renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras, 
        raster_settings=raster_settings
    ),
    shader=SoftPhongShader(
        device=device, 
        cameras=cameras,
        lights=lights
    )
)

# Load obj file
verts, faces, aux = load_obj(obj_filename, load_textures=True, create_texture_atlas=True, texture_atlas_size=64)
mesh = Meshes(
    verts=[verts],
    faces=[faces.verts_idx],
    textures=TexturesAtlas(atlas=[aux.texture_atlas]),
)

mesh = mesh.to(device)

# Rendering the image
images = renderer(mesh)
img = cv2.cvtColor(images[0].cpu().numpy(), cv2.COLOR_BGR2RGB)
cv2.imshow("img", img)
cv2.waitKey(0)

Link to the 3d model is provided here: data_folder.zip

Once again I would like to thank the contributors for their active guidance and kind support.

[gkioxari]

This happens because pixels end up living on the edge of the faces. They will go away if you increase your blur_radius.

[kaustubh-sadekar]

@gkioxari Thank you so much for solving my query. It is now clear to me why this error occurs. I also tried changing the blur_radius. I am facing the error discussed in #342 and #327. Unlike #342 I need to render the texture using the TextureAtlas method as I have more than one texture file.

[gkioxari]

We are working on a fix for the issue of differentiability with TextureAtlas. The fix will be published soon.