Render white image with PerspectiveCamera but project normally on OpenCV

[nguyenhoangvudtm23]

If you do not know the root cause of the problem/bug and wish someone to help you, please post according to this template:

🐛 Bugs / Unexpected behaviors

Currently, I am working on school project shoe try-on that required rendering 3D shoes based on R and T get from OpenCV. I got key points 2D, 3D, and customized camera matrix then put them through solvePnP and some other functions of cv2 to get R, T matrix. The project points worked well on cv2, however, when I use 'cameras_from_opencv_projection' of pytorch3d then it rendered nothing. Another point is that this function also works well on a different image of mine (with different 2D keypoints only). I do not know why, can u guys help me, please? Note that: image '3DR_Phase1_000051_9Labs' work normally and image '3DR_Phase1_000046_8Labs' work incorrectly

NOTE: Please look at the existing list of Issues tagged with the label 'bug`. Only open a new issue if this bug has not already been reported. If an issue already exists, please comment there instead..

Instructions To Reproduce the Issue:

Please include the following (depending on what the issue is):

1. Any changes you made (git diff) or code you wrote

   
   # lib
   import os
   import sys
   import torch
   import os
   import torch
   import cv2 
   import numpy as np
   import json
   import matplotlib.pyplot as plt
   # Util function for loading meshes
   from pytorch3d.io import load_objs_as_meshes, load_obj

Data structures and functions for rendering

from pytorch3d.structures import Meshes from pytorch3d.vis.plotly_vis import AxisArgs, plot_batch_individually, plot_scene from pytorch3d.vis.texture_vis import texturesuv_image_matplotlib from pytorch3d.renderer import ( look_at_view_transform, FoVPerspectiveCameras, PerspectiveCameras, PointLights, DirectionalLights, Materials, RasterizationSettings, MeshRenderer, MeshRasterizer,
SoftPhongShader, TexturesUV, TexturesVertex ) from pytorch3d import utils

device

if torch.cuda.is_available(): device = torch.device("cuda:0") torch.cuda.set_device(device) else: device = torch.device("cpu")

path

name = '3DR_Phase1_000051_9Labs' img_path = f'{name}.jpg' json_path = f'{name}.json'

read image

img = cv2.imread(img_path) img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) kps = get_keypoint_object(json_path)

get keypoints

shs = [[], []] new_img = np.copy(img) for id, kp in enumerate(kps): id = id // NUM_TYPE_KEYPOINT
shs[id].append([int(kp.x), int(kp.y)]) shs = np.array(shs, dtype=np.int32)

file object

obj_path = 'left4.obj' verts, facesidx, = load_obj(obj_path) faces = faces_idx.verts_idx

Initialize each vertex to be white in color.

verts_rgb = torch.ones_like(verts)[None] # (1, V, 3) textures = TexturesVertex(verts_features=verts_rgb.to(device))

Create a Meshes object

mesh = Meshes( verts=[(verts).to(device)],
faces=[faces.to(device)], textures=textures )

some id on shoe 3D

idx = [10515, 3671, 22194, 3429, 3576, 13856, 13565, 9463] # shoe right

shoe points 3D shoe right

points_3D = verts.numpy()[idx, :]

shoe points 2D of shoe right

points_2D = shs[1].astype(np.float32)

opencv

camera matrix

distortion_coeffs = np.zeros((4,1)) size = img.shape focal_length_x = size[0] focal_length_y = size[0] focal_length = size[0] center = (size[1]/2, size[0]/2) matrix_camera = np.array( [[focal_length_x, 0, center[0]], [0, focal_length_y, center[1]], [0, 0, 1]], dtype = "double" )

solvePnP

success, vector_rotation, vector_translation = cv2.solvePnP(points_3D, points_2D, matrix_camera, distortion_coeffs)

convert to rotation matrix

rmat, jac = cv2.Rodrigues(vector_rotation)

project 3D point by opencv

project_point3D, jab = cv2.projectPoints(verts.numpy(), vector_rotation, vector_translation, matrix_camera, distortion_coeffs)

vis

prj_img_cv2 = np.copy(img) for p in project_point3D: cv2.circle(prj_img_cv2, (int(p[0,0]), int(p[0,1])), 1, (0,0,255), -1)

pytorch 3d camera params

sz = torch.from_numpy(np.array(img.shape[0:2]).reshape(1, 2)).to(device) R = torch.from_numpy(rmat.astype(np.float32).reshape(1,3,3)).to(device) T = torch.from_numpy(vector_translation.astype(np.float32).reshape(1,3)).to(device) mtx_cam = torch.from_numpy(matrix_camera.astype(np.float32).reshape(1,3,3)).to(device)

camera

cameras = utils.cameras_from_opencv_projection(R, T, mtx_cam, sz)

raster settings

raster_settings = RasterizationSettings( image_size=(img.shape[0], img.shape[1]), blur_radius=0.0, faces_per_pixel=1, )

mesh render

renderer = MeshRenderer( rasterizer=MeshRasterizer( cameras=cameras, raster_settings=raster_settings, ), shader=SoftPhongShader( device=device, cameras=cameras,

lights=lights

)

)

render

render_img = renderer(mesh)[0,...,:3].cpu().numpy().astype(np.uint8) * 255

plt.subplot(1,2,1) plt.imshow(prj_img_cv2) plt.title("Project points with opencv")

plt.subplot(1,2,2) plt.imshow(render_img) plt.title("render object pytorch3d")

plt.show()

2. The exact command(s) you ran:
[Here](https://drive.google.com/drive/folders/1bH7SaDaz9VCRQYh2V2DjIeDX4Rzng2H7?usp=sharing) is the link to the data that you can check: 
3. What you observed (including the full logs):

For image '3DR_Phase1_000051_9Labs': 
![image](https://user-images.githubusercontent.com/68849645/186076367-078d29ec-3eda-4419-98ca-c6a79f2873df.png)
For image '3DR_Phase1_000046_8Labs': 
![image](https://user-images.githubusercontent.com/68849645/186076508-0a358089-2433-4e6b-99e5-b33713f3c937.png)

I have checked the issue for days but not work. Please help me :(((

[nguyenhoangvudtm23]

I realized that if the vector translation in index 2 (z axis) be negative then the renderer will render nothing. Any idea about that?

[nguyenhoangvudtm23]

Also that the projected points from transform_points_screen and opencv are completely equal. Still don't get why render nothing ?

[bottler]

Can you use the plotly_vis functionality to plot the mesh and the (new) camera together? That should help you see what is going wrong.

[nguyenhoangvudtm23]

Sorry for late reply, I did try and found this the camera direction was not correct and i dont know why :(

[nguyenhoangvudtm23]

Can you use the plotly_vis functionality to plot the mesh and the (new) camera together? That should help you see what is going wrong.

I am a newbie in pytorch3d :((( so i don't know how to fix this, can you please help me on this?

[nguyenhoangvudtm23]

also see that when plot mesh I see that the object wihout camera seem to be different. what is the point here?

[minhlenhat21]

I also face the same problem. @nguyenhoangvudtm23 did you solve that?

[bottler]

Sorry - I missed this. You can see in the first plot that the camera is not facing the object. This should help debug. The difference in the plot may just be an axis difference because the camera has to be fitted in.

[oneThousand1000]

Same problem. I use cv2.solvePnP to get R and t, and construct a camera like:

camera = cameras_from_opencv_projection(R, T, K, img_size)

And the projection seems correct, I validate it by:

out_screen = camera.transform_points_screen(points_3d, image_size=img_size)

The projected points out_screen DO align with points_2d, but the camera is somewhat "behind" the object.

[oneThousand1000]

The most strange thing is, camera.transform_points_screen seems to work really well, but the results of rendering are confusing.

I also tried to use camera.transform_points_screen to project all the vertexes of the object onto the screen, the pixel coordinates also fall in the region of the screen ( 0<x<W and 0<y<H), but still nothing in the rendering results.

[nguyenhoangvudtm23]

The exact issue that I faced, I solved this by using epnp algorithm to find initialized rotation matrix and translation vector of the iterative pnp algorithm