How to setup cameras?

[xhuan8]

Hi,

when try to render with cameras = FoVPerspectiveCameras(device=device, R=cam_R, T=translation) the rendered object on image is very small, translation[0][2] /= 10. makes it more closer but still smaller than expected.

if add K=cam_K get blank image. what have I missed?

raster_settings = RasterizationSettings(
        image_size=512, 
        blur_radius=0.0, 
        faces_per_pixel=1, 
    )

renderer = MeshRenderer(
                                        rasterizer=MeshRasterizer(
                                            cameras=cameras, 
                                            raster_settings=raster_settings
                                        ),
                                        shader=SoftPhongShader(
                                            device=device, 
                                            cameras=cameras,
                                            lights=lights
                                        )
                                    )

cam_R: 
tensor([[[1., 0., 0.],
         [0., 1., 0.],
         [0., 0., 1.]]], device='cuda:0')

translation:
tensor([[2.8361e-03, 1.3537e-01, 2.0617e+01]], device='cuda:0')

cam_K:
tensor([[[5.0000e+03, 0.0000e+00, 2.5600e+02, 0.0000e+00],
         [0.0000e+00, 5.0000e+03, 2.5600e+02, 0.0000e+00],
         [0.0000e+00, 0.0000e+00, 1.0000e+00, 0.0000e+00],
         [0.0000e+00, 0.0000e+00, 0.0000e+00, 1.0000e+00]]], device='cuda:0')

[bottler]

I don't know what you expect. May try keeping R=cam_R, T=translation but try also setting fov to something less than 60 (the default, in degrees)?

[xhuan8]

It was compared with a WeakPerspectiveCamera from pyrender, the result seems fit the 2D image.

the scale is round 1.0, so translation for z axis is around 20 according to the function, but 20 is quite far away in PyTorch3D.

either need add the parameter K to camera, or scale the translation, please advice.

s, tx, ty = cam_wp
camera = WeakPerspectiveCamera(
            scale=[s, s],
            translation=[tx, ty],
            zfar=1000.
        )

def convert_weak_perspective_to_camera_translation(cam_wp, focal_length, resolution):
    cam_t = np.array([cam_wp[1], cam_wp[2], 2 * focal_length / (resolution * cam_wp[0] + 1e-9)])
    return cam_t

def convert_camera_translation_to_weak_perspective(translation, focal_length, resolution):
    cam_wp = np.array([2 * focal_length / (resolution * translation[2] + 1e-9), translation[0], translation[1]])
    return cam_wp

[xhuan8]

here assume focal_length = 5000, I pass focal_length parameter to PerspectiveCameras but still didn't get things right

focal_length = torch.from_numpy(np.array([[5000.]])).float().to(device)
    cameras = PerspectiveCameras(device=device, R=cam_R, T=translation, focal_length=focal_length)

[xhuan8]

focal_length = 10

[bottler]

I'm sorry, I can't debug this for you.