Texture disorder

[zwl995]

Hello, I have a very strange problem. The texture of the image I rendered seems to be very deranged, as shown in the following picture. 我使用的代码如下：

def get_gpu_info(size_thre: int):
    result = subprocess.run(['nvidia-smi', '--query-gpu=index,name,memory.total,memory.free,memory.used', '--format=csv,noheader,nounits'], stdout=subprocess.PIPE)
    gpus_info = result.stdout.decode().split("\n")
    gpus_info = [info for info in gpus_info if info]
    ava_gpu = set()
    for gpu_info in gpus_info:
        gpu = gpu_info.split(", ")
        if int(gpu[3]) > size_thre:
            ava_gpu.add(gpu[0])
    ava_gpu = list(ava_gpu)
    return ava_gpu

def render_thum_img(sfm_path: Path, render_obj_path: Path, json_path: Path):
    obj_path = render_obj_path
    # get intrinsics and view
    rec = pycolmap.Reconstruction(sfm_path)
    # transformer_json = [json_path]
    # print(f"transformer_info is: {transformer_json}")
    # if len(transformer_json) >= 1:
    #     with open(str(transformer_json[0]), 'r') as fp:
    #         transformer_info = json.load(fp)
    #         T_3x4 = np.array(transformer_info['transform']) * transformer_info['scale']
    #         sim3d = pycolmap.Sim3d().from_matrix(T_3x4)
    #         rec.transform(sim3d)
    # rec.write(render_obj_path.parent)
    # obj_scene = o3d.io.read_triangle_mesh(str(obj_path))
    # render_bbox = obj_scene.get_axis_aligned_bounding_box()
    # render_bbox_min = render_bbox.get_min_bound()
    # render_bbox_max = render_bbox.get_max_bound()
    # rec = rec.crop((render_bbox_min, render_bbox_max))
    max_vis_p3d = 0
    iid = 0
    for img in rec.images.values():
        if img.num_points3D() > max_vis_p3d:
            iid =  img.image_id
    Twc = rec.images[iid].cam_from_world.matrix
    Twc = np.concatenate([Twc, np.array([[0, 0, 0, 1]])], 0)
    # device = torch.device("cpu")
    # choice cuda
    ava_gpu = get_gpu_info(10240)
    if len(ava_gpu)==0:
        device = torch.device("cpu")
    else:
        device = torch.device(f"cuda:{ava_gpu[0]}")
    logger.info(f"Select device is: {device}")
    Twc = torch.from_numpy(Twc).float().to(device)
    Rwc = Twc[:3, :3].unsqueeze(0)
    twc = Twc[:3, 3].unsqueeze(0)
    cam = rec.cameras[rec.images[iid].camera_id]
    scale = 0.25
    fx = cam.focal_length_x*scale
    fy = cam.focal_length_y*scale
    cx = cam.principal_point_x*scale+0.5*scale-0.5
    cy = cam.principal_point_y*scale+0.5*scale-0.5
    width = int(cam.width*scale)
    height = int(cam.height*scale)

    image_size = torch.Tensor(((height, width),)).float().to(device)
    camera_matrix = torch.Tensor([[[fx, 0, cx], [0, fy, cy], [0, 0, 1]]]).float().to(device)
    cameras = camera_conversions._cameras_from_opencv_projection(Rwc, twc, camera_matrix, image_size)
    raster_settings = RasterizationSettings(
        image_size=(height, width), 
        blur_radius=0.0, 
        faces_per_pixel=1,
        bin_size=None
        # cull_backfaces=True,
        # max_faces_per_bin=len(obj_scene.triangles)
    )
    # Place a point light in front of the object. As mentioned above, the front of the cow is facing the 
    # -z direction. 
    lights = PointLights(device=device, location=[[0.0, 0.0, 0.0]])
    blend_params = BlendParams(sigma=1e-2, gamma=1e-2, background_color=(0.537, 0.537, 0.537))  
    renderer = MeshRenderer(
        rasterizer=MeshRasterizer(
            cameras=cameras, 
            raster_settings=raster_settings
        ),
        shader=SoftPhongShader(
            device=device, 
            cameras=cameras,
            lights=lights,
            blend_params=blend_params,
        )
    )

    mesh = load_objs_as_meshes([obj_path], device=device)
    images = renderer(mesh, zfar=500)
    images_texture_numpy = images[0, ..., :3].cpu().numpy()
    images_texture_numpy = cv2.normalize(images_texture_numpy, None, 0,255, cv2.NORM_MINMAX).astype('uint8')
    images_texture_numpy = cv2.cvtColor(images_texture_numpy, cv2.COLOR_RGB2BGR)
    cv2.imwrite(f"{render_obj_path.parent.parent}/test4.jpg", images_texture_numpy)
    # plt.figure(figsize=(10, 10))
    # plt.imshow(images[0, ..., :3].cpu().numpy())
    # plt.axis("off")
    # plt.savefig(f"{render_obj_path.parent.parent}/test3.jpg", bbox_inches='tight')
    logger.info(f"================ Finish Render Thum Img ================")

[zwl995]

Oddly enough, I have tried the following situations: 1.Colmap+OpenMVS generates multipart obj (small pieces are rendered successfully, and the texture disorder above will not occur until it is larger than 600m). The above problem occurs after the single obj generated by 2.Colmap+OpenMVS is cut by pymeshlab (about 300m). It is also normal for 3.Colmap+OpenMVS to generate small pieces of obj after clipping with pymeshlab. It seems that 1 can be speculated to be a matter of size, but 2 seems to overturn 1, 2 seems to speculate that it is the problem of pymeshlab tailoring, but 3 overturns 2. Can you give me some help?

[bottler]

I suspect the problem here might not be in the renderer. The actual obj that sits in obj_path, can you view it in an external viewer? What does it look like? Only if it looks "correct" but the pytorch3d renderer output does not would you know that this is a PyTorch3D issue.

[zwl995]

It's normal for me to open the texture with Meshlab, although it looks a little ugly (I use fewer images for reconstruction)

[zwl995]

In addition, I have a black noise when I render the room, which may be caused by some reason.

[bottler]

If you think there's a bug with either of these things, can you post exactly the files and code which produce a surprising output (as well as the background info). There is not enough information or explanation to help. E.g. in the last comment, what is in each image?

[zwl995]

I type the file and the code are packed here.

[bottler]

(not shared)

[zwl995]

ok，I opened the access.

[bottler]

The mesh is large with 3.6M vertices and twice as many faces. The obj file itself has meaningful colors. Is it possible to do the rendering on the CPU instead? I cannot see anything which would cause this.

[zwl995]

Hello, I just tried your suggestion, using cpu to render, and the result is the same; I also tried to use meshlab streamlined (90w face), but the result is still wrong

[bottler]

Can we say that if meshlab is "wrong" too, then this probably isn't a pytorch3d problem?

[zwl995]

No, I mean, after I use meshlab to simplify, meshlab opens normally, pytorch3d rendering is wrong.