Intermediate Output In Figure 1.

[18724799167]

How can I get the intermediate output in figure 1 in paper? thanks.

[evonneng]

Hi! An easy way to do it is use pytorch3d to plot the mesh of the geometry that is outputted by the renderer https://github.com/facebookresearch/audio2photoreal/blob/548aeeb2057465045ca1568d65ea059cea633d80/visualize/render_codes.py#L107

I haven't gotten around to writing this support, but if you beat me to it, would super appreciate a PR for rendering these meshes!! Otherwise, I will add this feature and ping this thread :)

[18724799167]

Hi！Thank you very much for your answer. I have been able to get the point cloud of the model.But there is no mesh.How can I get mesh. Thank you.

[evonneng]

Ah i see, here I think you only saved the 3d vertices from the output geom. But in order to plot the mesh, you will also need to save the faces. Here's some code you can use to do so:

def write_mesh(
    vertices_list,
    face_list,
    save_dir,
    frame,
    tex_coord=None,
    face_coord=None,
    texture=None,
    target=None,
):
    for i, (vertices, face) in enumerate(zip(vertices_list, face_list)):
        if target is not None:
            assert tex_coord is None
            mesh_simplifier = pyfqmr.Simplify()
            mesh_simplifier.setMesh(vertices, face)
            mesh_simplifier.simplify_mesh(
                target_count=target,
                aggressiveness=7,
                preserve_border=True,
                verbose=False,
            )
            vertices, face, _ = mesh_simplifier.getMesh()

        if not os.path.exists(save_dir):
            os.makedirs(save_dir)
        lines_obj = ""
        for v in vertices:
            lines_obj += f"v {v[0]} {v[1]} {v[2]}\n"
        if tex_coord is not None:
            for f, vti in zip(face, face_coord):
                lines_obj += f"f {int(f[0]+1)}/{int(vti[0]+1)} {int(f[1]+1)}/{int(vti[1]+1)} {int(f[2]+1)}/{int(vti[2]+1)}\n"

            lines_mtl = [
                f"newmtl material{frame}",
                "Ka 1.000 1.000 1.000",
                "Kd 1.000 1.000 1.000",
                "Ks 0.000 0.000 0.000",
                "d 1.0",
                "illum 2",
                "Ns 1.00000000",
                f"map_Kd material{frame}.png",
            ]
            lines_mtl = [line + "\n" for line in lines_mtl]
            with open(f"{save_dir}/material{frame}.mtl", "w", encoding="utf-8") as f:
                f.writelines(lines_mtl)
            lines_obj = f"mtllib material{frame}.mtl\nusemtl material{frame}\n"
            for vt in tex_coord:
                lines_obj += f"vt {vt[0]} {vt[1]}\n"
            texture = cv2.cvtColor(texture, cv2.COLOR_BGR2RGB)
            texture = cv2.flip(texture, 0)
            cv2.imwrite(f"{save_dir}/material{frame}.png", texture)
        else:
            for f in face:
                lines_obj += f"f {int(f[0]+1)} {int(f[1]+1)} {int(f[2]+1)}\n"

        with open(f"{save_dir}/human{frame}_p{i}.obj", "w", encoding="utf-8") as f:
            f.writelines(lines_obj)

to call into it, you can place the below line of code into the spot where you are saving your geometry:

write_mesh(
    vertices_list=[
        geom.squeeze().detach().cpu().numpy()[0],
    ],
    face_list=[
        self.model.geo_fn.vi.squeeze().detach().cpu().numpy(),
    ],
    save_dir=save_dir,
    frame=b,
)

b here is just the frame number you are processing, and save_dir is where to output everything.

Please let me know if there's more questions!

[evonneng]

Closing this for now due to inactivity. But please feel to reopen accordingly.