own dataset

[Dong09]

hello,tks for this amazing work. And how to run this on my own dataset without annot_file?

[JaehaKim97]

Hi @Dong09,

The dataset without an annotation file can not be used for training our method, as our method requires a 3D hand pose at each time step during training.

But I guess visualization using a pre-trained model is still possible, and you can do that by modifying the code to not calculate the number by commenting out the evaluation process.

[Dong09]

Thanks for your reply. Actually, I want to be able to see the rendered effect on the local image or video, but the load_data function of the test script needs 4 json files to get the datalist, is there any way to do it

[JaehaKim97]

I guess you need to modify some code to do that.

I recommend you make your own dataset python file, e.g., src/data/own_dataset.py, which not loads json file, and just loads the image.

[Dong09]

I guess you need to modify some code to do that.

I recommend you make your own dataset python file, e.g., , which not loads file, and just loads the image.src/data/own_dataset.py``json

okk, i will try it. thks for your reply! :)

[Dong09]

I guess you need to modify some code to do that.

I recommend you make your own dataset python file, e.g., src/data/own_dataset.py, which not loads json file, and just loads the image.

sorry, i am back again. i have made my own dataset.py and now i have no idea about result visualization like example in README, could you mind do a demo? i would be grateful if could.

[Dong09]

hello, when i try to render mesh in the img, i get some problem. img and code are following.

def render_mesh(img, mesh, face, cam_param):
    # mesh
    mesh = trimesh.Trimesh(mesh, face)
    rot = trimesh.transformations.rotation_matrix(
    np.radians(180), [1, 0, 0])
    mesh.apply_transform(rot)
    material = pyrender.MetallicRoughnessMaterial(metallicFactor=0.0, alphaMode='OPAQUE', baseColorFactor=(1.0, 1.0, 0.9, 1.0))
    mesh = pyrender.Mesh.from_trimesh(mesh, material=material, smooth=False)
    scene = pyrender.Scene(ambient_light=(0.3, 0.3, 0.3))
    scene.add(mesh, 'mesh')
    focal, princpt = cam_param['focal'], cam_param['princpt']
    camera = pyrender.IntrinsicsCamera(fx=focal[0], fy=focal[1], cx=princpt[0], cy=princpt[1])
    scene.add(camera)
    # renderer
    renderer = pyrender.OffscreenRenderer(viewport_width=img.shape[1], viewport_height=img.shape[0], point_size=1.0)
    # light
    light = pyrender.DirectionalLight(color=[1.0, 1.0, 1.0], intensity=0.8)
    light_pose = np.eye(4)
    light_pose[:3, 3] = np.array([0, -1, 1])
    scene.add(light, pose=light_pose)
    light_pose[:3, 3] = np.array([0, 1, 1])
    scene.add(light, pose=light_pose)
    light_pose[:3, 3] = np.array([1, 1, 2])
    scene.add(light, pose=light_pose)
    # render
    rgb, depth = renderer.render(scene, flags=pyrender.RenderFlags.RGBA)
    rgb = rgb[:,:,:3].astype(np.float32)
    valid_mask = (depth > 0)[:,:,None]
    # save to image
    img = rgb * valid_mask + img * (1-valid_mask)
    return img
self.opt_params['focal'] = [5000,5000]
self.opt_params['input_img_shape'] = [256,256]
focal = [self.opt_params['focal'][0] / self.opt_params['input_img_shape'][1] * out['bbox'][2], self.opt_params['focal'][1] / self.opt_params['input_img_shape'][0] * out['bbox'][3]]
princpt = [self.opt_params['princpt'][0] / self.opt_params['input_img_shape'][1] * out['bbox'][2] + out['bbox'][0], self.opt_params['princpt'][1] / self.opt_params['input_img_shape'][0] * out['bbox'][3] + out['bbox'][1]]

rendered_img = render_mesh(blur_img, mesh_out*np.array([1,-1,-1]), mano.face['right'], {'focal': focal, 'princpt': princpt})

[JaehaKim97]

Sorry for the late reply.

Did you get a solution?

As we used cropped hand images in both the training and inference phases, I think inputting images with the whole body may cause a problem.