annotation mismatch in GrabNet dataset

[lixiny]

Hi Taheri Thanks for the great work you have made.

Currently we are working on integrate GrabNet dataset into other project, we find an annotation mismatch in this dataset.

Here I will describe how the mismatch occurs.

From line # 68 - 73 in your code: GrabNet/grabnet/tests/test.py. we think the proper way to load object from your GRAB/data/object_meshes/contact_meshes is:

1. Rotating the original mesh w.r.t frame_data['root_orient_obj_rotmat']
2. Translate the rotated mesh w.r.t frame_data['trans_obj']

But after we followed the same steps as you described, the transformed object mesh has an apparent rotation and translation mismatch from the annotation frame_data['verts_object'].

I provide an example to reproduce the mismatch: For the GrabNet dataset train split, of object flashlight and frame id 17221.

            ### Acquire the frame_data ...

            rot_mat = frame_data["root_orient_obj_rotmat"].numpy().reshape(3, 3)
            transl = frame_data["trans_obj"].numpy()
            obj_verts_downsampled = frame_data["verts_object"].numpy()  # (2048, 3)

            ## obj_mesh is a TriMesh Object directly load from your GRAB/data/object_meshes/contact_meshes
            obj_verts_trasnformed = (rot_mat @ obj_mesh.vertices.T).T + transl

Then we use Open3D to visualize:

• obj_verts_downsampled (in PointCloud format)
• obj_verts_trasnformed (in TriangleMesh format)

We provide the screenshot of the example:

Thanks for your excellent work again! joy face And really hope you can help us to figure this out.

Lixin

[KailinLi]

Same here. I wonder whether I could get the scripts that convert from GRAB data to GrabNet data?

[lixiny]

Hi Taheri Thanks for the great work you have made.

Currently we are working on integrate GrabNet dataset into other project, we find an annotation mismatch in your this dataset.

Here I will describe how the mismatch occurs.

From line # 68 - 73 in your code: GrabNet/grabnet/tests/test.py. we think the proper way to load object from your GRAB/data/object_meshes/contact_meshes is:

1. Rotating the original mesh w.r.t frame_data['root_orient_obj_rotmat']
2. Translate the rotated mesh w.r.t frame_data['trans_obj']

But after we followed the same steps as you described, the transformed object mesh has an apparent rotation and translation mismatch from the annotation frame_data['verts_object'].

I provide an example to reproduce the mismatch: For the GrabNet dataset train split, of object flashlight and frame id 17221.

            ### Acquire the frame_data ...

            rot_mat = frame_data["root_orient_obj_rotmat"].numpy().reshape(3, 3)
            transl = frame_data["trans_obj"].numpy()
            obj_verts_downsampled = frame_data["verts_object"].numpy()  # (2048, 3)

            ## obj_mesh is a TriMesh Object directly load from your GRAB/data/object_meshes/contact_meshes
            obj_verts_trasnformed = (rot_mat @ obj_mesh.vertices.T).T + transl

Then we use Open3D to visualize:

• obj_verts_downsampled (in PointCloud format)
• obj_verts_trasnformed (in TriangleMesh format)

We provide the screenshot of the example:

Thanks for your excellent work again! joy face And really hope you can help us to figure this out.

Lixin

Hi, Taheri Problem solved! Still thank you for the GrabNet dataset.

After load the object rot_mat from disk:

rot_mat = frame_data["root_orient_obj_rotmat"].numpy().reshape(3, 3)

One should not directly apply the rot_mat on object vertices, but the rot_mat.T. eg.

obj_verts_trasnformed = (rot_mat.T @ obj_mesh.vertices.T).T + transl

Everything works fine.

I am also wondering whether would you are planned to release the code that convert GRAB to GrabNet data, especially the one that filtering the unstable hand-object interaction.

Best! Lixin

[otaheri]

Dear @lixiny and @KailinLi,

Thanks for your interest and your message.

As you already found the solution, the right way to go is to do the following:

obj_verts_trasnformed = obj_mesh.vertices@rot_mat + transl

Which is equivalent to what you are doing. The reason that we have .T at the end of L68 is that the Mesh package automatically does the transpose again.

About going from GRAB to GrabNet data we have used some heuristics to get stable grasps which unfortunately are not in our released code currently. I can make it available later once I have time. However, you can also do it yourself since this is not very difficult.

In the GRAB repo, you can add a filtering function similar to this to filter frames based on some heuristics. Here we only choose frames that have contact, but you can add other constraints like:

• the object should be moved from its initial position
• the thumb should always be in contact and etc.

I can add such function to the repo so you can use the exact same heuristic that we used.

I hope this helps and enjoy the data.

Best, Omid

[lixiny]

Thanks for your quick response and suggestion! I will close this issue.

Best Lixin