Plotting Hand2Dboxes

[JoaoM07]

Hi. I'm struggling to adapt the code on your HOT3D_Tutorial for object2Dbbox to plot the hand 2d bboxes. Object have unique identifiers which can be called in to create a list. Any suggestions on how to adapt this to hands2Dbboxes? Also, is it possible to pull eye gaze info using the hand 2D bbox timestamp?

Many thanks, Joao

P.S. - Awesome dataset!

[SeaOtocinclus]

Hello @JoaoM07, thank you for your questions and comment on the dataset.

Object2Dbbox and Hand2DBbox are using the same concepts, (hands will have an index to depict the HANDedness, rather than boxes with hand uid)

Also, is it possible to pull eye gaze info using the hand 2D bbox timestamp? All data_providers are based on timestamp query.

• So, for each 2d bounding box you can retrieve a timestamp_ns in timecode domain.
• To query the corresponding eye_gaze information you can use a similar code path as the Visualizer (https://github.com/facebookresearch/hot3d/blob/main/hot3d/Hot3DVisualizer.py#L244C9-L248C10)

aria_eye_gaze_data = (
       hot3d_data_provider.device_data_provider.get_eye_gaze(timestamp_ns)
       hot3d_data_provider.get_device_type() is Headset.Aria
        else None
    )

and see here how to handle the reprojection (https://github.com/facebookresearch/hot3d/blob/main/hot3d/Hot3DVisualizer.py#L329)

• You are right, we should extend the HOT3D_Tutorial to show how to use the hands2Dbboxes and eyegaze and will most likely do in the coming weeks.

Regarding eye_gaze I would advise you to look to the code of the Visualizer.

[JoaoM07]

Thanks for your reply.

Re: hands 2D boxes : https://github.com/facebookresearch/hot3d/blob/main/hot3d/data_loaders/HandBox2dDataProvider.py

def parse_box2ds_from_csv_reader(csv_reader) -> HandBox2dTrajectoryCollection: ..... hand_index = int(row[header.index("hand_index")])

I'm trying to adapt the code on the tutorial file. The bit /i'm struggling with is finding a reference to hand index. Object 2D box seems to have a method for this but not hands. The only reference I can spot if the above snippet of code. Apologies if this seems a silly question but how would I create the list of indices for hand left and right so I can get a different colour boxes for left and right hands.

Many thanks, Joao

[SeaOtocinclus]

Handedness indexes are defined here https://github.com/facebookresearch/hot3d/blob/ec57c636c24356d3d3424862ba0ca58d26357e88/hot3d/data_loaders/loader_hand_poses.py#L23

Then each Hand2dBoxes will be indexed with those Handedness here https://github.com/facebookresearch/hot3d/blob/main/hot3d/data_loaders/HandBox2dDataProvider.py#L47C5-L47C11

[JoaoM07]

Thanks for your tips! I've been able to add some code to the tutorial jupyter notebook and the HOT3D visualizer.py files, allowing me to visualize hand2Dbboxes. Happy to share the code, probably not professional level.

Will look to do the same for the eye gaze data.

[SeaOtocinclus]

Following your feedback I'm on my way to update the Tutorial with a hand2Dbboxes and eye gaze section, so you will have more resources soon to look at.

[SeaOtocinclus]

See commit above. I'm closing the issue, feel free to reopen if you need more assistance.