search

facebookresearch / hand_tracking_toolkit

This repository contains utilities for loading data from Meta's datasets with hand pose annotations, computing hand tracking metrics, and visualising results of a hand tracking method.
Apache License 2.0
30 stars 1 forks source link

Error when submitting shape estimation by hot3d #5

Closed hiroking0523 closed 1 month ago

hiroking0523 commented 1 month ago

Thank you very much for hosting this challenge.

I have one question regarding the submission process: when I submit to the hand shape estimation challenge using the hot3d dataset according to the submission format, I get the following error

Traceback (most recent call last):. File "/code/scripts/workers/submission_worker.py", line 538, in run_submission submission_metadata=submission_serializer.data, in File "/usr/local/lib/python3.7/site-packages/hand_tracking_toolkit/evaluation.py", line 270, in evaluate pred_dir, gt_dir, mano_model, dataset_suffix=f"{dataset_suffix}" File "/usr/local/lib/python3.7/site-packages/hand_tracking_toolkit/evaluation.py", line 203, in evaluate_shapedataset , shape_gt = load_gt_files(gt_dir, dataset_suffix) File "/usr/local/lib/python3.7/site-packages/hand_tracking_toolkit/evaluation.py", line 80, in load_gt_files shape_gt = json.load(stack.enter_context(open(shape_file))) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpkmcijilv/gt/gt_shape_hot3d.json'

The above error says gt_shape_hot3d.json, is there an annotation file available regarding test data for hot3d?

Thank you in advance.

lg-zhang commented 1 month ago

Hi, thank you for the questions. There seems to be some issue in the backend of the server. I’m working on fixing it today.

lg-zhang commented 1 month ago

Hi @hiroking0523, the issue has been fixed. Since HOT3D is a large dataset, for the pose evaluation, we will only evaluate every 10th frame, i.e. frame_idx=0, 10, 20, ... Please try reuploading the results, thanks!

hiroking0523 commented 1 month ago

@lg-zhang Thank you for your response. I just tried uploading again and got the following error

Traceback (most recent call last): File "/code/scripts/workers/submission_worker.py", line 538, in run_submission submission_metadata=submission_serializer.data, File "/usr/local/lib/python3.7/site-packages/hand_tracking_toolkit/evaluation.py", line 270, in evaluate pred_dir, gt_dir, mano_model, dataset_suffix=f"{dataset_suffix}" File "/usr/local/lib/python3.7/site-packages/hand_tracking_toolkit/evaluation.py", line 209, in evaluate_shape_dataset mano_model=mano_model, File "/usr/local/lib/python3.7/site-packages/hand_tracking_toolkit/evaluation.py", line 160, in compute_overall_shape_metrics shape_pred = sequence_name_to_shape_pred[seq]["mano_beta"] KeyError: 'test_aria_private/clip-003506'

Is this also a problem on the backend side of the server? If you have already addressed the submission regarding hot3d's shapeprediction, we would appreciate it if you could let us know.

Thank you in advance.

lg-zhang commented 1 month ago

@hiroking0523 It looks like the sequence_names are not consistent. Could you send your result file to my email linguang@meta.com so that I can take a closer look?

lg-zhang commented 1 month ago

@hiroking0523 Thanks for sending me the file. I fixed the annotation file on the server and tested with your submission. I noticed that you included results for clips that don't belong to "test_ht_shape" defined in clip_splits, for example "clip-003392". Please only use the clips in "test_ht_pose" for the pose track and "test_ht_shape" for the shape track.

hiroking0523 commented 1 month ago

@lg-zhang Thank you for your response. We are pleased to report that the submission was successful!

Thank you for your continued support. We would appreciate it if you could close this issue.