different results

[jkpark0825]

When I run the code as written in the readme.md, the result is different from trained model. Why is it different and how to get trained model?

[fuqichen1998]

I can't reproduce the results of hand pose estimation using the provided checkpoint as well. I got even worse results when trained from scratch. Could @stevenlsw please help explain how to reproduce the performance claimed in the paper? Thanks in advance.

[stevenlsw]

Hi, @fuqichen1998, can you provide more training and testing logs as well as result details. Thanks!

[fuqichen1998]

Hi @stevenlsw , thanks for your reply! Below is the score output of the results generated using the checkpoint provided:

Collecting open3d-python
  Downloading https://files.pythonhosted.org/packages/5f/5c/a86082dc5efc3d22585e8aa22f9840667d9faa5e727b47c43137090caed4/open3d_python-0.7.0.0-cp27-cp27mu-manylinux1_x86_64.whl (3.7MB)
Requirement already satisfied: numpy in /opt/conda/lib/python2.7/site-packages (from open3d-python)
Requirement already satisfied: notebook in /opt/conda/lib/python2.7/site-packages (from open3d-python)
Collecting widgetsnbextension (from open3d-python)
  Downloading https://files.pythonhosted.org/packages/d7/31/7c1107fa30c621cd1d36410e9bbab86f6a518dc208aaec01f02ac6d5c2d2/widgetsnbextension-3.5.2-py2.py3-none-any.whl (1.6MB)
Requirement already satisfied: ipywidgets in /opt/conda/lib/python2.7/site-packages (from open3d-python)
Installing collected packages: widgetsnbextension, open3d-python
Successfully installed open3d-python-0.7.0.0 widgetsnbextension-3.5.2
Loading predictions from /tmp/codalab/tmpyXVmia/run/input/res/pred.json
Evaluation 3D KP results:
auc=0.490, mean_kp3d_avg=3.00 cm
Evaluation 3D KP PROCRUSTES ALIGNED results:
auc=0.797, mean_kp3d_avg=1.02 cm
Evaluation 3D KP SCALE-TRANSLATION ALIGNED results:
auc=0.507, mean_kp3d_avg=2.93 cm

Evaluation 3D MESH results:
auc=0.503, mean_kp3d_avg=2.89 cm
Evaluation 3D MESH ALIGNED results:
auc=0.804, mean_kp3d_avg=0.98 cm

F-scores
F@5.0mm = 0.232     F_aligned@5.0mm = 0.529
F@15.0mm = 0.685    F_aligned@15.0mm = 0.950
Scores written to: /tmp/codalab/tmpyXVmia/run/output/scores.txt
Evaluation complete.

and below is the content of the option.txt:

====== Options ======
HO3D_root: /mnt/ssd/qichen/ho3d_v2
blocks: 1
channels: 256
epochs: 60
evaluate: True
host_folder: exp_results
inp_res: 512
lambda_joints2d: 100.0
lambda_objects: 500.0
lr: 0.0001
lr_decay_gamma: 0.7
lr_decay_step: 10
mano_lambda_joints3d: 10000.0
mano_lambda_manopose: 10
mano_lambda_manoshape: 0.1
mano_lambda_regulpose: 1
mano_lambda_regulshape: 100.0
mano_lambda_verts3d: 10000.0
mano_neurons: [1024, 512]
mano_root: assets/mano_models
manual_seed: 0
momentum: 0.9
network: honet_transformer
obj_model_root: assets/object_models
resume: /home/qichen/Semi-Hand-Object/pretrained_models/model.pth.tar
save_results: True
snapshot: 10
stacks: 1
test_batch: 24
test_freq: 10
train_batch: 24
transformer_depth: 1
transformer_head: 1
use_cuda: 1
weight_decay: 0.0005
workers: 16
=====================
launched traineval.py at 2021-12-09 02:21:53.174338

and below is the content of object_results.txt

REP-5
{'021_bleach_cleanser': 0.3959691760521636, '006_mustard_bottle': 0.22134831460674156, '010_potted_meat_can': 0.055685131195335275}
ADD-10
{'021_bleach_cleanser': 0.8847065797273266, '006_mustard_bottle': 0.5696629213483146, '010_potted_meat_can': 0.48892128279883385}

I can also provide the logs about training the model by myself and its score.

[stevenlsw]

Thanks for the info. I am not fully sure the linked model is the right one. I am in travel now. I upload my local saved checkpoints https://drive.google.com/file/d/1Y4fICIY63MA4J1FiY8QfBAF_AKzMEjDn. Could you please @fuqichen1998 help me to take some evaluation. Thanks in advance.

[fuqichen1998]

The new checkpoint gives the same results as reported in the paper, thanks @stevenlsw. But after training the model from scratch multiple times, I can't achieve, or even get close to, the performance of the new checkpoint. Is there any difference in the training of the released checkpoint?

[stevenlsw]

Thanks so much for testing. The only difference in the provided checkpoint is that we are incorporating more pseudo labels from Something-v2 dataset as training data.

[fuqichen1998]

Got it, thanks for your reply!

[fuqichen1998]

Hi @stevenlsw, after several rounds of training, I could not reach the performance you reported training solely on the HO3D dataset. Could you please provide the instruction to reproduce the performance reported in the paper? (Also is this: https://github.com/stevenlsw/Semi-Hand-Object/blob/1aaf6eea5bdcfbacfdc3a04c8958baa2932e4de7/utils/options.py#L36 a typo or there is only 1 head for the contextual reasoning?)

[dncfjy]

Hi @fuqichen1998. Could you share the evaluation of groundtruth json to me? Thanks in advance.

[fuqichen1998]

@dncfjy we don't have it, it's hidden in the backend of the official HO3D challenge evaluation portal.

[dncfjy]

@fuqichen1998 Now the channel uploaded for the result prediction I do not see on the web page. Can you share a link with me? Thank you very much.