rongakowang / DenseMutualAttention

[WACV 2023] Interacting Hand-Object Pose Estimation via Dense Mutual Attention
MIT License
36 stars 3 forks source link

Same pretrained checkpoint (HO3D & DexYCB)? #10

Closed byydzh closed 1 year ago

byydzh commented 1 year ago

According to the two pre-trained models (snapshot_ho3d.pth.tar, snapshot_dexycb.pth.tar) you provided, the measured results are in line with the results mentioned in the paper. However, the two pre-trained checkpoints are identical except for the name, and cross-testing got the same results. This seems to indicate that this is the result of your single training, I am puzzled by this. Does that mean there's something special about its training?

rongakowang commented 1 year ago

Hi,

Thank you for your testing, In the training, we perform diverse data augmentation in view points, grasping approaches and background to improve the generalizability, and performed uniform data preprocessing on two benchmarks. Hence the model should behaves similarly on the two benchmarks.

In the final version, we re-evaluated all metrics several times and tided the code to ensure consistency. It might be that the model are incorrected uploaded from the same experiments. We will re-examinate these checkpoints soon. However, the performance should not be differring too much based on pervious observations.

Please let me know if you have other concerns.

byydzh commented 1 year ago

I see. Then I am thinking that your one pre-training model can achieve the results of the paper on two datasets at the same time, so will separate training get better results for separate datasets?

rongakowang commented 1 year ago

Hi,

Adding more data will in general improve the performance. However, since the augmented data are much more and diverse compared to the raw training data, from my recall we didnt observe significant performance difference in our first and final version.

byydzh commented 1 year ago

What you are talking about in your paper is the results obtained by training separately. So you mean that there is no difference between the metrics of DexYCB obtained by training the DexYCB dataset alone and the metrics of DexYCB after training two data sets at the same time? which seems strange.

rongakowang commented 1 year ago

Hi,

I meant by "no significant performance" not "identical", the actual metrics can differ but the conclusion of comparing with related works in the paper remains unchanged.

byydzh commented 1 year ago

thank you for the explanation, another question is do you have any plans to release the data augmentation code or related datasets?

rongakowang commented 1 year ago

Hi,

These parts come with the train code, which I'll try best finding time to tidy up and release should I am available.

Meanwhile, the detailed processes are described in the paper should you wish to re-implement. As they are generated online with the training, there are no data saved.