ArcticSF on HO3D

[heydarshahi]

Hi and thanks for the awesome HOLD code release!

I wonder if you have also tried adapting Arctic models to work with HO3D without an off-the-shelf pose estimator?

Best, Amin

[zc-alexfan]

It was asked in one of our reviewer question when we submitted ARCTIC. In general, it won't work because you need to align the canonical space of ARCTIC and HO3D objects, then the object 6d poses will be consistent across datasets.

We tried an experiment where we pretrain on ARCTIC and finetune on HO3D and it helped a bit with the object pose estimation.

[heydarshahi]

Thanks for the response. Trying this for my master thesis, I got one observation and one question:

1. On ARCTIC data: I noticed that the trained ArcticSF can reconstruct hands even when I change the object query_name to a wrong object and without giving the network any information on the object keypoints in the canonical space, using only wrapper.model.forward--which gets only the image and the query_name. Is it the case that ArcticSF is agnostic to the object when estimating hand pose, or is there something I'm missing?

2. For HO3D data: Do you happen to have the errors you observed when trying ArcticSF on HO3D? Using the evaluation script provided with HO3D, I got a very high joint/vertex average mean distance--in the order of 52cm 23cm.

Thanks and Best, Amin

[zc-alexfan]

1. Yes, the ArcticNetSF model should be able to estimate hand poses regardless of object type.
2. 23cm seems like an obvious error somewhere (our hand is around 15cm; imagine if we randomly select 3d points on your hand, it will be lower than 23cm). If you are computing root-relative MPJPE, you should subtract hand joints by its root.

[heydarshahi]

The 23cm is the mean joint/vertex distances of the right hand. However, we don't subtract the means and that's why the number is much higher than the root-relative MPJPE.

Thanks for the response! Please feel free to close the issue :)