Obtain better result?

[kexul]

Hi, I'm trying to recover the motion of my favorite player from the game, but the result seems not good so far. Any advice?

https://github.com/user-attachments/assets/e50daf5b-9ce9-437d-b337-481e7e5f7300

https://github.com/user-attachments/assets/5c8b7cd3-5e68-4323-9bda-355f9db614c1

I did not use the -s option since I've failed installing the DPVO package in my notebook, would adding -s give better result? Or it's not possible to recover the motion when the camera is shaking so badly?

[carlosedubarreto]

From previous codes I've tried, yeah, DPVO is important to get the correct position regarding the global position on video.

and from the previous tests I've done, DPVO is also a pain to install on windows 😅

[Lidhra]

Install Ubuntu WSL on Windows work for me for DPVO

From previous codes I've tried, yeah, DPVO is important to get the correct position regarding the global position on video.

and from the previous tests I've done, DPVO is also a pain to install on windows 😅

Install Ubuntu WSL on Windows work for me for DPVO

[kexul]

Thanks for your info @carlosedubarreto & @Lidhra , I'll try WSL.

[kexul]

Result with dpvo, is it getting better? I'm not sure..

https://github.com/user-attachments/assets/ef93106b-6512-45f3-a0ff-104cae27806e

[zehongs]

Hi, thanks for your interest! We're very happy to see the potential use cases of GVHMR. From my personal observation, global translation doesn't perform as well during large movements as it does for smaller, more constrained motions. This is primarily due to the fact that large-scale movements are almost nonexistent in the training data—we trained on very limited academic datasets. We're currently working on a follow-up paper to address this issue—stay tuned!

[kexul]

Hi, thanks for your interest! We're very happy to see the potential use cases of GVHMR. From my personal observation, global translation doesn't perform as well during large movements as it does for smaller, more constrained motions. This is primarily due to the fact that large-scale movements are almost nonexistent in the training data—we trained on very limited academic datasets. We're currently working on a follow-up paper to address this issue—stay tuned!

Wow! Can't wait!

[kexul]

@zehongs Do you think adding similar data in the training would help? I can record some videos using my own character in the game engine and shake the camera.

[zehongs]

Sure. Do you have access to the global motion annotations?

[kexul]

Sure. Do you have access to the global motion annotations?

Yes, I can access the motion data when using my own character.

[zehongs]

This sounds good. Please refer to https://github.com/zju3dv/GVHMR/blob/main/hmr4d/dataset/h36m/h36m.py and prepare the necessary preprocessed data, namely,

            "smpl_params_w": smpl_params_w,
            "T_w2c", # (F, 4, 4)
            "gravity_vec": gravity_vec,  # (3)  gravity direction in world coordinates
            "K_fullimg": K_fullimg,  # (F, 3, 3)
             ######
            "bbx_xys": bbx_xys,  # (F, 3), this could be computed from images
            "f_imgseq": f_imgseq,  # (F, D), this could be computed from images

Other data could be derived from these data.

[kexul]

This sounds good. Please refer to https://github.com/zju3dv/GVHMR/blob/main/hmr4d/dataset/h36m/h36m.py and prepare the necessary preprocessed data, namely,

            "smpl_params_w": smpl_params_w,
            "T_w2c", # (F, 4, 4)
            "gravity_vec": gravity_vec,  # (3)  gravity direction in world coordinates
            "K_fullimg": K_fullimg,  # (F, 3, 3)
             ######
            "bbx_xys": bbx_xys,  # (F, 3), this could be computed from images
            "f_imgseq": f_imgseq,  # (F, D), this could be computed from images

Other data could be derived from these data.

Thanks for you help! I'll take a look.