search

google-research / mint

Multi-modal Content Creation Model Training Infrastructure including the FACT model (AI Choreographer) implementation.
Apache License 2.0
502 stars 87 forks source link

freezing motion video #9

Open NewCoderQ opened 3 years ago

NewCoderQ commented 3 years ago

@liruilong940607 Hi, Ruilong, i run the evaluation code with the pretrained model you provided on google drive, however the result video like this, the first two seconds motion from the seed motion, and then got freezing motion in the following seconds. I don't know why. Could you please give me some suggestions?

oxygenman commented 3 years ago

same problem,after 120 frams,the range of motion becomes smaller,some times it keep still, is there need to set some parameters? @liruilong940607

tosaka0822 commented 3 years ago

where is the evaluation code can result video? only python tools/calculate_fid_scores.py 100%|████████████████████████████████| 268/268 [14:10<00:00, 3.17s/it] Evaluation: FID_k: 33566817085848.8477, FID_g: 190.4702 Evaluation: Dist_k: 138815.0493, Dist_g: 3.4433

NewCoderQ commented 3 years ago

You need to visualize the predicted dance motion like this.

caijianfei commented 2 years ago

same problem, is there any suggestion to solve the problem ?

NewCoderQ commented 2 years ago

@shanyangmie Any update for this issue?

shanyang-me commented 2 years ago

Please make sure that the parameters are set correctly, otherwise this shouldn't be a problem.

caijianfei commented 2 years ago

@shanyang-me many thx for your feedback, I used audio feature from cached features output by "tools/preprocessing.py" (aist_audio_feats/mBR0.npy) with 3241 frames, and used first 120 frames from motion annotation data (annotations/motions/gBR_sBM_cAll_d04_mBR0_ch01.pkl) as motion seed, but the output motion still freezing after about 300-400 frames, my code and generated motion can be gotten from https://github.com/caijianfei/google-dancing, Is there any wrong setting? I would appreciate it if you could give me some suggestions, thx very much

YiAthena commented 2 years ago

I met the same problem as @caijianfei .

Concernin9 commented 2 years ago

You need to visualize the predicted dance motion like this.

@NewCoderQ Hello, the result of the code is the data in NPY format. I found that to form an smpl model, do I need to convert NPY format into PKL format first? I want to visualize the results. Is this the process? Looking forward to your reply, thank you very much!

Concernin9 commented 2 years ago

@shanyang-me many thx for your feedback, I used audio feature from cached features output by "tools/preprocessing.py" (aist_audio_feats/mBR0.npy) with 3241 frames, and used first 120 frames from motion annotation data (annotations/motions/gBR_sBM_cAll_d04_mBR0_ch01.pkl) as motion seed, but the output motion still freezing after about 300-400 frames, my code and generated motion can be gotten from https://github.com/caijianfei/google-dancing, Is there any wrong setting? I would appreciate it if you could give me some suggestions, thx very much

@caijianfei Hello, the result of the code is the data in NPY format. I found that to form an smpl model, do I need to convert NPY format into PKL format first? I want to visualize the results. Is this the process? Looking forward to your reply, thank you very much!

Concernin9 commented 2 years ago

where is the evaluation code can result video? only python tools/calculate_fid_scores.py 100%|████████████████████████████████| 268/268 [14:10<00:00, 3.17s/it] Evaluation: FID_k: 33566817085848.8477, FID_g: 190.4702 Evaluation: Dist_k: 138815.0493, Dist_g: 3.4433

@tosaka0822 Hello, have you solved this problem?

NewCoderQ commented 2 years ago

You need to visualize the predicted dance motion like this.

@NewCoderQ Hello, the result of the code is the data in NPY format. I found that to form an smpl model, do I need to convert NPY format into PKL format first? I want to visualize the results. Is this the process? Looking forward to your reply, thank you very much!

No, you don't need to convert pose data from .npy to .pkl file, you can read the pose and translation data from the .npy file, and inference with SMPL model.

YiAthena commented 2 years ago

hi @NewCoderQ I found It could be slightly better if set the bpm to 120.

h310558606 commented 2 years ago

You need to visualize the predicted dance motion like this.

@NewCoderQ Hello, the result of the code is the data in NPY format. I found that to form an smpl model, do I need to convert NPY format into PKL format first? I want to visualize the results. Is this the process? Looking forward to your reply, thank you very much! Hello, have you solved this problem?

YiAthena commented 2 years ago

hi @NewCoderQ i trained from scratch and solved the freezing problem.

h310558606 commented 2 years ago

Hi, Could you tell me how to transfer the npy file to the bvh file? The bvh_writer.py can not work at all.

------------------ 原始邮件 ------------------ 发件人: "google-research/mint" @.>; 发送时间: 2022年2月25日(星期五) 下午5:09 @.>; @.**@.>; 主题: Re: [google-research/mint] freezing motion video (#9)

hi @NewCoderQ i trained from scratch and solved the freezing problem.

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: @.***>

NewCoderQ commented 2 years ago

hi @NewCoderQ i trained from scratch and solved the freezing problem.

Well done! I will have a try.

caijianfei commented 2 years ago

hi @NewCoderQ i trained from scratch and solved the freezing problem.

Great!! How long did u take to retrain the model ?

hzy-del commented 1 year ago

You need to visualize the predicted dance motion like this.

@NewCoderQ Hello, the result of the code is the data in NPY format. I found that to form an smpl model, do I need to convert NPY format into PKL format first? I want to visualize the results. Is this the process? Looking forward to your reply, thank you very much!

hello，have you solved the problem?