search

setarehc / diffusion-motion-inbetweening

The official implementation of Flexible Motion In-betweening with Diffusion Models, SIGGRAPH 2024
https://setarehc.github.io/CondMDI/
Other
92 stars 6 forks source link

About body_models #5

Closed PeterWangyi closed 2 months ago

PeterWangyi commented 3 months ago

Hello author, after I configured according to the readme. When running： python -m sample.synthesize --model_path ./save/condmdi_uncond/model000500000.pt --num_samples 10 --num_repetitions 3

Report an error：

(condmdi) wangyubo@3c293718-02d8-11ef-9e72-9a357108b988:/mnt/AFS_wangyubo/projects/diffusion-motion-inbetweening$ python -m sample.synthesize --model_path ./save/condmdi_uncond/model000500000.pt --num_samples 10 --num_repetitions 3 Traceback (most recent call last): File "/mnt/AFS_wangyubo/.conda/envs/condmdi/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/mnt/AFS_wangyubo/.conda/envs/condmdi/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/mnt/AFS_wangyubo/projects/diffusion-motion-inbetweening/sample/synthesize.py", line 20, in <module> from data_loaders.amass.utils import utils File "/mnt/AFS_wangyubo/projects/diffusion-motion-inbetweening/data_loaders/amass/utils/utils.py", line 11, in <module> fk = ForwardKinematicsLayer() File "/mnt/AFS_wangyubo/projects/diffusion-motion-inbetweening/data_loaders/amass/utils/fk.py", line 30, in __init__ smpl_data = np.load(smpl_fname, encoding='latin1') File "/mnt/AFS_wangyubo/.conda/envs/condmdi/lib/python3.7/site-packages/numpy/lib/npyio.py", line 417, in load fid = stack.enter_context(open(os_fspath(file), "rb")) FileNotFoundError: [Errno 2] No such file or directory: './body_models/smpl/male/model.npz'

I found that the download link does not include the smpl model. And if I want to prepare this model myself, is it smplh or just smpl? In Humanml3d, it is smplh.

Looking forward to your reply!

JasonNero commented 3 months ago

Hey @PeterWangyi, see PR #6 for a quick fix :)

setarehc commented 3 months ago

Hi Peter,

Thanks for catching this! I modified data_loaders/amass/utils/utils.py which resolves synthesize.py's dependence on the smpl/male folder.

Please note that the smpl/male is only required if you're working with the AMASS dataset, which isn't part of the paper and is partially implemented in the current version of the code. It's from a modified version of SMPL used in [NeMF] (https://github.com/c-he/NeMF). You can downloaded it from [here] (https://drive.google.com/file/d/1a5P1eJfNsDO3vaEfkYI_fKwl_sad0GAz/view?usp=sharing) if needed. I'll update the README and add information about AMASS support.

Best, Setareh

PeterWangyi commented 3 months ago

Hi author, thank you for your patience in explaining this, this is an outstanding work!

Also thanks to @JasonNero for the effort.

I can now use the model for inference correctly, and I am also training to replicate the results in the paper, thank you!

PeterWangyi commented 3 months ago

Sorry for reponing this issue. How can I achieve the motion generation conditional on a certain trajectory mentioned in the paper?

Figure 4 shows examples of the model provided with only the root joint trajectory (projected on the ground in the left figure), or with only the right wrist joint.

setarehc commented 3 months ago

No problem at all. You can use the --edit_mode flag to specify the spatial condition. For Figure 4's examples:

We have several pre-defined conditions in the get_keyframes_mask function located in utils/editing_utils.py that you can use. If you need a spatial condition that isn't included, feel free to write your own conditioning mask.

I will update the README with information regarding edit_mode shortly.

Best, Setareh

PeterWangyi commented 3 months ago

Thanks for your reply!

When I tested python -m sample.conditional_synthesis --model_path ./save/condmdi_randomframes/model000750000.pt --edit_mode benchmark_sparse --transition_length 5 --num_samples 10 --num_repetitions 3 --text_prompt "a person walking straight" , the result I got was that input_motion was still the motion of: a person throwing a ball.

image

python -m sample.conditional_synthesis --model_path ./save/condmdi_randomframes/model000750000.pt --edit_mode pelvis --transition_length 5 --num_samples 10 --num_repetitions 3 --text_prompt "a person walking straight" the result was the same, it seems that prompt input has no effect.

What could be the reason?

Looking forward to your reply!

setarehc commented 3 months ago

With --edit_mode set to benchmark_sparse and transition_length set to 5, keyframes are being defined every 5 frames. This creates a very strong keyframe condition that leaves little room for the text condition to influence the results.

For optimal use of text conditioning, it's better to condition on specific joint trajectories. This way, you can control these joints while allowing the text prompt to guide the movements of the other, free joints.

You can also increase the classifier-free sampling weight for text conditioning by setting guidance_param to values higher than 2.5 to increase the effect of text.