search

nghorbani / moshpp

Motion and Shape Capture from Sparse Markers
Other
201 stars 30 forks source link

change the suface_model type from smplx to smpl #4

Open minushuang opened 2 years ago

minushuang commented 2 years ago

the code can work normally when the model type is smplx. and now I try to fit the smpl params instead of the smplx following the smplx moshpp process. at first I donwloaded the smpl basic models, and put them under support_files/smpl directory, then change the surface_model.typefrom smplx to smpl, and addedopt_weights.smpl config which is a copy of smplx as below.

opt_weights:
  smpl:
    stagei_wt_poseH: 3.0
    stagei_wt_poseF: 3.
    stagei_wt_expr: 34.
    stagei_wt_pose: 3.
    stagei_wt_poseB: 3.
    stagei_wt_init_finger_left: 400.0
    stagei_wt_init_finger_right: 400.0
    stagei_wt_init_finger: 400.0
    stagei_wt_betas: 10.
    stagei_wt_init: 300
    stagei_wt_data: 75.
    stagei_wt_surf: 10000.
    stagei_wt_annealing: [ 1., .5, .25, .125 ]
    stageii_wt_data: 400
    stageii_wt_velo: 2.5
    stageii_wt_dmpl: 1.0
    stageii_wt_expr: 1.0
    stageii_wt_poseB: 1.6
    stageii_wt_poseH: 1.0
    stageii_wt_poseF: 1.0
    stageii_wt_annealing: 2.5

the error when I run the mosh job(task)

2022-07-07 08:11:49.227 | INFO     | soma.tools.parallel_tools:run_parallel_jobs:54 - #Job(s) submitted: 83
2022-07-07 08:11:49.227 | INFO     | soma.tools.parallel_tools:run_parallel_jobs:67 - Will run the jobs in random order.
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:95 -- mocap_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/custom_low_head2_yz_swap.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:97 -- stagei_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/soma_subject1/male_stagei.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:98 -- stageii_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/soma_subject1/custom_low_head2_yz_swap_stageii.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:103 -- surface_model: type: smpl; gender: male; fname:/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/support_files/smpl/male/model.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:107 -- optimize_fingers: False, optimize_face: False, optimize_toes: False, optimize_betas: True, optimize_dynamics: False
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:prepare_stagei_frames:157 -- Selecting 12 frames using method manual on frames with 100% least_avail_markers
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:prepare_stagei_frames:197 -- Using stagei_fnames for stage-i: ['/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/run_002.pkl_001091'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_001.pkl_000137'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/run_001.pkl_001366'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_001.pkl_000509'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/throw_001.pkl_000596'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/dance_003.pkl_001488'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_001.pkl_000588'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/squat_002.pkl_001134'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_002.pkl_000471'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/run_001.pkl_000032'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/dance_001.pkl_001042'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/dance_001.pkl_000289']
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:mosh_stagei:241 -- Attempting mosh stagei to create /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/soma_subject1/male_stagei.pkl
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:120 -- using marker_layout_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/SOMA_unlabeled_mpc_smpl.json
soma_subject1 -- custom_low_head2_yz_swap -- bodymodel_loader:load_moshpp_models:93 -- Loading model: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/support_files/smpl/male/model.pkl
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:172 -- can_model.betas.shape: (300,)
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:173 -- opt_models[0].betas.shape: (300,)
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:192 -- Estimating for #latent markers: 53
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:229 -- Number of available markers in each stagei selected frames: (F00, 53), (F01, 53), (F02, 53), (F03, 53), (F04, 53), (F05, 53), (F06, 53), (F07, 53), (F08, 53), (F09, 53), (F10, 53), (F11, 53)
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:237 -- Rigidly aligning the body to the markers
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:262 -- MoSh stagei weights before annealing:
stagei_wt_poseH: 3.0
stagei_wt_poseF: 3.0
stagei_wt_expr: 34.0
stagei_wt_pose: 3.0
stagei_wt_poseB: 3.0
stagei_wt_init_finger_left: 400.0
stagei_wt_init_finger_right: 400.0
stagei_wt_init_finger: 400.0
stagei_wt_betas: 10.0
stagei_wt_init: 300
stagei_wt_data: 75.0
stagei_wt_surf: 10000.0
stagei_wt_annealing: [1.0, 0.5, 0.25, 0.125]
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:266 -- head_marker_corr_fname is provided and is being loaded: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/support_files/ssm_head_marker_corr.npz
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:274 -- Successfully took into account the correlation of the head markers
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:280 -- Beginning mosh stagei with opt_settings.weights_type: smpl
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:354 -- Step 1/4 : Opt. wt_anneal_factor = 1.00, wt_data = 1.00, wt_poseB = 65.09
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:356 -- stagei_wt_init for different marker types body = 300.00:
Traceback (most recent call last):
  File "test_soma.py", line 115, in <module>
    'randomly_run_jobs': True,
  File "/data/hxh/project/soma/src/soma/tools/run_soma_multiple.py", line 283, in run_soma_on_multiple_settings
    run_parallel_jobs(func=run_moshpp_once, jobs=mosh_jobs, parallel_cfg=moshpp_parallel_cfg)
  File "/data/hxh/project/soma/src/soma/tools/parallel_tools.py", line 79, in run_parallel_jobs
    func(job)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/mosh_head.py", line 598, in run_moshpp_once
    mp.mosh_stagei(mosh_stagei)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/mosh_head.py", line 245, in mosh_stagei
    v_template_fname=self.cfg.moshpp.v_template_fname)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/chmosh.py", line 421, in mosh_stagei
    [f'{k} = {np.sum(opt_objs[k].r ** 2):2.2e}' for k in sorted(opt_objs)])))
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/chmosh.py", line 421, in <listcomp>
    [f'{k} = {np.sum(opt_objs[k].r ** 2):2.2e}' for k in sorted(opt_objs)])))
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 708, in compute_r
    return self.a.r * self.b.r
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/reordering.py", line 376, in compute_r
    return np.concatenate([t.r for t in self.our_terms], axis=self.axis)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/reordering.py", line 376, in <listcomp>
    return np.concatenate([t.r for t in self.our_terms], axis=self.axis)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 594, in r
    self._call_on_changed()
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 589, in _call_on_changed
    self.on_changed(self._dirty_vars)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/prior/gmm_prior_ch.py", line 62, in on_changed
    for logl, w in zip(self.loglikelihoods, self.weights)])
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/prior/gmm_prior_ch.py", line 62, in <listcomp>
    for logl, w in zip(self.loglikelihoods, self.weights)])
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 319, in compute_r
    return np.sum(self.x.r, axis=self.axis)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 584, in compute_r
    return self.safe_power(self.x.r, self.pow.r)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 708, in compute_r
    return self.a.r * self.b.r
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 731, in compute_r
    return self.a.r.dot(self.b.r)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 566, in compute_r
    return self.a.r - self.b.r
ValueError: operands could not be broadcast together with shapes (69,) (63,)
minushuang commented 2 years ago

the length of pose_body_ids of smpl is 69 according to the code inchmosh.py, which is different from smplx 63

    if cfg.surface_model.type == 'smpl':
        pose_body_ids = all_pose_ids[3:] #69
    elif cfg.surface_model.type == 'smplh':
        pose_body_ids = all_pose_ids[3:66]
        if cfg.moshpp.optimize_fingers:  # dont chop chumpy variables two times
            pose_finger_ids = all_pose_ids[66:]
    elif cfg.surface_model.type == 'smplx':  # orient:3, body:63, jaw:3, eyel:3, eyer:3, handl, handr
        pose_body_ids = all_pose_ids[3:66]

modify the smpl pose_body_ids from pose_body_ids = all_pose_ids[3:] to pose_body_ids = all_pose_ids[3:66]
is a temporary way for running the process but not the ultimate solution I think.

minushuang commented 2 years ago

the problem is at the pose_body_prior.pkl used inmoshpp.prior.gmm_prior_ch.py, where the gmm['covars'] and gmm['means'] shapes are (8, 63, 63)and (8,63) respectively in the pkl file, so the npose=69doesn't work when the model_surface_type is smpl.

npose = 63 if exclude_hands else 69

covars = gmm['covars'][:, :npose, :npose]
means = gmm['means'][:, :npose]
weights = gmm['weights']
minushuang commented 2 years ago

the blender output mesh result is like #5 , the pose information is lost with correct shape and orientation.

ritaank commented 1 year ago

Hi,

This is super helpful, and I am trying to recreate this currently. To my understanding, you downloaded the SMPL python model files, and then restructured them into the support_files/smpl folder?

As part of this process, I renamed the SMPL pkl files from the original basicmodel_f_... to model.pkl to match the structure of smplx. I then copied the pose_body_prior.pkl file. My MoSh run is currently throwing an error since I don't have the corresponding model.npz file for each body type. Is this file just derived from the pkl file?

ritaank commented 1 year ago

It seems the requisite .npz files stem directly from the .pkl files. The pkl files for SMPL can be opened as per the guidance in this comment. Special care must be taken for the scipy sparse matrices. Here's some boilerplate code:

import pickle
import numpy
import scipy

with open('./support_files/smpl/male/model.pkl', 'rb') as f:
    u = pickle._Unpickler(f)
    u.encoding = 'latin1'
    male_smpl_pkl = u.load()
    # print(male_smpl_pkl)

npz = {}
for k,v in male_smpl_pkl.items():
    if type(v) == np.ndarray:
        npz[k] = v
    elif type(v) == scipy.sparse.csc.csc_matrix:
        npz[k] = v.todense().astype(np.float32)
    else: # for strings and chumpy arrays
        npz[k] = np.array(v)

np.savez('./support_files/smpl/male/model.npz', **npz)