Unable to use inferred npy to generate mesh

[icedwater]

I'm trying to generate some animation-friendly data from the inferred .npy, such as .obj but preferably .bvh or .fbx.

After following the suggested command, I get the error below. Is there a problem with the way the data is stored in the numpy ndarray?

python -m visualize.render_mesh --input_path save/results/x/condsamples000100000__benchmark_sparse_T\=60_CI\=0_CRG\=0_KGP\=1.0_seed39_doing_push-ups/sample00_rep00.mp4
Traceback (most recent call last):
  File "/home/iced/anaconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/iced/anaconda3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/iced/projects/CondMDI/visualize/render_mesh.py", line 29, in <module>
    npy2obj = vis_utils.npy2obj(npy_path, sample_i, rep_i,
  File "/home/iced/projects/CondMDI/visualize/vis_utils.py", line 18, in __init__
    self.bs, self.njoints, self.nfeats, self.nframes = self.motions['motion'].shape
ValueError: too many values to unpack (expected 4)

I checked the shape of the output and it is 5 dimensions, but if I add a _ in front of line 18 to ignore the first variable, I run into a different error:

./body_models/
WARNING: You are using a SMPL model, with only 10 shape coefficients.
Running SMPLify For sample [0], repetition [0], it may take a few minutes.
Traceback (most recent call last):
  File "/home/iced/anaconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/iced/anaconda3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/iced/projects/CondMDI/visualize/render_mesh.py", line 29, in <module>
    npy2obj = vis_utils.npy2obj(npy_path, sample_i, rep_i,
  File "/home/iced/projects/CondMDI/visualize/vis_utils.py", line 29, in __init__
    motion_tensor, opt_dict = self.j2s.joint2smpl(self.motions['motion'][self.absl_idx].transpose(2, 0, 1))  # [nframes, njoints, 3]
ValueError: axes don't match array

Is there something I'm missing about the dimensions of the data?

[JasonNero]

Hey @icedwater, The indexing seems to be broken, I fixed it in my fork a while ago (https://github.com/JasonNero/diffusion-motion-inbetweening/commit/8689999828c8d1f7ef6f19d1c15fe16b4dadcaae).

If you prefer BVH files, I recommend looking at joints2bvh.py from MoMask or originally "A Deep Learning Framework For Character Motion Synthesis and Editing". This takes the SMPL joint positions, converts them to joint rotations via Inverse Kinematics and saves those to a BVH file.

[icedwater]

Thanks @JasonNero! I had a comment saved in my other machine about the indexing but other tasks got in the way :)

[icedwater]

I'm not sure if it's because something in the generation process added an extra dimension to the ndarray somehow, but I was able to make a little bit of progress by replacing ['motion'] with ['motion'][0] in lines 15 and 26. (Probably also need to do this in 33.) However, this led to another issue later on in render_mesh, where npy2obj.real_num_frames is no longer an int/int array. So I amended ['length'] to ['length'][0] in line 34 as well.

[icedwater]

If you prefer BVH files, I recommend looking at joints2bvh.py from MoMask ...

I'm not sure how the joints2bvh.py can be used for the results.npy or the sample00_rep00_smpl_params.npy format, because the joints sequence isn't obviously represented in either of those outputs. Do you have an idea of what needs to be extracted or converted from there?

[JasonNero]

Heyhey,

you can just load the results.npy and pass the joint positions from results["motion"] to the converter.convert(...) function. Both projects use the SMPL skeleton so they have the same joint order.

I might have slightly different shapes than you, but I hope this helps you anyways:

path = Path("./path/to/your/results.npy")
results = np.load(path, allow_pickle=True).item()
motion = results["motion"]

converter = Joint2BVHConvertor()

for i_rep in range(results["num_repetitions"]):
    for i_sample in range(results["num_samples"]):
        joints = motion[i_rep, i_sample].transpose(2, 0, 1)  # (J, 3, F) -> (F, J, 3)
        file_name = f"sample{i_sample:02d}_rep{i_rep:02d}"
        bvh_path = path.parent / (file_name + ".bvh")
        converter.convert(joints, bvh_path, foot_ik=False)

[icedwater]

Thanks! I might have taken much longer to figure out the transposition of the motion on my own. I was finally able to export a usable BVH. I've cleaned up the main() bit of joints2bvh.py in my fork of momask, and convert_one_result() works fine for me.

[kbrodt]

Hi @JasonNero, I see that you have forked this repo and I was wondering if you support custom inputs? (ps. your repo doesn't allow to create issues)

[icedwater]

Hi @kbrodt what specifically do you mean by custom inputs? I'm trying to figure out how to train on new rigs which are not AMASS-based (e.g., 30-joint skeletons, 50-joint, etc).

[kbrodt]

I have 2 (or more) poses in SMPL format (21 joint rotations + 1 global orientation) and I want to inbetween.