Frankmocap and VIBE

[zeegan]

Hello Carlos,

Thanks for the detailed info on each method! I ran Frankmocap but have no way to convert the output file(s) to fbx or bvh. Is that part completed? Seeing the VIBE method DID have fbx import to blender i ran this also but found that the hands did not capture. Did you find that or did i do something wrong on the VIBE installation? Hands captured fine on FRANK but not on VIBE

[carlosedubarreto]

Both methods vibe and frankmocap lack fingers. On their own scripts and also mocao import.

Vibe does not have fingers yet, but frankmocap could import then. I just have to have more time available.

Today the only solution that works with fingers is easymocap.

And seems that the new version of easymocap is not working with mocao import.

But probably fix the easymocap will be much easier than making the frankmocap to work.

If you want fingers mocao, i would suggest you to install easymocap. I think it's the best one (that's why I gave more attention to it)

[zeegan]

Great, thank you! I saw the tut but wasn't sure it was the one you work on most. I will give it a try. Thank you!

[carlosedubarreto]

The installation of easymocap is really easy, but the calibration is a bit hard.

But i can help you on that.

But it will take some days to finish, i guess.

But what do you have to loose?

☺️

There are some tutorials I've done in the same YouTube channel that there is the vibe tutorial.

[zeegan]

Thanks. I am at the end of tut 1 for easy mocap and the very last command is not completing and giving this error

RuntimeError: The size of tensor a (42) must match the size of tensor b (0) at non-singleton dimension 1

[zeegan]

this is the full error before the RENDER should start...

Traceback (most recent call last): File "apps/demo/mv1p.py", line 109, in mv1pmf_smpl(dataset, args) File "apps/demo/mv1p.py", line 71, in mv1pmf_smpl weight_shape=weight_shape, weight_pose=weight_pose) File "c:\mocap\easymocap\easymocap\pipeline\basic.py", line 77, in smpl_from_keypoints3d2d params = multi_stage_optimize(body_model, params, kp3ds, kp2ds, bboxes, Pall, weight_pose, cfg) File "c:\mocap\easymocap\easymocap\pipeline\basic.py", line 29, in multi_stage_optimize params = optimizePose3D(body_model, params, kp3ds, weight=weight, cfg=cfg) File "c:\mocap\easymocap\easymocap\pyfitting\optimize_simple.py", line 301, in optimizePose3D params = _optimizeSMPL(body_model, params, prepare_funcs, postprocess_funcs, loss_funcs, weight_loss=weight, cfg=cfg) File "c:\mocap\easymocap\easymocap\pyfitting\optimize_simple.py", line 246, in _optimizeSMPL final_loss = fitting.run_fitting(optimizer, closure, opt_params) File "c:\mocap\easymocap\easymocap\pyfitting\optimize.py", line 38, in run_fitting loss = optimizer.step(closure) File "c:\mocap\easymocap\easymocap\pyfitting\lbfgs.py", line 307, in step orig_loss = closure() File "c:\mocap\easymocap\easymocap\pyfitting\optimize_simple.py", line 231, in closure loss_dict = {key:func(kpts_est=kpts_est, new_params) for key, func in loss_funcs.items()} File "c:\mocap\easymocap\easymocap\pyfitting\optimize_simple.py", line 231, in loss_dict = {key:func(kpts_est=kpts_est, new_params) for key, func in loss_funcs.items()} File "c:\mocap\easymocap\easymocap\pyfitting\lossfactory.py", line 63, in hand diff_square = (kpts_est[:, 25:25+42, :3] - self.keypoints3d[:, 25:25+42, :3])*self.conf[:, 25:25+42] RuntimeError: The size of tensor a (42) must match the size of tensor b (0) at non-singleton dimension 1

[carlosedubarreto]

Please erase the output folder the script created and run the command again

[zeegan]

HAHA YES sorry....it was very late and before i saw your reply i realized i was running to different results into the same folder. Thanks! The things a few hours sleep and a coffee can do!! :D OK...on to part two! have a good day bro!

[zeegan]

So...i ran it and worked fine. Got both the blender addon..(already had it from VIBE TUT)....browsed the basicModel_m_lbs_10_207_0_v1.0.2 and then the JSON files...waited. Then the armature and male model appears with no animation data at all.

console gave this error:

Python: Traceback (most recent call last): File "C:\Users\zeegan\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 484, in execute data.append(json.load(f)) File "C:\Program Files\Blender Foundation\Blender 2.93\2.93\python\lib\json__init__.py", line 293, in load return loads(fp.read(), File "C:\Program Files\Blender Foundation\Blender 2.93\2.93\python\lib\codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

location: :-1

[zeegan]

so im running the DEMO sample from scratch and will try again.

[zeegan]

Ok forget that error....that was something else. But after retry same result. I get the armature and model but no animation data in the timeline.

[zeegan]

on choosing the folder with the JSON files i got this error;

Python: Traceback (most recent call last): File "C:\Users\zeegan\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 367, in execute motions = load_smpl_params(path_smpl) File "C:\Users\zeegan\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 274, in load_smpl_params motions = load_motions(datapath) File "C:\Users\zeegan\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 259, in load_motions infos = read_smpl(filename) File "C:\Users\zeegan\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 231, in read_smpl data_pkl = read_pkl(filename+'.pkl') File "C:\Users\zeegan\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 223, in read_pkl with open(path_pkl, 'rb') as pk: FileNotFoundError: [Errno 2] No such file or directory: 'C:\MOCAP\EASYMOCAP\1_OUTPUT\SMPL_SAMPLE\smpl\000000.pkl'

location: :-1

then i selected everyone with select all and the error never happened again BUT still not animation data. Sorry to bombard you like this.

[carlosedubarreto]

did you update mocap_import to the version 0.722?

it looks like an error from 0.721

[carlosedubarreto]

and I saw your folder.... it is the worng one. you should load the folder SMPL_FULL, that is created using the latest version of easymocap (dowload if again and overwrite the one you have unzipped)

and add the command --write_smpl_full

[zeegan]

yes i noticed the addon and swapped it for the latest.

But i don't understand what you mean SAML_FULL. i downloaded from the links in the 'anotacao_para_tutorial' doc

https://github.com/zju3dv/EasyMocap

https://github.com/carlosedubarreto/easymocap_win_install

[carlosedubarreto]

To make it work you have to update the addon, and easymocap (that I understtod you already did.

Besides that you must add --write_smpl_fullon the code to run the process on easymocap.

For example, previously, you might run this comando to process the sample data python apps/demo/mv1p.py 0_input/sample --out 1_output/sample/smplx2 --undis --sub_vis 1 7 13 19 --body bodyhandface --model smplx --gender male

To create the correct data to import in blender, with the body and hands data, you should run the command bellow

python apps/demo/mv1p.py 0_input/sample --out 1_output/sample/smplx2 --undis --sub_vis 1 7 13 19 --body bodyhandface --model smplx --gender male --write_smpl_full

Notice the --write_smpl_full at the end, that´s what you have to add.

[zeegan]

Great. i will run it with the new files and updated command. thanks

[carlosedubarreto]

BTW if you want to run faster, remove the VIS options

usuallty I rund this command python apps/demo/mv1p.py 0_input/sample --out 1_output/sample/smplx --vis_det --vis_repro --undis --sub_vis 1 7 13 19 --body bodyhandface --model smplx --gender male --vis_smpl --write_smpl_full

But it creates a lot of images and takes too much more to render. If the processing is allright, you might not want those rendered images. do process without the images, you can remove

--vis_det --vis_repro --vis_smpl

[zeegan]

It all seems to work fine but still nothing i the blender. I see the JSON files in SMPL_FULL and the maya rig uploads to blender no errors, only without the animation data.

[zeegan]

this is what i see in blender system console. Non of the frames in your video:-

FBX Import: start importing C:\MOCAP\smpl_for_maya\basicModel_m_lbs_10_207_0_v1.0.2.fbx FBX version: 7400 FBX import: Prepare... Done (0.000000 sec)

    FBX import: Templates...
            Done (0.000000 sec)

    FBX import: Nodes...
            Done (0.000000 sec)

    FBX import: Connections...
            Done (0.000000 sec)

    FBX import: Meshes...
            Done (0.281250 sec)

    FBX import: Materials & Textures...
            Done (0.000000 sec)

    FBX import: Cameras & Lamps...
            Done (0.000000 sec)

    FBX import: Objects & Armatures...
            Done (0.046875 sec)

    FBX import: ShapeKeys...
            Done (6.593750 sec)

    FBX import: Animations...
            Done (0.000000 sec)

    FBX import: Assign materials...
            Done (0.000000 sec)

    FBX import: Assign textures...
            Done (0.000000 sec)

    FBX import: Cycles z-offset workaround...
            Done (0.000000 sec)

    Done (7.265625 sec)

Import finished. success load file_json: [] file_pkl: []

[zeegan]

i noticed also that, the maya rig has no finger bones. How can that work without them?

[zeegan]

Looking around the smpl info i see that the two different types of models ...one being 10 shape and the other being the 300 shape...or SMPL_FULL. But the addons for maya/blender and unity only provide rigs with 10 shapes. Not the full 300 used with the smpl~_full. I obviously have no real idea if it means anything?

Im doing a full uninstall of everything and reinstalling the entire thing to see if i get any different results.

[carlosedubarreto]

You have to download the blender addon from SMPLX to generate your FBX to use on the addon. I dont remember where I wrote the explanation.... I'll see if I can find it.

probably is in this home page https://smpl-x.is.tue.mpg.de/

I forgot my password, but on the download page you got to the end of the screen and you must find the blender addon to download.

After download, install it in your blender, chose a male, female or neutral character. and export it as FBX.

Then you can use that fbx file to load the animation using mocap_import addon

[zeegan]

Yes i tried this. I saw it in another post somewhere. Made another FBX with that smpl addon...imported with the updated mocap addon ...same results. Get the armature without anything in the timeline.

[carlosedubarreto]

can you record a video showing how you are doing this? Or can you send me the files, so I can test?

I would need only the SMPL_FULL folder with the json files.

[zeegan]

sure i can send the jsons smpl_full.zip

[carlosedubarreto]

I ran here, look how it went. 🤔 There is something odd ... https://user-images.githubusercontent.com/4061130/122600173-c74d8800-d045-11eb-9b7b-00644623df05.mp4

[zeegan]

Did it work?... that looks different from what I get. I get non of the shape frames loading.... think it is my installation somewhere is wrong? Surely I would receive errors along the way?

[carlosedubarreto]

wit a minute.... I think there is a problem on the fbx you are using....

look mine at the left and yours, at the right

these should be the names of mesh/armature

[zeegan]

No thats old...from the TUT. this is the one i have now..

[zeegan]

Ok thanks. i have them. Used the MALE. looks totally different to those before...even that i downloaded from the website or generated from the blender addon...

[carlosedubarreto]

its working now?

[zeegan]

No. Should i just start from scratch and reinstall everything?

[carlosedubarreto]

hum.... do you have anydesk? do you want me to try to help you remotely we could do that as soon as I finish my work day (probably in about 1 hour

[carlosedubarreto]

No. Should i just start from scratch and reinstall everything?

I dont think that is the problem.

[zeegan]

new error at the end

   FBX import: Prepare...
            Done (0.000000 sec)

    FBX import: Templates...
            Done (0.000000 sec)

    FBX import: Nodes...
            Done (0.015625 sec)

    FBX import: Connections...
            Done (0.000000 sec)

    FBX import: Meshes...
            Done (0.515625 sec)

    FBX import: Materials & Textures...
            Done (0.000000 sec)

    FBX import: Cameras & Lamps...
            Done (0.000000 sec)

    FBX import: Objects & Armatures...
            Done (0.093750 sec)

    FBX import: ShapeKeys...
            Done (11.187500 sec)

    FBX import: Animations...
            Done (0.000000 sec)

    FBX import: Assign materials...
            Done (0.000000 sec)

    FBX import: Assign textures...
            Done (0.000000 sec)

    FBX import: Cycles z-offset workaround...
            Done (0.000000 sec)

    Done (12.421875 sec)

Import finished. success load Traceback (most recent call last): File "C:\Users\zeegan\AppData\Roaming\Blender Foundation\Blender\2.90\scripts\addons\b3d_mocap_import-main\load_mocap.py", line 344, in execute bpy.data.shape_keys["Key"].key_blocks[k].slider_min = -10 KeyError: 'bpy_prop_collection[key]: key "Exp000" not found'

location: :-1

location: :-1

[carlosedubarreto]

Hum. blender 2.90.... can you download 2.92 to test?

Download the zip one, so you can have multiple instalations without causing problems to one another.

[carlosedubarreto]

but from the error, looks more like a problem with the fbx... That error is from the part that I copied from easymocap script... I dont understand it fully.

[zeegan]

Ok. So we have something. Looks like its only working with 2.92. i have 2.78....2.8..i use them for different features and i downloaded 2.92 seeing it was the one you used and having version problems in the past...but i ran the old files with the old addon with it....

[zeegan]

..so its loading shape frames now...works perfectly. Very sorry for all the confusion. A least we know now! XD

[carlosedubarreto]

Thanks a lot for the testing... I didnt know there was a problem with 2.90.

[zeegan]

No idea. The smpl site says blender 2.82 onwards and the vibe import worked on older blenders.

[zeegan]

and thanks for the help Carlos! Great work on everything and no doubt you will hear from me during the second tut! XD

[carlosedubarreto]

and thanks for the help Carlos! Great work on everything and no doubt you will hear from me during the second tut! XD

I'm sure too LOL, its the worst part!!!! 🤣😅