Multiple model modelling

[elizabethjcobs]

https://user-images.githubusercontent.com/92538100/208305738-76d82123-3285-4fe5-866f-8ada56e49de7.mp4

Hi there,

I was wondering if it was possible to extract the .pkl/.fbx data for more than one model in a visual scene. The video output I've been getting has been for 2 smplx models, but when I import the file into blender/unity, it only displays one and I am unable to toggle or get the data for the second model within the scene. Any advice would be appreciated, thanks.

[carlosedubarreto]

Using vibe, you can import the person using mocap import, in this part mark in red you select which person you want to load in blender

[elizabethjcobs]

Despite toggling with the Person ID as indicated, when I try to import through VIBE, I get the following error:

Python: Traceback (most recent call last):
  File "C:\Users\eliza\AppData\Roaming\Blender Foundation\Blender\3.1\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 2171, in execute
    create_bones_wo_face(bones)
  File "C:\Users\eliza\AppData\Roaming\Blender Foundation\Blender\3.1\scripts\addons\Mocap_PE_data_import\load_mocap.py", line 1896, in create_bones_wo_face
    obs[len(obs)-1].data.edit_bones['Bone'].select_tail=True
KeyError: 'bpy_prop_collection[key]: key "Bone" not found'

location: <unknown location>:-1

Which is why I've been changing .pkl to .fbx - do you have any advice for this?

[carlosedubarreto]

@elizabethjcobs Seems like you are using with blender 3.1 I'm almost sure it wont work with blender 3.x

try with Blender 2.9.3

[elizabethjcobs]

I installed Blender 2.9.3 as advised and managed to get it to distinguish between the person IDs, however, the smpl model has been replaced with (no human model) armature , is there any way to rectify this?

[carlosedubarreto]

Probably you wont be able to change it.

By the time I was making this addon, I did some alternative ways to import the data. Some using the script from the creator, but packed to use inside blender, and some other I did myself. Probably this one that shows only the skeleton is one that I did (I actuyally dont remember)

But if you have the SMPL model you can retarget it from the skeleton, using rokoko addon for example.

But in the end, why make the animation on the smpl model? Usually you would get the animation to put in another character, so its kind of unecessery to put the animation on the SMPL model, IMHO.

[elizabethjcobs]

I've been trying to model interpersonal conflict/intimidation scenes for my PhD project, so far this seems to be the only way I can model both characters within 1 scene. Ideally, I'd like to have the armature attached to smplx models for when I implement it into a VR enviroment to record a third person's responses to the ongoing scene. To my knowledge, VIBE doesn't work with smplx. I've tried smplify-x and openpose prior to this, VIBE seems to be the best out of the softwares I've tried so far.

[carlosedubarreto]

Hum interesting... so let me suggest you another option that could be the best for you. HAve tried Easymocap Monocular?

I made a version that is integrated in blender, it has a few bugs, like, I thin k it cant get multiple person in the same video (but you can maks out one of the people, run it, then mask the other person, for example) and the other problem is that if the script cant recognie a person in a frame, it throws an error (i couldnt taclkle this error yet) but it can also be done by making the process in parts, like, run until the error shows, skip the frame with the problem, and continue on the rest of the footage.

Seems complicated but its not that much.

Please take a lot at the CEB Easymocap monocular addon. https://carlosedubarreto.gumroad.com/l/ceb_em_monocular

I hope it makes things easier to you. it uses SMPL.

[carlosedubarreto]

btw, its free, just use the coupon on the description

[elizabethjcobs]

Thank you for the suggestion :) I have given it a shot prior, but I noticed the motion capture from it seemed less accurate than VIBE's (especially limbs when the model isn't front facing). I will give it a shot again with more videos in the meantime

[carlosedubarreto]

Oh yeah, the easymocap monocular, gives a best result if you keep the same angle from the recording.

Another option, and im my opinion the best one, is the esymocap multicam, but you need at least 2 cameras, and to do the camera calibration, but it is by far the best result you can get for free.

It even uses SMPLX (yes you can have finger tracking too) in case you want to try, here is the link

https://carlosedubarreto.gumroad.com/l/ceb_easymocap_multicam

[elizabethjcobs]

When you say two cameras, do you mean that if I'm using movie scenes I'd need the same scene but with two separate camera angles?

Thank you for this link! I will check it out :)

[carlosedubarreto]

some results with easymocap (the multicam one) https://www.youtube.com/watch?v=8CQP5xb-85c https://www.youtube.com/watch?v=MIOtrw67XLo https://www.youtube.com/watch?v=bnVD92pFhj8

[carlosedubarreto]

When you say two cameras, do you mean that if I'm using movie scenes I'd need the same scene but with two separate camera angles?

Thank you for this link! I will check it out :)

Not exactly that, you would need to record the video yoursefl with multiple cameras, becasue you will need to calibrate them.

If you are using footage from movies, the multicam wont work for you.

[elizabethjcobs]

I am currently using footage from movies, but am not against recording my own action scenes if I can't find a software compatible with the scenes I've been trying to use. Thank you again for these examples and links.

[carlosedubarreto]

if you need help with it, let me know.

[carlosedubarreto]

I was thinkiung again and came to a conclusion, if that method that show only the skeleton seems good to you, you just needed to put in a smpl model, I have an idea.

If you can send me the vibe result files, I coudl try to make something to show you.

that means to retarget the animation to another character for you to use.

Probably you wont need anything else besides what you already got.

And you will have more options, because you can get the animation in any rigged character you want. And there is a couple free ones from blender studio tht could be usefull to you.

take a look at this link https://studio.blender.org/characters/

there are some free character like Snow, Rain and Vincent.

dont you think your work would look better using characters like this? Just a suggestoin

[elizabethjcobs]

Thank you so much for all these suggestions you've honestly been so helpful and I couldn't be more thankful for your expertise! Initially i wanted to do it with a smpl/x model just to eliminate gender/race biases but it can definitely be an alternative. Would you like the .pkl files or .fbx files from VIBE?

[carlosedubarreto]

it could be the pkl. I'll try to use them, If I have problem i canlet you know.

Maybe it would be better to move this subject to another communication platform.

If you have, and like, we could talk on discord or twitter. at discord I'm Carlosedubarreto#1924 or you can get in my server about blender addons https://discord.gg/ajF8fKP3

or at twitter: https://twitter.com/carlosedubarret

[elizabethjcobs]

Sounds good! I will hop into your discord now :)