how you guys render dance animation?

[Leon0427]

numpy vector don't dance, right?

[liruilong940607]

There are 3rdparty tools you can try, for example this one:

https://github.com/softcat477/SMPL-to-FBX

We did rendering using the blender so we don't have a script to share

[Leon0427]

There are 3rdparty tools you can try, for example this one:

https://github.com/softcat477/SMPL-to-FBX

We did rendering using the blender so we don't have a script to share

thanks

[Leon0427]

There are 3rdparty tools you can try, for example this one:

https://github.com/softcat477/SMPL-to-FBX

We did rendering using the blender so we don't have a script to share

i did use this repo converting smpl to fbx ,and successfully played in unity. But the dancer still remain a raw figure without texure. I wander how much effort do have to pay to create dance with autodesk cartoon models like you did in your paper. Or should I just seek some 3d artist for help?

[Leon0427]

There are 3rdparty tools you can try, for example this one: https://github.com/softcat477/SMPL-to-FBX We did rendering using the blender so we don't have a script to share

i did use this repo converting smpl to fbx ,and successfully played in unity. But the dancer still remain a raw figure without texure. I wander how much effort do have to pay to create dance with autodesk cartoon models like you did in your paper. Or should I just seek some 3d artist for help?

Shit bro,I mean wonder

[liruilong940607]

My collaborator @shanyang-me did that cartoon for the paper, so I don't know the details. I only know she was using blender to render, and there are some manual efforts to match the skeleton of SMPL to the skeleton of the cartoon actor.

Also. The actor is downloaded from https://www.mixamo.com/#/. You might want to try first to upload your fbx to this website at here: https://www.mixamo.com/#/?page=1&type=Motion%2CMotionPack, and see if it can automatically help you to match the skeleton. If not, which I guess it would be the case, there would need some manual efforts like what my collaborator did.

[shanyangmie]

To render it,

• I used the Blender auto-rig-pro to rig the Mixamo character
• Convert the prediction results to bvh format, refer to our bvh converter: https://github.com/google-research/mint/blob/main/tools/bvh_writer.py to convert the npy file into a bvh file. bvh is a format for motion caption. You can import the bvh file into Blender.
• Once one has a rigged character and the bvh file for the animation, Blender will automatically create the animation

[Leon0427]

go on. "the prediction results" is a numpy vector, but the mapping between vector's row and blender model joints was not clear. I found a answer in https://files.is.tue.mpg.de/black/talks/SMPL-made-simple-FAQs.pdf, P23

[Leon0427]

succeed in animation with "https://www.mixamo.com/#/" model and https://github.com/softcat477/SMPL-to-FBX. and there are 4 key steps:

• create a smpl model in blender(with smpl blender add-on https://smpl.is.tue.mpg.de/download.php), take the bones(similar to armature) of smpl model as reference
• download mixamo model(fbx format) and import it into blender(a new project is prefered)
• modify the mixamo model's bones refer to smpl bones, and rename the bones accordingly. Pay attention to the direction and rotation of every bone. Leave the bones only in mixamo model as it is.
• export the modified mixamo model as fbx and use SMPL-to-FBX to migrate dance to it

[zhangsanfeng86]

Hi @shanyangmie

Do you know how to Convert 3D keypoint positions(which generate by SMPl model) to bvh format, or how to drive 3D character? Thank you!!

[vvright]

https://files.is.tue.mpg.de/black/talks/SMPL-made-simple-FAQs.pdf

succeed in animation with "https://www.mixamo.com/#/" model and https://github.com/softcat477/SMPL-to-FBX. and there are 4 key steps:

• create a smpl model in blender(with smpl blender add-on https://smpl.is.tue.mpg.de/download.php), take the bones(similar to armature) of smpl model as reference
• download mixamo model(fbx format) and import it into blender(a new project is prefered)
• modify the mixamo model's bones refer to smpl bones, and rename the bones accordingly. Pay attention to the direction and rotation of every bone. Leave the bones only in mixamo model as it is.
• export the modified mixamo model as fbx and use SMPL-to-FBX to migrate dance to it

seems you realize their work reversely By editing Mixamo models to ajust SMPL models which didn't publish in Github and really require quite a lot of patience Then use SMPL-to-FBX to migrate dance to the model but the compilation sucks on Mac OS with Conda envs In addition, we also have to use Autodesk FBX Conventer to convert ASCII FBX to Binary FBX so that Blender can load the final visulization. While Shan Yang convert SMPL prediction results to BVH file directly then use Blender to generate final 3D animation automatically

But admire your realization. Good work.

[miibotree]

SMPL-to-FBX project only support pkl format, to transfer the npy result to fbx, I modified SimpObject.py referenced by google-dancing:

def __init__(self, read_path):
    self.files = {}

    # For AIST naming convention
    #paths = PathFilter.filter(read_path, dance_genres=["gBR"],  dance_types=["sBM"], music_IDs=["0"])
    paths = PathFilter.filter(read_path, dance_genres=None,  dance_types=None, music_IDs=None)
    for path in paths:
        filename = path.split("/")[-1]

        # load npy file
        if filename.endswith(".npy"):
            with open(path, 'rb') as f:
                data = np.load(f)
                data = np.array(data)  # (N, 225)
                f.close()
            trans = data[:, 6:9]
            poses = data[:, 9:]
            poses = R.from_matrix(poses.reshape(-1, 3, 3)).as_rotvec().reshape(-1, 72)

            self.files[filename] = {"smpl_poses": poses,
                                    "smpl_trans": trans}

        # load pkl file
        else:
            with open(path, "rb") as fp:
                data = pickle.load(fp)
            self.files[filename] = {"smpl_poses":data["smpl_poses"],
                                    "smpl_trans":data["smpl_trans"]}

    self.keys = [key for key in self.files.keys()]

as well as PathFilter.py Line30:

#for path in sorted(glob.glob(osp.join(base_path, "*.pkl"))):
for path in sorted(glob.glob(osp.join(base_path, "*.npy"))):

now the --input_pkl_base can support npy result. I use maya to view the result, Blender seems not support ascii fbx. PS: I find the issue that can save the binary format fbx: https://github.com/softcat477/SMPL-to-FBX/issues/1 For example, my FbxCommon.py are copied and located at: ~/opt/anaconda3/envs/my_env/lib/python3.8/site-packages/FbxCommon.py and change Line29 to: if "binary" in lDesc: