export animatable glTF

[LyndzP]

Hello,

Thank you so much for your wonderful work on this project! I've been trying to convert the deformable banmo output into glTF format.

After discovering issue #46, I've been learning about the structure of glTF and skinning weights, but I still find myself at a loss when it comes to exporting to glTF.

Could you perhaps share some insights on how to export to an animatable glTF? Which libraries did you use to accomplish this? I've been experimenting with Blender's Python API, but I am still making static mesh.

Additionally, do you happen to have an ETA for when the code for exporting to glTF might be shared?

Thank you very much for your time and help. I greatly appreciate it!

[gengshan-y]

Hi, I wrote some utility functions and scripts to do the conversion. Hope it can help a bit, although it is likely those would not work out of the box for this repo.

The plan is to integrate the gltf conversion to our new repo lab4d in a few weeks after I finish another code release task.

[LyndzP]

@gengshan-y Thank you so much for your detailed explanations and guidance. They have helped me immensely in successfully writing code to apply skinning weights to glTF. I'm truly grateful for your help.

However, I'm currently facing a problem. I am trying to understand how to obtain a "urdf object of the ragdoll." From my understanding, it seems like there is a certain sequence or hierarchy involved in the relationship between the bones, likely pertaining to the parent node. However, I'm still not entirely sure about this.

The related code is as follows:

    if urdf is not None:
        gltf.nodes[1].children = []
        bone_rtks_local = []
        # go through all joints and find parents
        for idx, parent_idx in enumerate(urdf.parent_idx):
            # forward kinematics
            # G(xo-xt) = [G(xo-xt''')] [G(xo-xt''')^-1 ... G(xo-xt')] [G(xo-xt')^-1 G(xo-xt)]
            # update topology
            gltf.nodes[parent_idx+2].children += [idx+2]
            if parent_idx>=0:
                # update rest pose
                global_ga = np.eye(4); global_ga[:3,:3] = rest_rot[parent_idx]; global_ga[:3,3] = rest_trn[parent_idx]
                global_gb = np.eye(4); global_gb[:3,:3] = rest_rot[idx];        global_gb[:3,3] = rest_trn[idx]
                local_g = np.linalg.inv(global_ga)@global_gb
                gltf.nodes[idx+2].rotation = R.from_matrix(local_g[:3,:3]).as_quat().tolist()
                gltf.nodes[idx+2].translation = local_g[:3,3].tolist()
                # update deformed pose
                bone_rtk_local = np.linalg.inv(bone_rtks[:,parent_idx]) @ bone_rtks[:,idx]
            else:
                bone_rtk_local = bone_rtks[:,idx]
            bone_rtks_local.append( bone_rtk_local )
        bone_rtks = np.stack(bone_rtks_local,1)

Could you possibly provide some insight into this issue? I deeply appreciate your time and assistance

[gengshan-y]

Sorry for the very late reply, in case it still helps, the urdf object corresponds to robot.urdf in this line.

parent_idx is a list of indices such as [-1, 0, 1, 2, 3] (represents a kinematic chain; for quick reference, the quadruped skeleton is exported to lab4d at here). Each element points to the parent node, as computed here.

The template-simp.gltf can be found here

[LyndzP]

@gengshan-y I've finally succeeded in creating a moving glTF. Thank you so much for your help!