yanqinJiang / Animate3D

[NeurIPS 2024] Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
https://animate3d.github.io/
Apache License 2.0
132 stars 4 forks source link

Export to Game Engine #5

Open sekkit opened 2 weeks ago

sekkit commented 2 weeks ago

Is it possible?

yanqinJiang commented 2 weeks ago

Yes, it is possible when you choose the mesh animation in our repo. Currently we export mesh animation as .fbx file. But .fbx format animation files may encounter some compatibility issues when imported into game engines, because our method of animating the mesh is deformation using shape keys not skeleton.

This animation result is actually more suitable for exported as .abc files or VAT (Vertex Animation Texture). Both of these formats can be used in game engines, such as UE5. However, .abc files exported from Blender do not include textures, which may require manual texture binding, while exporting VAT files is a bit more complicated. If you need, I can look into whether there are any automated workflows to address these issues in my spare time.

BTW, please tell me which game engine you would like to use if you need me to look into some export details.

sekkit commented 2 weeks ago

Unity first, if skeleton that could be perfect! thanks for your help

yanqinJiang commented 1 week ago

I'll have a look at Unity, but our mesh animation is not skeleton animation, it is more like shape deformation. Hope that is okay for you.

sekkit commented 1 week ago

nice if u make it happen

yanqinJiang commented 1 week ago

Good to find that FBX animation files can be directly imported into Unity. However, since we're using embedded textures, you'll need to extract the textures manually, following steps bellow:

  1. Select the imported FBX file in Unity.
  2. In the Inspector window, find the Materials tab.
  3. Click the "Extract Textures" button. Note that sometimes normal map images cannot be correctly recognized, so you may need to manually assign it.

I also write a simple c# script to help you automatically extract the texture and recognize normal map, your normal map should use '_noraml' as suffix in this case. You can name the file as TexturePostprocessor.cs and put it in your project folder. (It may take a few seconds to extract the textures in this way)

using UnityEngine;
using UnityEditor;
using System.IO;

public class TexturePostprocessor : AssetPostprocessor
{
    void OnPreprocessTexture()
    {
        TextureImporter textureImporter = assetImporter as TextureImporter;
        if (textureImporter != null)
        {
            // Automatically set normal maps
            if (assetPath.ToLower().Contains("_normal"))
            {
                textureImporter.textureType = TextureImporterType.NormalMap;
            }
            // You can add automatic recognition for other texture types, e.g.:
            // else if (assetPath.ToLower().Contains("_metallic"))
            // {
            //     textureImporter.textureType = TextureImporterType.Default;
            //     textureImporter.sRGBTexture = false;
            // }
        }
    }
}

public class FBXPostprocessor : AssetPostprocessor
{
    void OnPostprocessModel(GameObject go)
    {
        if (assetPath.ToLower().EndsWith(".fbx"))
        {
            ExtractTexturesFromMaterials(go);
        }
    }

    void ExtractTexturesFromMaterials(GameObject go)
    {
        ModelImporter modelImporter = assetImporter as ModelImporter;
        if (modelImporter != null)
        {
            modelImporter.ExtractTextures(Path.GetDirectoryName(assetPath));
        }
    }
}

As for animation clip, You'll need to manually bind the animations from the FBX file to the objects. I guess you are more familiar with this process than I am, as I haven't used Unity before.

Please give it a try🚀 and if you encounter any issues, feel free to post them here~

sekkit commented 1 week ago

brilliant,thx. i'll use the code