[models] Support animated models

[haudan]

Proposal for animated models

I have been using raylib for a little toy project and I love it, it works really well. There's one thing I miss though, and that's support for animated models. Loading and parsing 3d animated models can be a lot of work, so I thought that integrating Assimp into raylib wouldn't be a bad idea. That way we would also gain support for a parsing a million different file formats. Assimp is written in C++ but comes with a clean C api. I'd be happy to help with implementing this.

At this point, I also want to mention something else: Is it really such a good idea that we pass structs by value to functions like DrawModelEx? Sure it's easier for beginner programmers to understand, but I think it hurts the library more than it helps. It's especially confusing for seasoned C programmers, wo generally considered this a bad practice.

[raysan5]

Hi @Lisoph! Nice project!

I know 3d animated models support is one of the weakness in raylib but no plans to integrate Assimp.

Assimp is a HUGE library with lots of dependencies and one of the goals of raylib is just removing external dependencies, all required libraries (mostly single-file header-only) are integrated with base code.

I'm looking for some simple solution, probably in the line of IQM models animation support, I know @culacant is working on it (single-file header only?). In the meantime my recomendation is just using Assimp or any other 3d models lib at user side, not directly integrated in raylib.

About DrawModelEx(), you're right, Model struct has grown a lot since first version. Changing it now requires some internal redesign... I'm evaluating the benefits of that change because, despite I agree with the theory (value vs reference), I'd like to know what's the impact of that change in performance terms... I bet is lower than expected...

[a3f]

Sure it's easier for beginner programmers to understand

It might be possible to maintain API compatibility while still passing by reference if we typedef an array, e.g. for Model:

typedef struct Model {
    Mesh mesh;              // Vertex data buffers (RAM and VRAM)
    Matrix transform;       // Local transform matrix
    Material material;      // Shader and textures data
} Model[1];

This could be an option as well…

[raysan5]

@procedural did a quick profiling sample comparing pass by value vs pass by pointer. It seems there is not much difference between both... actually pass by value works unexpectly faster.

[haudan]

I don't think that profling sample is a very good metric. It's trivial for the compiler to optimize that function, no matter how the data is passed. To get a true idea of performance we would have to simply try it out on raylib, ideally in prebuilt library form.

Still, I played around with that profiling sample on godbolt and found that pass-by-value forces the compiler to generate a memcpy call, to copy the data onto the function's stack (even on -O3), while for pass-by-reference no such call is generated. Tested with gcc 8.1 and I marked procedure as __attribute__ ((noinline)).

Here's my sample code:

Expand

```c enum {BYTES = 10000}; struct TYPE { unsigned char bytes[BYTES]; }; extern struct TYPE output; #define BY_VAL 1 #if BY_VAL == 1 void __attribute__ ((noinline)) procedure(struct TYPE input, struct TYPE * output) { for (int i = 0; i < BYTES; i += 1) { output->bytes[i] = input.bytes[i] + input.bytes[i]; } } #else void __attribute__ ((noinline)) procedure(struct TYPE const *input, struct TYPE *output) { for (int i = 0; i < BYTES; i += 1) { output->bytes[i] = input->bytes[i] + input->bytes[i]; } } #endif int main() { for (int i = 0; i < 10000; i += 1) { struct TYPE input; // commented out for less assembly // for (int j = 0; j < BYTES; j += 1) { // input.bytes[j] = rand() % 255; // } #if BY_VAL == 1 procedure(input, &output); #else procedure(&input, &output); #endif } } ```

[ghost]

@Lisoph

I don't think that profling sample is a very good metric.

Is it? :) Because your code shows the same results, around 41 microseconds for -O0 (Clang 6.0, weak ass 2 core 2.2 GHz AMD processor), 9 microseconds for -O1 and less than or equal to 1 microsecond for -O2. For #define BY_VAL 0 results are similar but slightly slower at -O0.

Can I merge your changes under unlicense (Public Domain) license so you could test too?

I added Visual Studio 2017 project file if someone wants to try it on Windows.

[haudan]

@procedural sure, feel free to use the code. I'll check it out at home, thanks for the VS2017 project files!

[haudan]

I don't know how to use Chrome's Tracing, but I ran the profiling program myself (VS2017) and the pass-by-ptr is faster for me in both Debug and Release builds.

I create another example on Godbolt which does something similar to our benchmark and the pass-by-ptr wins here as well. The assembly for by_ptr and by_val is nearly identitcal (only some reordering), but the invocation is quite different. by_ptr just pushes the address onto the stack, while by_val pushes every member of the struct. Both functions operate through 1 pointer indirection. Unless the reordering of the generated simd assembly instructions is that vital, I don't think that by_val will be faster than by_ptr here.

Anyway, I think this pass by pointer debate deserves it's own issue.

[raysan5]

Hi @Lisoph, actually, we were discussing this issue further with @procedural on his C programming Discord channel. There was some measure precission issues and effectively pass-by-ptr is faster than pass-by-val. But difference was not that big, if I remember correctly it was around 50ns for a 10Kb structure.

Anyway, I think this pass by pointer debate deserves it's own issue.

Agree, just opened it: https://github.com/raysan5/raylib/issues/563

PD. About the animated models issue, I'm working now through @culacant IQM implementation to try to integrate it into raylib in some way...

[SolarLune]

Yo! I'm interested in doing 3D with raylib as well!

(And great work on the framework, by the way! It's really comfortable and easy to use!)

I was just wondering about the status of this. I'm wondering why it might be a good idea to go with the IQM format instead of something more standard, like OBJ (assuming OBJ can support animations), DAE, or OGEX (for a hopefully more future-proofed standard driven by community needs)?

Also, I was wondering if you were dealing with possible issues regarding the design of an animated model struct and how that might work with raylib's current API. Seems like you'd be good to just stick a list of animations onto the Model struct itself, import animations when you import a Model from a Model File, and then transform the verts every frame before rendering if necessary (or have the developer call a function to do this), but I imagine I must not fully grasp all of the complexities.

[culacant]

I chose IQM and the ascii-counterpart IQE because it is basically OBJ with animation data added in. Keep in mind I had no idea what animations were when i made the loader, and the readability of the files was a big help in figuring out what actually needed to happen to make stuff move on screen.

Turns out there's not a whole lot of choices in formats if your main concern is ease of implementation. The doom 3 md5 format is easy enough, but it's getting pretty old and blender doesn't play nice with the exporters I found. FBX would be great to support because everyone's using it, but I couldn't find alot of documentation since it's supposed to be closed source. (Blender reverse engineered it tho, so it can't be that hard to figure it out) Alot of the newer would-be standards like OGEX, DAE and glTF are more focused on stuffing alot of useless specialized data in their formats like physics and lights, which is great if you plan to use them but it makes the format way more complicated than it needs to be if you just want animations.

The good part of all this is that once the structs for the animated meshes and animations are in place it wouldn't be that hard to write loaders for different formats, because from what I saw most formats basically store the same data, with a couple minor differences (like local rotations instead of global etc).

Ray was looking at ways to put my code in Raylib in a way that makes sense and doesn't break everything else.

[raysan5]

@culacant hopefully, this week I'll upload the reviewed version.

[SolarLune]

@culacant Ah, I see, I see. That might be a good middle-ground, then, especially if it's inherently simple to load and 3d modelers should have stable readily available exporters. Thanks for the explanation!

@raysan5 Great to hear! Hope to see animated models soon~

[raysan5]

Working sample added on commit https://github.com/raysan5/raylib/commit/198a02352764a78eb0fc3b94c3c8418d2d0c1432

API requires some changes and review...

[justinclift]

Came across glTF today.

It seems like a super-set of what's needed here, so maybe way too complicated?

[haudan]

Came across gITF today It seems like a super-set of what's needed here, so maybe way too complicated?

I don't know, since glTF is JSON based we could simple ignore the stuff we don't care about while parsing. It looks as though going with glTF would force us to write our own parser, because I couldn't find any C libraries to do that. While it helps that glTF is based on JSON, we would still need a JSON library.

glTF definitely is the most modern 3d model format that I know of. A lot of software already supports it out of the box.

[legends2k]

It looks as though going with glTF would force us to write our own parser, because I couldn't find any C libraries to do that. While it helps that glTF is based on JSON, we would still need a JSON library.

Yeah, lots of C++ libraries but no C ones except AssetKit which is under development. However, since it's JSON it should be fairly straight forward. I've seen the minimal-gltf-loader code. It creates buffers, loads data and makes links within structure; basically a deserializer, but nothing complicated.

glTF definitely is the most modern 3d model format that I know of. A lot of software already supports it out of the box.

Yeah, I came here looking for GLTF support for a personal C++ project. It's supported by many engines and companies already. Why we should all support glTF 2.0 as THE standard asset exchange format for game engines (an article by the Godot engine author) explains its design strengths and also does a compare-and-contrast against FBX, Collada, OBJ, 3DS and OpenGEX.

[raysan5]

Nice read. Undoubtely glTF is a great format to be supported by raylib...

[haudan]

Regarding a JSON parser: I came across jsmn. It's quite low level (basically just a tokenizer) and requires a lot of manual work, but in exchange it is extremely small, easy to integrate and even quite flexible and fast. It might be a good base should we decide to roll our own glTF parser.

[raysan5]

@Lisoph already started working on glTF v2.0 support! I'm using cgltf, looks amazing! Actually, it embeds JSMN in a single header file!

Implementation requires taking care of https://github.com/raysan5/raylib/issues/596 also. Need some time to figure out the best way to redesign this, it's quite a big change...

Unfortunately, animation is not supported by the loader yet...

[raysan5]

Quick update on animation support. I'm currently implementing multi-mesh and multi-material support and, as long as Model struct is been changed, I also added some data for animation. Here there are the proposed new structs:

// Model bone pose (transform)
typedef struct Pose {
    Vector3 translation;    // Translation 
    Quaternion rotation;    // Rotation
    Vector3 scale;          // Scale
} Pose;

// Model type
typedef struct Model {
    Matrix transform;       // Local transform matrix

    int meshCount;          // Number of meshes
    Mesh *meshes;           // Meshes array

    int materialCount;      // Number of materials
    Material *materials;    // Materials array
    int *meshMaterial;      // Mesh material number

    // Animation data
    bool animated;          // Model is animated
    int boneCount;          // Number of bones
    Pose *basePose;         // Pose by bone
} Model;

// Model animation
typedef struct Animation {
    int boneCount;          // Number of bones (joints)
    int frameCount;         // Number of animation frames
    float frameRate;        // Frame change speed
    Pose **framePoses;      // Poses array by frame and bone
} Animation;

I reviewed the data needed by IQM but I was wondering if I'm missing some important piece of data to be added to model to properly support animations. Feedback, please!

[raysan5]

After some reading and a more careful analysis, here it is a new structs proposal for dealing with animation, probably some element still missing (maybe struct Bone?) but it seems pretty clear to understand:

// Transformation properties
typedef struct Transform {
    Vector3 translation;    // Translation 
    Quaternion rotation;    // Rotation
    Vector3 scale;          // Scale
} Transform;

// Animation key frame
typedef struct KeyFrame {
    int boneCount;          // Number of bones (joints)
    Transform *poses;       // Bones transformations
    //float timeStamp;        // Keyframe time
} KeyFrame;

// Model type
typedef struct Model {
    Matrix transform;       // Local transform matrix

    int meshCount;          // Number of meshes
    Mesh *meshes;           // Meshes array

    int materialCount;      // Number of materials
    Material *materials;    // Materials array
    int *meshMaterial;      // Mesh material number

    // Animation data
    bool animated;          // Model is animated
    KeyFrame basePose;      // Model base animation pose
} Model;

// Model animation
typedef struct Animation {
    int frameCount;         // Number of animation frames
    KeyFrame *frames;       // Key frames (bones poses)
    //float frameRate;        // Frame change speed
}

Note that Mesh struct also contains animation data, specifically, bones indices and weight for every vertex. IQM animation also requires storing the base position and normal data to be used on pose transform.

typedef struct Mesh {
    ...
    // Animation vertex data
    float *baseVertices;    // Vertex base position (required to apply bones transformations)
    float *baseNormals;     // Vertex base normals (required to apply bones transformations)
    float *boneWeights;     // Vertex bone weight, up to 4 bones influence by vertex
    int *boneIds;           // Vertex bone ids, up to 4 bones influence by vertex
} Mesh;

I really care for structs naming, I want to be be clear and comprehensive, I had doubts between KeyFrame or AnimFrame but I feel KeyFrame is clearer, after all, one key-pose could be independent of animation.

Any piece missing? Please, feedback!

[raysan5]

Big update in commit https://github.com/raysan5/raylib/commit/d89d24c5e8930c18a93a6403e26914a9e2b23b84!

Finally, the implementation selected has been:

// Transformation properties
typedef struct Transform {
    Vector3 translation;    // Translation
    Quaternion rotation;    // Rotation
    Vector3 scale;          // Scale
} Transform;

// Bone information
typedef struct BoneInfo {
    char name[32];          // Bone name
    int parent;             // Bone parent
} BoneInfo;

// Model type
typedef struct Model {
    Matrix transform;       // Local transform matrix

    int meshCount;          // Number of meshes
    Mesh *meshes;           // Meshes array

    int materialCount;      // Number of materials
    Material *materials;    // Materials array
    int *meshMaterial;      // Mesh material number

    // Animation data
    int boneCount;          // Number of bones
    BoneInfo *bones;        // Bones information (skeleton)
    Transform *bindPose;    // Bones base transformation (pose)
} Model;

// Model animation
typedef struct ModelAnimation {
    int boneCount;          // Number of bones
    BoneInfo *bones;        // Bones information (skeleton)

    int frameCount;         // Number of animation frames
    Transform **framePoses; // Poses array by frame
} ModelAnimation;

Still some doubts about having BoneInfo data duplicated in Model and ModelAnimation... but it definitely makes sense and, actually, it can be used to verify if an animation fits in a model.

Some work left on animation functions implementation.

[justinclift]

Sounds good @raysan5. :smile:

[raysan5]

Multiple functions added and some reviewed in commit https://github.com/raysan5/raylib/commit/92733d6695e0cdab3b42972f2cd6ed48d98ec689.

Despite a lot of work is still required (I'll list everything in future issues), I consider this feature implemented. raylib officially supports animated models.