"Modern A3D" file format and support

[Pyogenics]

After flash EOL in 2020 Alternativa rewrote their engine and main product, Tanki Online, in kotlin to deploy for mobile and HTML5. Recently I discovered that Alternativa has further extended the A3D model format past original A3D1 and A3D2 formats and is actively using it in Tanki Online. It only makes sense that this tool should also support the modern A3D formats, I will document my research and findings todo with this format here and in future may open a PR to follow up and add support for this newer format to my other PR.

[Pyogenics]

There appear to be two variants of this format, which I assume are different versions: one is seemingly used for map/prop geometry and is marked with (00 02 00 00) after the file signature ("A3D", 41 33 44) and the other is used in the tank models and is marked with (00 03 00 00) instead.

Version 2

All files I have inspected so far only contain a single object without any submeshes, instead the models are split into multiple files with an object each (unlike version 3) (this is also reflected in their naming scheme, for example: vihou_3_obj0 is split into 6 files named vihou_3_obj0_sub0.a3d,vihou_3_obj0_sub1.a3d,etc....). These files also do not contain any visible reference to textures or images (unlike version 3). This is strange because the object name string is always preceded by 01 00 00 00 which could be a count, implying that the object name is part of an array that could have multiple entries, however the endianess doesn't follow the rest of the file so is likely not.

Structure

// A3DM2 e.g. A3D Modern 2
struct A3DM2
{
    char signature[4]; // "A3D\0"
    int version; // 0x02000000, 2
    int unknownInt1; // 0x01000000, 1
    int fileSize; // This is not exactly the file size (always abit larger) but the range seems to be <10 bytes
    char unknownBytes1[12];
    char* objectName; // Null terminated string that has the object name
    Vector3 unknownComponent; // Potentially origin or scale
    char unknownBytes2[13];
    int vertexCount; // Number of float triplets in the vertex buffer
    char unknownBytes3[8];
    Vector3* vertexBuffer; // Vertex buffer only contains float triplets
    int unknownInt2; // this is usually equal to 2 
    Vector2* uvCoordinates; // Pairs of floats that are likely UVs, same length as the vertex buffer
    int unknownInt3; // this is usually equal to 3
    float* unknownFloatData; // Some kind of buffer containing sane float values
    char* restOfFile;
};

Version 3

I have focused my efforts on version 2 as of now however I made some brief observations. The file seems to name multiple objects as well as texture files; these strings are encoded using a preceding int instead of being null terminated (unlike version 2). The files I have seen are named object.a3d unlike the descriptive names of version 2 files.

Structure

struct A3DM3
{
    char signature[3]; // "A3D"
    short versionMajor; // 0x03, (this may be a variant field rather than a version)
    short versionMinor; // 0x00, (unsure, older a3d2 format used major,minor shorts for version)
    char* restOfFile;
};

[Pyogenics]

The version 2 format contains visible, uncompressed, vertex and index buffers but I haven't yet figured out how to identify their lengths. However the vertex buffer seems to often be straight after the object name and Vector3 component, the index buffer seems to be the last/second to last data structure (there is sign of another data structure that looks abit like an index buffer but not quite right at the end). Here is a rough very hacky data read from a terrain mesh which only contains index and vertex buffer objects (I only read vertices + some junk because I simply don't know when the vertex buffer ends):

[Pyogenics]

I have identified the vertex count, here's a building model I managed to load (from 6 separate files):

[davidejones]

@Pyogenics As far as I know my A3D2 format in the plugin was was fully working with all the files i had tested. I notice you mention vihou_3_obj0_sub0.a3d, vihou_3_obj0_sub1.a3d files. Are these from a specific source and do they not load with the existing blender plugin here?

Can you upload your test files?

[Pyogenics]

These files are from an even newer format than A3D2.0, only implemented in the newer kotlin versions of Alternativa3D; the version 2.0 I mention is from one of these newer files. The format is much different as shown by my earlier comments.

[davidejones]

Ah ok, i think i added support for 2.0 2.4 2.5 and 2.6 in the existing plugin

[Pyogenics]

Here is a sample of models I am using currently (extracted from Tanki Online) + some hex dumps I left in (whoops) models.zip

[Pyogenics]

There seem to be a few data structures which contain coordinate data inside version 2 files (apart from UV and vertex buffers), each of these sections is preceded by an int value (could potentially denote type of data), most files contain atleast 01, 02 and 03 types. Here are some of the potential types I have observed:

00 00 00 00 ??
01 00 00 00 Vertex
02 00 00 00 UV
03 00 00 00 ??
05 00 00 00 ??

Here's a screenshot from a read of display.a3d (all data past the "unknown" point is just read as pairs of floats to try identify possible sane float data): And here is the code I used to read it:

from sys import argv
from struct import unpack, calcsize

def unpackFileStream(format, file):
    size = calcsize(format)
    fileData = file.read(size)
    return unpack(format, fileData)

def readNullTerminatedString(file):
    string = b""
    char = file.read(1)
    while char != b"\x00":
        string += char
        char = file.read(1)
    return string.decode("utf8")

def readA3D(file):
    signature = file.read(4)
    if signature != b"A3D\0":
        print("This is not a modern A3D file!")
        return -1

    version, unknownInt1, fileSize = unpackFileStream("<3I", file)
    print(f"Version {version}, unknown int {unknownInt1}, file size {fileSize}")

    unknownBytes1 = file.read(12)
    print(unknownBytes1)

    objectName = readNullTerminatedString(file)
    unknownTransform = unpackFileStream("=fff", file)
    print(f"Object name {objectName}, unknown transform {unknownTransform}")

    unknownBytes2 = file.read(13)
    print(unknownBytes2)
    vertexCount, = unpackFileStream("<i", file)
    print(f"vertex count {vertexCount}")
    unknownBytes3 = file.read(4)
    print(unknownBytes3)

    # Read vertex data
    print(file.read(4))
    input("Vertex data")
    for vertexI in range(vertexCount):
        startPosition = file.tell()
        vertex = unpackFileStream("=fff", file)
        print(f"{startPosition} {vertexI}: {vertex}")
    print(file.read(4))
    input("UV data")
    for vertexI in range(vertexCount):
        startPosition = file.tell()
        uvVertex = unpackFileStream("=ff", file)
        print(f"{startPosition} {vertexI}: {uvVertex}")
    print(file.read(4))
    input("Unknown")
    for vertexI in range(1000):
        startPosition = file.tell()
        vertex = unpackFileStream("=ff", file)
        print(f"{startPosition} {vertexI}: {vertex}")

with open(argv[1], "rb") as inputFile:
    readA3D(inputFile)

[Pyogenics]

Modern A3D Type 2 model loaded with face data

[Pyogenics]

Modern A3D Type 3 model loaded (basic loading)

[Pyogenics]

The models are separated into multiple data blocks which encode various pieces of data about the model, they are started with a unique int (a marker) which is used to verify the data during import. Here are the types seen so far and their markers:

01 00 00 00 - Root block, this contains everything
02 00 00 00 - Mesh block, contains mesh data such as vertex and index buffers
03 00 00 00 - Transform block, contains transform data about the stored objects
04 00 00 00 - Material block, contains material data
05 00 00 00 - Object block, stores hierarchy information and ties some of the read data to objects (like transform data)

Each block has their own structure and may vary between variants (for example: the material block of type 3 contains more data than type 2), here are their definitions in order of appearance in the file:

Root block

Contains all other blocks.

struct A3D3_2_RootBlock
{
    int marker; // 1
    int unused;
    A3D3_2_MaterialBlock materialBlock;
    A3D3_2_MeshBlock meshBlock;
    A3D3_2_TransformBlock transformBlock;
    A3D3_2_ObjectBlock objectsBlock;
};

Material block

struct A3D3_2_MaterialBlock
{
    int marker; // 4
    int unused;
    int materialCount;
    struct A3D3_2_Material {
        char materialName[];
        float unused[3]; // Unsure
        char diffuseMap[]; // type 2 often has no textures assigned to its materials
    } materialArray[]; // length of materialCount

}

Mesh block

struct A3D3_2_Mesh
{
    int vertexCount;
    int vertexBufferCount;
    struct A3D3VertexBuffer {
        int bufferType;
        float bufferDataArray[]; // length of vertexCount * vertex size
    } vertexBufferArray[];
};

struct A3D3_2_Submesh
{
    int faceCount;
    short indexArray[]; // length of faceCount * 3
    int smoothGroupArray[]; // length of faceCount
    int materialID;
};

struct A3D3_2_MeshBlock
{
    int marker; // 2
    int unused;
    int meshCount;
    A3D3_2_Mesh meshArray[];
    int submeshCount; // Referred to as "surfaces" in old A3D formats
    A3D3_2_Submesh submeshArray[];
};

Transform block

struct A3D3_2_TransformBlock
{
    int marker; // 3
    int unused;
    int transformCount;
    struct A3D3Transform {
        float position[3];
        float rotation[4]; // quaternion
        float scale[3];
    } transformArray[]; // length of transformCount
    int transformIDArray[]; length of transformCount
};

Object block

struct A3D3_2_ObjectBlock
{
    int marker; // 5
    int unused;
    int objectCount;
    struct A3D3_2_ObjectInfo {
        char objectName[]; // type 2 often has empty object names, this is likely why most are separated into multiple files (to avoid naming conflicts)
        int meshID;
        int transformID;
    } objectInfoArray[]; // length of objectCount
};

[Pyogenics]

Here are the blocks for version 3:

Strings are read differently than the previous variants:

struct string
{
    int stringLength;
    char string[]; // length of stringLength
    char paddingBytes[]; // number of padding bytes depends on string length.
                                    // python implementation: `paddingLength = (((stringLength + 3) // 4) * 4) - stringLength`
                                    // the rounding determines how many bytes it is
}

Root block

Contains all other blocks.

struct A3D3_3_RootBlock
{
    int marker; // 1
    int blockSize;
    A3D3_3_MaterialBlock materialBlock;
    A3D3_3_MeshBlock meshBlock;
    A3D3_3_TransformBlock transformBlock;
    A3D3_3_ObjectBlock objectsBlock;
};

Material block

The same as type 2 but with the new string format.

struct A3D3_3_MaterialBlock
{
    int marker; // 4
    int blockSize;
    int materialCount;
    struct A3D3_3_Material {
        string materialName;
        float unknown[3]; // Unsure, could be origin?
        string diffuseMap; // type 2 often has no textures assigned to its materials
    } materialArray[]; // length of materialCount

}

Mesh block

struct A3D3_3_Mesh
{
    string meshName;
    float unknown[7]; // Bound box data? (x-min, y-min, z-min, x-max, y-max, z-max, unknown)
    int vertexCount;
    int vertexBufferCount;
    struct A3D3VertexBuffer {
        int bufferType;
        float bufferDataArray[]; // length of vertexCount * vertex size
    } vertexBufferArray[];
};

struct A3D3_3_Submesh
{
    int indexCount;
    short indexArray[]; // length of indexCount
    char paddingBytes[]; // length is calculated the same as string
};

struct A3D3_3_MeshBlock
{
    int marker; // 2
    int blockSize;
    int meshCount;
    A3D3_3_Mesh meshArray[];
    int submeshCount; // Referred to as "surfaces" in old A3D formats
    A3D3_3_Submesh submeshArray[]; // length of submeshCount
};

Transform block

struct A3D3_3_TransformBlock
{
    int marker; // 3
    int blockSize;
    int transformCount;
    struct A3D3Transform {
        string name;
        float position[3];
        float rotation[4]; // quaternion
        float scale[3];
    } transformArray[]; // length of transformCount
    int transformIDArray[]; length of transformCount
};

Object block

struct A3D3_3_ObjectBlock
{
    int marker; // 5
    int blockSize;
    int objectCount;
    struct A3D3_2_ObjectInfo {
        int meshID;
        int transformID;
        int materialCount;
        int materialIDArray[]; // length of materialCount
    } objectInfoArray[]; // length of objectCount
};