An option to keep custom extensions in gltfpack

[mixtur]

In our engine we use some custom glTF extensions. (Our prefix is not registered with the spec) Is there a way to keep them from being deleted by gltfpack? If there is no such way, we would be happy to see one.

[zeux]

Do you have details on structure/schema/general purpose of the extensions?

[mixtur]

There is a number of them. I don't have JSON schemas, but I can provide typescript interfaces. I can also briefly describe their semantics.

export interface WG_dielectric_specular {
    // this is a way to provide Fresnel's normal incidence factor for materials
    factor?: number;
}
export interface WG_emissive_map_luminance {
    // an equivalent of KHR_materials_emissive_strength, except we consider the value to be in cd/m2
    luminance?: number;
}

export interface WGImageKTXInfo {
    // not an extension by itself, but used in the following extension
    uri?: string;
    mimeType?: string;
    bufferView?: number;
}
export interface WG_image_ktx {
    // a way to provide compressed textures
    astc?: WGImageKTXInfo;
    pvrtc?: WGImageKTXInfo;
    etc?: WGImageKTXInfo;
    etc1?: WGImageKTXInfo;
    rgtc?: WGImageKTXInfo;
    s3tc?: WGImageKTXInfo;
    s3tc_srgb?: WGImageKTXInfo;
}
export interface WG_image_radiance_hdr {
    // a way to provide .hdr images for textures
    name?: string;
    uri?: string;
    mimeType?: string;
    bufferView?: number;
    luminance: number; // after decoding hdr is normalized linearly to 0..1 and then every texel is multiplied by this number
}
export interface WG_material_depth_properties {
    // depth test / write values
    test?: boolean;
    write?: boolean;
}
export interface WG_material_shadow_properties {
    // whether this material casts or receives shadows
    cast?: boolean;
    receive?: boolean;
}
export interface WG_scene_environment {
    // a way to provide a skybox
    texture: {
        index: number;
    };
}

[zeux]

Ok, thanks. While some of these could be preserved in a generic fashion, for example WG_material_depth_properties, a lot of others (anything that interacts with any indices or textures) require specific knowledge about the extension schema. See https://github.com/zeux/meshoptimizer/issues/465#issuecomment-1465027503 as well, that issue omits extension details but they had a similar story - some value-only extensions could be transported but many required custom handling.

gltfpack intends to support KHR prefixed extensions and, in some cases, EXT prefixed extensions, but fully customizable extension support is beyond the scope as it requires custom configuration. glTF-Transform has support for custom extensions I believe and there you need to provide the extensions in the form of TypeScript modules if I'm not mistaken.

One option for some of these is to use KHR extensions and extras which gltfpack does preserve if asked as these normally contain simple values that can be safely transported across scene changes.

[mixtur]

I see. Would it change anything if we were able to provide extension schemas?

The simplest strategy that comes to mind: if there is something that matches { "$ref": "glTFid.schema.json" } in the extension, the entire extension is deleted, but otherwise kept intact.

[arpu]

We also looking at this, our use case is to use MX_lightmap from thirdroom https://github.com/matrix-org/thirdroom/blob/main/docs/gltf/MX_lightmap/README.md https://github.com/matrix-org/thirdroom/blob/main/docs/gltf/MX_lightmap/schema/node.MX_lightmap.schema.json

[arpu]

more samples at https://github.com/matrix-org/thirdroom/tree/main/docs/gltf

[zeux]

After more consideration I'm going to close this as "not planned", just as #465.

Support for extensions really requires them to be integrated into the overall processing. It's fairly impossible to do while retaining the integrity of the document without a schema. A schema would help preserve the integrity, but it still doesn't give any information as to what the semantics are; for material extensions, some floating point parameters may need extra processing (for example they may need to be scaled, like volume thickness, or they may affect higher level logic like mesh merging when the meshes need to be sorted), and texture references need information about what channels encode what type of information, to come up with texture compression settings. There's other types of issues for extensions in other parts of the scene, like node attachments and whatnot.

This is on top of the schema implementation in general being a very non-trivial addition to the existing gltfpack codebase, which alone would give me a lot of pause, but it's not even sufficient anyhow.

My overall feeling is that files with custom extensions should probably be processed by glTF-Transform which has an extension API, and gltfpack should remain a simpler but opinionated tool that needs to understand the entire scene to process it well. glTF-Transform is not as strong of an optimization tool in the sense that some processing is not as thorough (although it does provide some of the basics, using meshoptimizer library for some of the compression/encoding/optimization), so maybe some sort of hybrid is feasible where materials or other extensions get re-attached after gltfpack processing using glTF-Transform as a scripting framework.

This (lack of support for vendor-specific and custom extensions) is now documented in gltfpack readme (https://github.com/zeux/meshoptimizer/blob/master/gltf/README.md#extensions).

[wallabyway]

glTFpack already supports custom vertices, including _feature0, and compresses them. However, it currently excludes the EXT_mesh_feature extension, which causes it to be (correctly) ignored by some viewers.

To quickly test support for meshOpt with the EXT_mesh_feature extension, I made a small modification to add the extension back:

           ...
        {"EXT_mesh_gpu_instancing", ext_instancing, true},
        {"EXT_mesh_features", true, true},
    };

I added the above lines to this section of code: meshoptimizer/gltf/gltfpack.cpp#L842

After making the change, I compiled the project:

cmake . -DMESHOPT_BUILD_GLTFPACK=ON -DMESHOPT_BASISU_PATH=basis_universal -DCMAKE_BUILD_TYPE=Release
cmake --build . --target gltfpack --config Release

I then ran the modified gltfpack on a sample 0.glTF file containing FeatureID’s. The output was a GLB file that included the EXT_mesh_features extension header and the FeatureID’s in the custom vertices _feature0 buffer:

0.glb.zip

FYI. @petrbroz