Support for UVs as Vector3+Vector4 requested

[JaXt0r]

Is your feature request related to a problem? Please describe. Our project leverages Export and Import of glTF files at runtime. To enhance our performance, we already implemented usage of Unity Texture Arrays and MipMaps. It means our texture coordinates (UVs) are of type Vector4 (x, y, textureArrayIndex, mipMapLevel). When we store a mesh, it saves Vector2 for UVs only and the textureArrayIndex+mipMapLevel information get removed.

Out-of-scope:
The feature request asks for Vector4 on UVs only. Texture Array support for Materials isn't needed as we retrieve the Arrays manually after Mesh restore from glTF.

Describe the solution you'd like It would be great if glTFast can support UVs as Vector2, Vector3, and Vector4. It would be fine, if you add a parameter which tells the Exporter/Importer which version to use.

Example usage we could think of:

// Exporting
var exportSettings = new ExportSettings
{
   TextureCoordinateType = typeof(Vector4) // Defaults to typeof(Vector2) for backwards compatibility
   ...
 };
var export = new GameObjectExport(exportSettings);
export.AddScene(...);
...
export.SaveToFileAndDispose();

// Importing
// No parameter for UV=Vector4 needed, if this information is stored inside gltf file itself.
// Otherwise we could leverage an ImportSettings property.
var import = new GltfImport();

Describe alternatives you've considered If glTF isn't providing a native solution, we need to store additional metadata in a JSON (etc.) file. After the meshes are loaded, we need to fetch all vertices and merge the missing two vector data points for all UV elements. Doable, but not preferred as:

1. We need to store additional metadata (additional file handling for mesh based data)
2. We need to loop through all the UVs (additional CPU cycles to iterate through the data again)
3. Unity doesn't allow updating UVs, but only to set them altogether. Which means we need to create a new UV array and store it in one chunk at the end of the loop (More RAM usage during execution)

Additional context

Unity's UV getter: for various Vector* implementations:

• Mesh.uv (legacy): https://docs.unity3d.com/ScriptReference/Mesh-uv.html
• Mesh.GetUVs(int channel, Vector2|Vector3|Vector4 uvs)

Test images from our project:

• 1: Before saving glT file

• 2: After loading the glT file

[atteneder]

Hi @JaXt0r ,

I investigated that a Vector4 texture coordinate is not covered by the glTF 2.0 specification.

I manually crafted such a file and the validation failed with the following error:

Invalid accessor format '{VEC4, FLOAT}' for this attribute semantic. Must be one of ('{VEC2, FLOAT}', '{VEC2, UNSIGNED_BYTE normalized}', '{VEC2, UNSIGNED_SHORT normalized}').glTF Validator(MESH_PRIMITIVE_ATTRIBUTES_ACCESSOR_INVALID_FORMAT)

That being said, that does not prevent anyone from implementing support for VEC4 UVs and optionally crafting a custom extension around it.

I don't think that would be too hard to implement, but given that this is a very narrow use-case I won't put a high priority on it.

However, I can offer guidance if you decide to beat me to it when it comes to implementation.

As a starting point, could you provide a mesh that has 4 dimensional UVs? Frankly I lack the knowledge/tools to do it quickly.

Thanks.

[JaXt0r]

Hi a@atteneder, Thank you for your reply. Interesting, that it's not covered by the specification.

For now, we moved along without caching at all. Feel free to close this topic.

Nevertheless, do you have some good examples on how to craft such extensions to add more data (uv3, uv4, color32, ...)? If we come back to it, I could even think of crafting these extensions on our own. (And potentially link it here for the community 😬)

Thanks. JaXt0r

[atteneder]

@JaXt0r

Yeah sure.

As stated before, technically there's no obstacle to add vec3/vec4 UV support to glTFast. The problem is that those glTFs are not within the specification and when you try to load them with another viewer/SDK it's not expected to work.

Crafting a glTF extension (let's pick a draft name like KHR_mesh_texcoords_dimensions) would solve this and I see two approaches.

No Fallback Extension

Adding KHR_mesh_texcoords_dimensions to the list of extensionsRequired is sufficient to inform glTF implementations that this glTF asset uses vec3/vec4 type UVs. Those implementations who don't support it will refuse to load it. For completeness extension needs to be added to extensionsUsed as well.

Otherwise there's no addition to the glTF JSON schema, which makes it simple and elegant.

The KHR_mesh_quantization extension is a good blueprint here.

Fallback enabled Extension

If it may be a hard requirement that the extended glTFs are supported (to some degree) in implementations that don't support the extensions, it gets a little more sophisticated.

One way is to provide alternative UVs in differing accessors/bufferViews. A mesh's primitives might look like this:

"primitives":[
        {
          "attributes":{
            "POSITION":0,
            "TEXCOORD_0": 1
          },
          "indices":99,
          "material":0,
          "extensions": {
            "KHR_mesh_texcoords_dimensions": {
                "TEXCOORD_0": 2
            }
          }
        }
      ]

Notice that vec3/4 UVs are in accessor 2 and fallback vec2 UVs behind accessor 1.

The compromise is that you'll have redundant data.

...unless Accessor 1's bufferView points to the same buffer as accessor 2's bufferView with a different byteStride (I'm not entirely sure that's conformant though).

You could also expose the 3rd/4th component of KHR_mesh_texcoords_dimensions.TEXCOORD_0 as TEXCOORD_1 in the fallback, but apart from preservation in content pipelines I don't see a good reason for that if the final implementation cannot make good use of it anyways.

NOTE: Just because with this approach having a fallback becomes feasible does not mean that's the only way. You could still require the extension and use vec3/4 UVs in the primitive attributes.

Whether a fallback is required is to be discussed. I personally wouldn't force it in if not absolutely required.

hth