Level of Detail (LoD) or Quality of glTF asset

[raviramachandra]

W3C immersive-web group is planning to use glTF as an asset for Favicons, WebApp Manifest Icons on XR devices. See this: https://github.com/immersive-web/spatial-favicons/blob/master/explainer.md

To allow Browsers/Platforms to select from multiple glTF icons appropriately to the use case, a quality/LoD metric is needed. There are many questions pertaining to quality of the asset.

1. Should this be a single weighted sum of individual quality vectors ?
2. Or should this be a set of metrics which together provide a quality measurement ?

This issue is opened to track this effort and derive at this.

[rsahlin]

Hi @raviramachandra

Have you looked at the Microsoft LOD extension? https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Vendor/MSFT_lod

Does this cover your needs to store the actual LODs?

Could you please explain how you would use the quality vectors? Do you mean to have a sum of the 'weight' of the model - as total number of nodes/vertices and textures?

From your usecase desciption I would guess that your browser/platform would choose the smaller LOD in case of bookmarks and use the heavier ones in other usecases - or am I missing something?

[donmccurdy]

The phrase "LOD" might be misleading here — I don't think the W3C question is about storage of the LODs, which would likely be in entirely separate glTF files unlike the Microsoft extension. The difficult question is how choose the metric(s), expressing to the browser the quality of each available glTF file in a way that enables it to choose the appropriate version.

The Microsoft extension uses screen coverage as the single metric for their extension. Which really describes the context into which the model will be loaded, as all LODs presumably have the same dimensions. I don't know whether that metric would be right for W3C? Alone, it leaves no room for flexibility based on factors like filesize, network bandwidth, or device GPU performance. But it is similar to HTML srcset.

[rsahlin]

Ah, thanks @donmccurdy This is also a setup we are looking into at Ikea.

I think we need to make distinction between, at least: A: Metric for model download size (bytes) - Download Metric B: Performance metric - eg in a very genaralized way describing how demanding the model is to render. - Runtime Metric

At least in our case the Runtime Metric would be used together with the Download Metric to determine which model is downloaded to a target (viewer).

@raviramachandra What's your view on these metrics with regards to what you want to achieve (your usecases)?

[raviramachandra]

@donmccurdy we did look at the MSFT extension. But we could not use it, because all LoDs are embedded in the same asset. Like you mentioned, Browser/Platform will make a choice of asset from a list of assets to use and may only download the appropriate asset.

MSFT extension considers only screen coverage, which is similar to an Image asset resolution. It does not consider vertex count, texture sizes which all contribute to "quality". This is a good use case for what we are trying to achieve: https://github.com/immersive-web/spatial-favicons/blob/master/explainer.md#multiple-icons

@rsahlin Download Metric is not something we considered. Runtime Metric may be more appropriate for the usecase. It has some overlaps in the sense that a high quality assets will use more CPU/GPU/Memory resources than a lower one. Are there details about Runtime Metric, like how do you calculate ? What factors (Node counts, Vertex Counts, Texture sizes etc) are considered to determine it ?

[rsahlin]

@raviramachandra Thanks for the clarififaction There are no specific metrics for the runtime metrics of a glTF mode - we will need to agree on how to calculate this. I think be best solution would be to break down in seperate areas that affect the performance (multiple values) - rather than providing one (scalar) value.

Maybe something along the lines of:

• Total verticecount - [verticecount]

• Number of primitives and / or meshes (ie what would end up as drawcalls unless batched/instanced) [nodecount] and / or [primitivecount]

• Material properties - PBR? [materialtype] pbrMaterialRoughness other / none

• Textures per material: baseColor, metallicroughness, normal, occlusion, emissive - ie number of texture 'channels' [texturecount] I guess for texturecount and materialtype it's best to count the heaviest / highest count since it's not known how the user views the model. Ie, if there is one material with pbr and 5 texture channels , then this is what will be stored.

• I'm not sure a texture quality (texel density) value is needed - I would assume viewers generate mipmaps so there would be no oversampling. Eg for one displayed pixel on screen each texture channel would be sampled once. Sure, more if viewer does filtering but that is probably not the normal case.

• Transparancy info?

What's your thoughts and comments on this?

[rsahlin]

I wrote a small program that parses glTF into memory, traverses each scene and collects some metrics:

WaterBottle

Loading into buffer with size 149412 from WaterBottle/gltf//WaterBottle.bin Model has 1 scenes, 4 defined textures, 1 defined materials Scene 0 has 1 nodes, nodes with mesh=1, total primitivecount=1, verticecount=2549 Unique materials used in scene=1 Average textures (per vertex)=5.0

GearboxAssy

Loading into buffer with size 4915704 from GearboxAssy/gltf//GearboxAssy0.bin Model has 1 scenes, 0 defined textures, 42 defined materials Scene 0 has 118 nodes, nodes with mesh=95, total primitivecount=95, verticecount=360958 Unique materials used in scene=42 Average textures (per vertex)=0.0

From this it would be easy to take a decision if the Waterbottle (1 node, 1 primitive, 5 texture channels) or GearboxAssy (118 nodes, 95 primitives, 0 texture channels) is heavy to render on your target platform.

Hope this helps!

[raviramachandra]

@rsahlin Thanks for the info. I am following up this internally in Magic Leap and will respond to your earlier comment.

[raviramachandra]

@rsahlin I did have a discussion with both internal (within MagicLeap) and W3C community folks. We agree that these metrics provide fair quality assessment of the glTF asset. One factor that Microsoft uses is "screen coverage". This was also suggested by @kearwood in our github issue. As a glTF standard determining the LoD based on the above attributes would be a good start. The web community could use some kind of script (or online tool) to upload an asset and determine this vector of independent metrics.

[rsahlin]

Hi @raviramachandra Yes the 'screen coverage' factor is the largest runtime factor - however I'm not sure that it is needed in the same way for your usecase.

For your usecases I think it could be enough to know how large the render destination is - eg for a given usecase the target screen 'factor' is 0 -> 1 - where 0 is no coverage and 1 is the whole screen. A bookmark model would then cover perhaps 0.01 of the screen and add very little performance overhead. This factor could of course be multiplied by the performance 'weight' to produce some sort of number that can be used in classification.

Maybe you would need the screen coverage metric for a usecase where most of the screen is occupied by the model. But that raises the question of how to pre-calculate the screen coverage as the model may include animation or the user may rotate or move closer - ie changing the screen coverage.

Personally I would favor a method that assumes worst case - ie the size of the render target is used as screen coverage.

[raviramachandra]

@rsahlin yes, it makes sense to assume worst case. Also screen coverage will depend on how the platform rendering the asset will scale to it's needs. What would you recommend be the next steps in formalizing this.

[rsahlin]

I guess the next steps are:

A: Formalizing the metrics used to performance classify a glTF model - eg name a metric, describe what it means and define how it is calculated. This is probably best done in a PR once B: is figured out.

B: Take decision if these metrics should be part of glTF as metadata or maybe added as part of 3D commerce metadata (XMP) or configuration - ie where does this feature belong?

I think this feature is a configuration parameter that will be used by (mostly) backends to figure out what models to send to clients. What are your views on this?

[donmccurdy]

I think this feature is a configuration parameter that will be used by (mostly) backends to figure out what models to send to clients.

For W3C purposes, my hunch would be something more like web responsive images and manifest icons. The user agent (client) is provided a list of glTF resources and (external to those resources) certain metrics about each, and chooses a resource as appropriate for the device capabilities, network speed, user preferences, and so on.

If that is the case, I believe it's useful to choose good metrics and formally describe them, but not necessarily a requirement to embed them into glTF files.

[raviramachandra]

@donmccurdy agree, you put it correctly.

@rsahlin Part A is of most interest for W3C. Depending on if Part B i.e inclusion of this metric in glTF metadata, we should provide a simple "online tool" for backend implementations to have the ability to get metrics out a given glTF asset, so it can be embedded into HTML tags or App Manifests as mentioned by @donmccurdy .

Would you like to take the lead since you are most knowledgeable in this area ?

[rsahlin]

Sure @raviramachandra I'll do my best :-)

I think the first step would be to figure out which working group of 3D Commerce that this feature would belong to. From what I can tell we all agree that the resulting metrics should probably not end up in the glTF file.

What do you think of having this as a feature in 3Dcommerce metadata? Or do you have some other suggestion?

I also think we need to setup repos for 3Dcommerce (metadata/certification/validation/configuration)

• Who usually takes care of creating repos?

[raviramachandra]

From what I can tell we all agree that the resulting metrics should probably not end up in the glTF file.

@rsahlin agree. Not sure if you have seen. FB has an extension of subset of these defined here

What do you think of having this as a feature in 3Dcommerce metadata? Or do you have some other suggestion?

Not sure. I have not been involved there. But I was hoping at the end of this exercise, we publish a clear Guide in glTF repo and also develop a simple Tool (python or Javascript) that will spew out the above metrics.

Should I bring this up at Weekly Meetings ?

[rsahlin]

Hi @raviramachandra

Not sure. I have not been involved there. But I was hoping at the end of this exercise, we publish a clear Guide in glTF repo and also develop a simple Tool (python or Javascript) that will spew out the above metrics.

Good suggestion - now I'm starting to understand the way forward :-)

If I understand the comments correctly we could move forward with a PR - this PR will declare the performance metrics and provide one (or more) tools to calculate these metrics. We cover this PR in the 3D formats (gltf) working group.

Yes, I did see the FB extension and it covers similar ground - however I think we need the additional metrics as discussed here.

I think it would be good to have a short discussion regarding the way forward in the coming 3D formats call.

[raviramachandra]

@rsahlin Did we covere this topic on Feb 26 call. Will it get covered next week ? I did not see any notes w/ this issue. I will plan to attend the next call.

[rsahlin]

@raviramachandra I put on the agenda but time ran out - hopefully next week :-)

[rsahlin]

I have opened up a PR to move forward as agreed: https://github.com/KhronosGroup/glTF/pull/1779

[johannesvollmer]

Embedding this data into a file would not be very valuable, as it could not be trusted, right? Otherwise, one could upload a model that claims to be small, but is actually complex, possibly resulting in denial of service?

[John-Nagle]

Did levels of detail ever make it into the glTF spec? The PR mentioned above, https://github.com/KhronosGroup/glTF/pull/1779 apparently died with Magic Leap.