search

immersive-web / spatial-favicons

A proposal for favicon support for a new data type. Feature lead: Rik Cabanier
Other
12 stars 7 forks source link

Need clarification on "sizes" #5

Open raviramachandra opened 5 years ago

cwilso commented 5 years ago

For detail: this issue arose from a discussion in the Feb 26th CG telecon - https://www.w3.org/2019/02/26-immersive-web-minutes.html, search for "size". The core issue here is that "sizes" is used in flat image icons to help the UA select an appropriate resource to download - based on how it will render, you can avoid upscaling (by choosing a "big enough" resource) as well as downloading too big or detailed a resource (by choosing one that's "not too big").

Unfortunately, this isn't quite the same as for 3D resources - where we probably want something that is more akin to "level of detail". You'd still want to only d/l a low-poly-count/small-texture model for a low-res display, or far-off LoD, but might want to offer a more detailed model for higher res or closer PoV.

raviramachandra commented 5 years ago

Our original intent was to use a bounding box computed by asset developer and specify this as a "hint" to UA/platform. There is a LoD extension from MSFT, but that would not be helpful to our case, since various levels are packaged in the same asset. Would something like lod: "high"/"medium"/"small" text work ? We will also explore other and get back here.

cwilso commented 5 years ago

Yeah, something like that, but we'd probably need to give some more direct guidelines as to what that meant in terms of physical-to-user size?

NellWaliczek commented 5 years ago

IIRC from TPAC, we'd discussed both LOD and physical size as information that would influence usage and that it was the specific use case that drove which property would be more important.

raviramachandra commented 5 years ago

That would an ideal case. If we want to do away with minimal spec changes, it would be best to interpret 'size' attribute and make a decision based on it. If we want both, we would have to introduce new fields in WebApp manifest and new attributes to tag. We prefer not to add additional attributes, but if the immersive-group thinks there is value to it, we are open. @NellWaliczek what do you think ?

DRx3D commented 5 years ago

These comments are mostly derived from the presentation during the F2F on June 5. If this is not the correct place to post them, please let me know where I should post.

I like the idea of 3D model icons; however, there are a number of issues that need to be address.

  1. The quality of a model is not solely determined by its (virtual spatial) size nor the number of triangles. Textures play a large part in the quality.

  2. Image FAVICONS are (almost) always separately designed and built. There are limitations on the number of pixels (device & use dependent) for the images so the designers need to rebuild the images rather than just shrinking them. The same would apply to models and/or textures.

  3. glTF models (with PBR) are best lit with environment maps. Would there be a standard and/or default one, or is it up to the experience builder to provide one?

  4. glTF mirrors current GPU API's (https://github.com/KhronosGroup/glTF/blob/master/specification/2.0/README.md#design-goals); however, there is no backward compatibility requirements within the specification or Khronos' process (see changes from V1.x to V2). What happens if a future version of glTF is not compatible with V2?

  5. How does a static, fixed-position model differ from an image attached to a billboard. If it is possible to "walk around" the model and see it from different sides, then does displaying the "back-side" cause problems with intended usage? Displaying the "back side" was not obvious to me in the provided examples.

raviramachandra commented 5 years ago

@DrX3D thanks for the feedback.

  1. Agree, but we are not really looking to measure exact quality here. We just need to provide a "hint" to platform. The notion that vertex count, or spatial bounds may provide a "hint" of the asset is all we were looking for.

  2. Yes and No. Mostly depends on website authors. Again same principal applied to 3D icons. If the content owner wants to specifically design for favicon (hopefully most profession websites will do) it will look best.

  3. Not sure about environment maps. For favicons we did not think we needed explicit environment maps.

  4. Yes, we are aware that it was not backward compatible with V1.x. But glTF 2.0 is the most agreed open source 3D file format. There are some members both in W3C and Khronos, and hopefully we can influence backward compatibility for future releases.

  5. Walking around the model will not be a problem. Did you mean "back side" in the sense "double side texture": https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#double-sided ?

Also, what is an alternative to guess-timating quality either on poly count of spatial size ?

kearwood commented 4 years ago 
3\. Not sure about environment maps. For favicons we did not think we needed explicit environment maps.

A goal of PBR rendering is to separate the material attributes of a model from the environment that the object is to be presented in. Rather than specifying the environment map as part of the favicon, it should be generated or chosen by the UA to represent the environment that the favicon is being displayed within.

The methods used to generate the diffuse irradiance and cube map / environment map for reflections in are described in the lighting-estimation incubation repo. For display within a VR headset, the same algorithms would be performed using the UA's internal scene geometry. For an AR headset, this can be augmented with sensor sampled data and AI algorithms.

A successful lighting model would be one that holds up in a future that incorporates real-time global illumination and raytraced reflections as well as current mainstream imaged based lighting techniques.

kearwood commented 4 years ago

IIRC from TPAC, we'd discussed both LOD and physical size as information that would influence usage and that it was the specific use case that drove which property would be more important.

In particular, a commonly used method of determining the LOD level decoupled from scaling or distance was described at TPAC.

This "screen coverage" method works by determining the maximum FOV covered by the bounding box of the model, as an angle (degrees / radians). This angle is used as the criteria to select the appropriate LOD model. An "LOD Bias" multiplier can be applied to this angle value by the UA or experience that is rendering the model to adjust for varying hardware capabilities and display fidelity. Often this can be done dynamically to target a minimum frame-time for the rendering of the objects.

raviramachandra commented 4 years ago

@kearwood did you see the proposal by @rsahlin ? If glTF standardize that, it would provide us a good starting point. IMHO, to get a standarized LoD/Quality may take some time. In the intermediate future, I suggest we use "small", "medium", "large" abstract values. Please advice.

thetuvix commented 4 years ago

Excited that @grorg brought back up spatial favicons today!

This "size" issue is the key thing I believe we need to solve before we standardize here. There are two key approaches I've seen us discuss for handling different levels of detail here:

  1. Multiple spatial favicon "size" values, each pointing to a different glTF file for different levels of detail, perhaps based on well-known sizes that we define here
  2. One spatial favicon, whose single glTF file contains multiple levels of detail, perhaps based on the glTF LOD mechanism to be agreed on in https://github.com/KhronosGroup/glTF/issues/1743

It sounds like Magic Leap's shell currently uses size of the placed model in meters to choose an LOD, and so something more static like approach 1 best fits there (since you don't need any other sizes unless the user can resize the model). Meanwhile, the HoloLens shell uses screen coverage to choose an LOD, does something more dynamic like approach 2 fits best there (since other sizes can be quickly loaded as you move closer).

It would be great to hear which of these approaches best matches what @grorg is thinking here!

grorg commented 4 years ago

FWIW, sizes has a similar problem with vector artwork, where you might want different variants based on the display resolution even though the intrinsic size is the same. And, like a 3d model with geometry and textures, you could have a huge amount of data in a small "size", or visa versa.

For this reason, I think it is ok to start with simply mapping icon size to the approximate resulting size of the icon in display pixels (taking scaling into account, etc). It is also ok for the browser to enforce a limit on the data size of the resource, or the rendering complexity - encouraging people to make the best icons they can, as small (in complexity) as they can.