Add support for static 3D favicons

[rcabanier]

Favicons are currently limited to 2 dimensional images. Create a specification that allows the display of 3D content.

This feature will build upon the existing infrastructure and must not introduce new fundamental concepts or security concerns. We will propose a very simple addition to the existing favicon syntax.

[fernandojsg]

Link to the thread started on the public IW mailing list for more context: https://lists.w3.org/Archives/Public/public-immersive-web/2018Aug/0007.html

[DRx3D]

There were two comments from the email thread that were not answered (at least that I noticed).

1) "...work with Khronos on a web-friendly version of gltf". Having worked with Khronos and Fraunhofer during the early development of glTF V2, I can say it is web friendly. Of course it could be web-friendlier; but at its basic, data comes in from a remote connection and with minimal CPU processing, goes into the GPU. I don't think you will get much throughput improvement with static geometry; however, textures and animation may have significant improvements.

2) glTF V2 is strictly Physically Based Rendering (PBR). That strong;y prefers use of the GPU to render the model. Preventing the use of shaders would put a severe limit on rendering.

[dwsinger]

As a point of information, the HEIF file format can support alpha, animation, and alternatives (and is not tied to the HEVC coding format). Not that I think in-file alternatives are a great way to go if there are out-of-file ways to express that.

[mkungla]

How would you expect future bookmark pages to work? Browser running 100-s of scripts to process individual 3D favicons as individual scenes or bookmarks would be independent scene which renders each bookmarked icon as object in bookmarks scene?

[rcabanier]

@DrX3D is it your belief that gltf is already perfectly safe to use on the web? At a minimum, I think we need to state that there is a maximum byte size and that the files should be self-contained. Is there a danger in the shaders that are used? Could they stall or crash the GPU?

[rcabanier]

@dwsinger Why would we use HEIF? Can it describe a 3D model?

[rcabanier]

@mkungla yes, managing all the assets for each favicon will be a burden on the browser. Having just a model file means that browser can keep using their existing logic.

[dwsinger]

@rcabanier HEIF can encapsulate any format. So, for example, we could define how to store a popular 3D model format as the primary item. (In fact, one can store anything that has a mime type, anyway). I just wanted to make sure you are aware; I'm neither proposing nor anti-proposing. But it would be easy to define HEIF variant for which the primary item is a 3D model, and there is an alternative that is GIF/JPEG/PNG coded. Similarly there are existing definitions to say "that item is a transparency mask for this one" and so on. HEIF allows MP4 tracks for animation, as well.

[rcabanier]

@dwsinger since this is for favicons which already have an established fallback mechanism, I think HEIF won't buy us much.

However, if we want to add support for 3D models natively to the web page, I can see this being very useful.

[TrevorFSmith]

@dwsinger I like the general idea of an encapsulation format but I'm concerned that the burden of adding this level of abstraction (especially with a format that isn't supported in other aspects of most UAs) would limit update among UA implementers.

glTF is gaining tool and workflow support as well as general use in existing web apps, so if we choose a single format for 3D favicons it seems like a reasonable choice.

[TrevorFSmith]

I've renamed this Issue to make it clear that it's for discussing the static favicon path. If anyone would like to discuss the possibility of discussing the dynamic, scriptable favicon path then please open a separate Issue.

[DRx3D]

@rcabanier: To some extent glTF safety is dependent on what you mean by "safe". It is possible to crash a browser with a glTF model. It is also possible to crash a browser with a sufficiently large JPEG. glTF does not contain code (including shader code) and by itself does not render. Except for size and animation complexity, I have not seen any glTF model that crashes browsers or otherwise leaks information.

The display of the model depends on the code used to display it. Khronos specifies the shader equations and may provide sample code. There are open source implementations of rendering code (e.g., THREE.js + glTF loader) that include shader code. Their safety/reliability depend on individual implementations.

I am not going to warrant that everything that can be expressed in a glTF file is risk free -- I have not studied the format with that in mind. There may be some pathological cases that fail. Also the standard continues to evolve. Also glTF has the capability of adding cameras and lights to the scene. These can cause more issues. OTOH, glTF is an open standard along with several implementations and that makes it easier to put lots of eyeballs on it.

-- Leonard Daly 3D Systems & Cloud Consultant LA ACM SIGGRAPH Past Chair President, Daly Realism - /Creating the Future/

[RafaelCintron]

I am good with having a 3D model serve as a favicon, so long as we prohibit the execution of arbitrary shaders or code. Formats that rely on a PBR-only based approach fit the bill for me.

[cwilso]

Favicons are defined by HTML - and I think you actually don't need to add anything there, as the rel=icon type isn't restricted type-wise IIRC, it will just be "now this type is recognized" (we used to support a very different set of file types for icon - pouring one out for .XBM files!). For app manifest, it's a bit harder, as "icons" is defined as an array of ImageResources (https://www.w3.org/TR/appmanifest/#icons-member), which are supposed to be type=image in terms of Request. Not sure if that will be a problem, will need to investigate.

[cvan]

For app manifest, it's a bit harder, as "icons" is defined as an array of ImageResources (https://www.w3.org/TR/appmanifest/#icons-member), which are supposed to be type=image in terms of Request. Not sure if that will be a problem, will need to investigate.

it's totally okay to extend the manifest format without needing the spec to change. see the Extensibility section in the spec.

[kenzkiran]

This is in line with what we were thinking at Magicleap, Here is our proposal

Problem Statement:

As companies increase focus on immersive web experiences, current two dimensional favicons fall short of providing a complete 3D experience. A favicon provides a quick visual representation of the web page/site and is used by user agents across many use cases. Here is a non-exhaustive list of use cases: Identify a tab in multi-tab browser / Address bar. As a bookmark or shortcut on platform. An entry in history.

With XR browsers now having ability to show 3D objects, having a 3D favicon augments well for a more complete and richer immersive experience.

Current Representation (HTML5 spec: Section 4.8.6.5):

A link element with ‘icon’ keyword is used to represent a page or site favicon. The ‘sizes’ attribute indicate width and height of the icon in terms of raw pixels. The ‘type’ attribute indicate the mime type of the icon. The ‘rel’ attribute is either ‘icon’ or ‘shortcut icon’. User agents support many image formats: ICO, PNG, SVG, JPEG, APNG etc

Proposal:

• Continue to use link element with but with ‘rel’ attribute as ‘world icon’ or a simpler ‘3d icon’.
• A user agent that can represent 3D favicon can choose to select the appropriate one when multiple favicons are provided.
• The ‘sizes’ attribute shall not be used for 3D favicon links.
• We could re-interpret an extended ‘sizes’ to contain : width x height x breadth but that needs more thought and discussion.
• 3D favicons shall also follow the same cache policy like current favicons.

The problem of 3D asset format:

• Generic requirements for a mime type that can be used to represent a favicon:
• Open, Extensible, inter-operable format with no private IPs or Patents.
• The 3D Asset shall be provided as a single self containing binary. Geometry and Textures (and any other data needed) embedded in the same file.
• Textures and other embedded images shall also be one of the supported open image formats.
• Industry accepted.
• Availability of free and open generation (optional) and viewer tools.
• Based on the above guidance, we have found that GLTF Version 2.0 binary (.glb) that best fits our needs. Web friendly, supported by 3js, Babylon, Sketchfab. Authoring tools support: AutoDesk, Adobe, Unity, Blender.

Constraints on GLTF (Favicon 3D Asset):

• Favicons are typically small (in KBs) , 3D assets can be order of magnitude larger (typically many MBs).
• We propose to restrict 3D favicon asset file size.
• Based on our current experience, we feel 1MB max is a good limit.
• Other restrictions such as vertex counts, etc may need additional runtime parsing.
• Built-in animations may not be supported initially.
• Backwards compatibility
• As per GLTF spec: Minor version updates shall be backward compatible, while Major version updates may not be backwards compatible.
• Shall be a self contained binary, no external textures or files.

Example: <link rel=’3d icon’ type=’model/gltf-binary’ href=’favicon3d.glb’>

[cvan]

:+1: really great write-up here. thanks for sharing!

The ‘rel’ attribute is either ‘icon’ or ‘shortcut icon’.

FYI: shortcut is not a valid link relation for <link rel="icon">. See this awesome blog post.

<link rel=’3d icon’ type=’model/gltf-binary’ href=’favicon3d.glb’>

So I'd suggest using <link rel="icon"> and make sure the type is not of a flat-image type, like so:

<link rel="icon" type="model/gltf-binary" href="favicon3d.glb">

IMO, developers should also be able to define a model without defining the type explicitly in the HTML, like so:

<link rel="icon" href="path/to/my/favicon3d.gltf">

And a small suggestion: If a browser loads /favicon.ico or path/to/my/favicon3d.gltf defined from a <link rel="icon"> in the document, the 3D model should be recognised and loaded as long as there is an HTTP Response header that matches (or starts with) Content-Type: gltf-binary or Content-Type: model/gltf+json.

Admittedly, I'd prefer to see this go in the icons member of the Web-App Manifest, as that seems like a more appropriate and extensible place to let developers define paths to 3D metadata, especially icons.

[riccardogiorato]

I totally agree with @kenzkiran

Being immersed in XR experiences and future browsers require some kind of 3d favicon, being able to showcase the logo of a brand in 3d or their mascotte is a must to start being more immersive.

Thinking ahead in the future 1mb for an optimized icon would be the best and even if it seems too much thinking about this in 5/10 years should make us realize that it's better to implement this now with a max file size of 1mb and maybe in the future expand it gradually the same way we have dozen of different favicons for retina devices, windows metro or android icons for the same website.

[rcabanier]

I'm glad to hear that almost everyone is in agreement. What should the next step be? Create a list of gltf constraints? Is there someone that is very familiar with this internals of this format in this group?

[kenzkiran]

I think there are a few open points:

1. Adding gltf-binary to web-app manifest ? I need to figure out how to do that.
2. Constraints on GLTF, should these be more like a guidance and let user agents deal with it or should it be enforced rule.

[rcabanier]

There should be a set of constraints so authors can create icons that can work on any user agent that supports this feature.

[cwilso]

I think I would disagree - the constraints can be shared as informative guidance, but constraints like that tend to get overrun in the future and become irrelevant. I'd much rather give suggestions but not normative restrictions.

[DRx3D]

Is the intent to just allow static models? It is important to note that static means that the relationship between the camera and model never changes, the model doesn't change, the lighting doesn't change, the texture doesn't change. So, if nothing changes, why not just an image? That way the author would not have any questions about the renderer on the remote end.

The appearance will certainly be more interesting if things aren't static. What would be allowed to move, camera, lights, model, texture?

I think the general concept of 3D icons is good. I am concerned about the ability to fully implement in a secure manner the desired set of features.

Perhaps it is sufficient to have a static 3D model with a single static camera view, static lights, and static texture but that is capable of being rendered stereoscopically.  Until that is determined, I think it is a bit premature to work on limitations of a particular format.

-- Leonard Daly 3D Systems Architect & Cloud Consultant President, Daly Realism - /Creating the Future/

[rcabanier]

Is the intent to just allow static models? It is important to note that static means that the relationship between the camera and model never changes, the model doesn't change, the lighting doesn't change, the texture doesn't change. So, if nothing changes, why not just an image?

GLTF can have built-in animations ie https://threejs.org/examples/#webgl_loader_gltf_extensions and select "monster" A static gltf would be completely 3D but just wouldn't move/change

Perhaps it is sufficient to have a static 3D model with a single static camera view, static lights, and static texture but that is capable of being rendered stereoscopically. Until that is determined, I think it is a bit premature to work on limitations of a particular format.

That's why it would be good to know what can be done with the format. Also, afaik favicon can't be observed by the document so things like timing attacks wouldn't work

[rcabanier]

I think I would disagree - the constraints can be shared as informative guidance, but constraints like that tend to get overrun in the future and become irrelevant. I'd much rather give suggestions but not normative restrictions.

we can always extend the constraints when UA become more powerful. My main worry is that if there are no constraints, designers will create icons that will make everything slow or that won't work well on systems that have limited resources

[DRx3D]

On 9/12/2018 10:54 PM, rcabanier wrote:

Is the intent to just allow static models? It is important to note
that
static means that the relationship between the camera and model never
changes, the model doesn't change, the lighting doesn't change, the
texture doesn't change. So, if nothing changes, why not just an image?

GLTF can have built-in animations ie https://threejs.org/examples/#webgl_loader_gltf_extensions and select "monster" A static gltf would be completely 3D but just wouldn't move/change

I know about the animations contained within glTF. These do require computation every frame to make the animations work. The standard glTF loader for THREE.js uses the Mixers (of THREE) to handle the frame-by-frame computations necessary for the animations. Are frame-by-frame computations going to be required?

Perhaps it is sufficient to have a static 3D model with a single
static
camera view, static lights, and static texture but that is capable of
being rendered stereoscopically. Until that is determined, I think it
is a bit premature to work on limitations of a particular format.

That's why it would be good to know what can be done with the format. Also, afaik favicon can't be observed by the document so things like timing attacks wouldn't work

Isn't that the backwards order? The required (or desired) capabilities should be first determined (with possible formats/technologies in mind), then determine which format/technology is best suited. Choosing the technology first gets everyone stuck just looking at a single line of capabilities. It also puts plans in a really bad position if the technology changes during specification and initial development.

-- Leonard Daly 3D Systems Architect & Cloud Consultant President, Daly Realism - /Creating the Future/

[cwilso]

We should definitely put an example in the note that covers Web App Manifest usage.

[NellWaliczek]

https://www.w3.org/TR/appmanifest/#icons-member

[dontcallmedom]

one thing that might be worth addressing in our guidance is what to (if anything) with the optional sizes attribute (both in the link element and in the app manifest)

[cwilso]

May want to force scaling on icons.

[cwilso]

Windows MR guidelines: https://docs.microsoft.com/en-us/windows/mixed-reality/creating-3d-models-for-use-in-the-windows-mixed-reality-home

[NellWaliczek]

https://html.spec.whatwg.org/multipage/links.html#rel-icon

[thetuvix]

Some other notes after the discussion:

• We should be consistent on what the origin means (is it on the ground or at the center of the icon?)
• Platforms like Windows MR allow users to resize the icon in the Home environment after placing. For the idea we discussed of supporting multiple appearances for a 3D icon based on spatial size, we should also consider just having the single .glb contain the various appearances. In a similar way, the Windows MR native 3D icon .gltf format uses an LOD extension to include multiple LODs in the same icon file. (which we may separately want to consider supporting here)

[thetuvix]

• Windows MR 3D icon design guidance: https://docs.microsoft.com/en-us/windows/mixed-reality/3d-app-launcher-design-guidance
• Windows MR 3D icon technical requirements: https://docs.microsoft.com/en-us/windows/mixed-reality/creating-3d-models-for-use-in-the-windows-mixed-reality-home

[thmignon]

Yup @thetuvix and @NellWaliczek the LOD extension we use could be used to support different appearances for a single object. We currently use a screen coverage value (what percentage of the screen the object is covering) to switch between LOD levels for an object. So rather than distance from the objects its more about how big it is in your current FOV. I imagine other runtimes could switch based on distance but it would be nice of course to align.

[rcabanier]

@thmignon Would that not look strange? This means that icons will change appearance depending on the distance from the observer.

[thmignon]

I'm mainly commenting on @thetuvix's notes from their discussion which said this might be desirable - "...discussed of supporting multiple appearances for a 3D icon based on spatial size". But yes depending on your experience you'll either use LODs to simply down res the tri count and texture resolution of the object and provide more headroom for others in the scene, or you can use LODs to for example remove text that would otherwise be unreadable at a distance. It depends on what you're conveying with the object and whether your experience is going for realism or legibility.

[thetuvix]

For the "spatial size" apperance tweaks, I was distinguishing two things:

• The angular size of the 3D icon in the user's view (for which LOD-style adjustments make sense)
• The physical size of the 3D icon in meters in the world, when the user explicitly resizes the 3D icon in the world using a drag handle.

I could see an app using a glTF LOD extension to accomplish the former, generally keeping the same logical appearance across LODs. There might then be a separate "physical size" extension to allow for similar customization based on the resulting spatial scale of the model in meters - this latter extension would perhaps be more appropriate for more drastic changes in appearance, such as reducing a full avatar to a torso or then to a head.

[kenzkiran]

@thetuvix w/ "origin". It made sense for us to always consider it to be the center of the model. It would make alignments and any animations (mostly rotation on any axis) look correct. I did take a look at LOD and it is pretty interesting. We will talk to our platform folks and get back to you this. We were considering physical size with models represented as cubic centimeters or cubic millimeters. The example you quoted with avatar head or torso would depend on where the icon would be used and this is what Ada suggested in TPAC.

Another point to consider is this: Since we are going to suggest some constraints w/ Size and or Polygon count, and considering these were for favicons and would be pretty restrictive, would you still consider LOD extension a better choice ?

[TrevorFSmith]

I received feedback that it's better to keep proposal Issues in here for search purposes, so I'm moving them back until we can think through how to handle it.

Rik, when you make the README for the repo, please add a link to this Issue.

[jgwinner]

Ah, this is closed, is this the right place to provide feedback on Ravi's md file?

https://github.com/immersive-web/spatial-favicons/blob/master/explainer.md

It looks great, but I have a few questions:

1. Why millimeters? WebVR is presumably in meters. Seems odd to have the two files use different units.
2. With animations, the phrase used is "Animations will not be supported, if animations are specificed[sic] user agents will ignore and render the asset in the bind pose." I would recommend: Animations need not be supported. If the browser does not support animations (or the browser, whatever the agent, disables them), then user agents will ignore and render the asset in the frame 1 pose.

When we say "bind pose" I think of T poses, not necessarily the 'start' pose. Not sure about this though, thought I'd throw it out.

I just don't like the idea of telling browsers "you will not use animations". Mind you, the idea of my bookmarks world having stuff flying all over the place might be a bit intense. Also, animations bring in a host of issues - what happens if the bookmark walks across the street and defaces another favicon? So we should also mention that animations should not involve translations or intrusions past the 'sizes' attributes.

[raviramachandra]

@jgwinner I think we can move this feedback to the the spatial-favicon repo.