Allow an extra dimension to the size array for 3D favicon

[rcabanier]

With the advent of VR and AR devices, we have browsers that can live in a 3 dimensional virtual space. To enhance the experience, we want to give sites the ability to specify 3D models instead of flat images for their favicons.

We created an explainer here and discussed it with the web manifest group (see this issue).

@zcorpan What do we need to do to get an extra value in the sizes array?

[zcorpan]

The explainer says that the unit is millimeters. I think having this change of units between 2d and 3d is confusing, if the unit is omitted.

The explainer says download size (in bytes) is a consideration that goes into the decision of which resource to load, where it's assumed that the resource that is smaller in square millimetres is also smaller in download size. Is this a good assumption?

The other consideration is the "art direction" use case, where you want e.g. just the dino's head in the list of bookmarks, but the full dino when showing the currently open tab. How many different variants like this do we expect developers will provide? Is the size in millimetres the best way to indicate these differences?

Can the icon adapt itself based on its "size", similar to how CSS can be responsive to the viewport size? (In principle, SVG favicons can be responsive using CSS @media.)

There is currently an any keyword, indicating 2D any-size icon. Should there be a 3d-any?

[raviramachandra]

The explainer says that the unit is millimeters. I think having this change of units between 2d and 3d is confusing, if the unit is omitted.

As per spec, the 'sizes' for Image Resource is absolute pixels and not CSS pixels. The closest equivalent to this unit in 3D is metric units. glTF uses 'meters', but icons are small in size and hence to keep the units in 'integers' we use 'mm'. We should explicitly mention this in new Image Resource github.

The explainer says download size (in bytes) is a consideration that goes into the decision of which resource to load, where it's assumed that the resource that is smaller in square millimetres is also smaller in download size. Is this a good assumption?

Unfortunately, it is not, but there are alternate proposals which use 'vertex count' x 'primitives count' which may be better in quality and size estimation, but that will mean we have a new definition for 'sizes'. It confuses even more. Most 3D assets dimensions are just in reference units, platforms may treat 1 unit as 1 meter or customize the o/p scale as needed.

The other consideration is the "art direction" use case, where you want e.g. just the dino's head in the list of bookmarks, but the full dino when showing the currently open tab. How many different variants like this do we expect developers will provide? Is the size in millimetres the best way to indicate these differences?

Can the icon adapt itself based on its "size", similar to how CSS can be responsive to the viewport size? (In principle, SVG favicons can be responsive using CSS @media.)

The other consideration is the "art direction" use case, where you want e.g. just the dino's head in the list of bookmarks, but the full dino when showing the currently open tab. How many different variants like this do we expect developers will provide? Is the size in millimetres the best way to indicate these differences?

There are extensions that can help in achieving these in a single file format: MSFT's LoD. But for the first draft, we would like to avoid custom extensions. Short answer, no we can't "adapt" using only one file. IMHO, developers who extensively develop for XR devices can be mindful and provide multiple variants and UA's can appropriately select the variant most appropriate based on hint provided by 'sizes'. This specific instance of 'art direction' was added with the view that XR devices can range from a mobile phone to a dedicated headset like Magicleap One. The decision to provide multiple variants/or not is left to developers.

Can the icon adapt itself based on its "size", similar to how CSS can be responsive to the viewport size? (In principle, SVG favicons can be responsive using CSS @media.)

From the spec: 'any' is applicable for SVG or scalable icons. 'any' can be applicable to 3D favicons as well. Platforms can always scale the 3D asset, but can't ensure same quality is maintained at all scales. The asset may or may not degrade with scaling.

There is currently an any keyword, indicating 2D any-size icon. Should there be a 3d-any?

Not really, we can continue to use 'any' w/ 3D icons too in the same way we treat 2D icons.

[raviramachandra]

@zcorpan were you able to review our explanation above ? Do you see any other concerns.

[zcorpan]

Sorry for the delay.

As per spec, the 'sizes' for Image Resource is absolute pixels and not CSS pixels. The closest equivalent to this unit in 3D is metric units. glTF uses 'meters', but icons are small in size and hence to keep the units in 'integers' we use 'mm'. We should explicitly mention this in new Image Resource github.

OK. The format could be changed to allow floats. Would you use meters or mm if floats are allowed?

Unfortunately, it is not, but there are alternate proposals which use 'vertex count' x 'primitives count' which may be better in quality and size estimation, but that will mean we have a new definition for 'sizes'. It confuses even more. Most 3D assets dimensions are just in reference units, platforms may treat 1 unit as 1 meter or customize the o/p scale as needed.

OK. I take it that UAs shouldn't factor in bandwidth constraints when deciding which resource to use.

IMHO, developers who extensively develop for XR devices can be mindful and provide multiple variants and UA's can appropriately select the variant most appropriate based on hint provided by 'sizes'. This specific instance of 'art direction' was added with the view that XR devices can range from a mobile phone to a dedicated headset like Magicleap One. The decision to provide multiple variants/or not is left to developers.

How commonly is this desired? How should the UA choose which size to use?

If it's very uncommon to want to provide multiple 3D icons, then it seems better to avoid the complexity and only support providing one variant.

Not really, we can continue to use 'any' w/ 3D icons too in the same way we treat 2D icons.

My thinking was that developers may want to provide both SVG 2D icon and a 3D icon, so both would use 'any'. The type attribute can be used to distinguish them, though.

To the question

What do we need to do to get an extra value in the sizes array?

See https://whatwg.org/working-mode#additions

Who are interested in supporting 3D favicons?

[raviramachandra]

The proposal suggested we use 'mm' since we wanted to use existing 'integer' only in 'sizes'. If float is allowed, we might be ok. Considering these will be mostly used as a 'tab' indicator or a 'bookmark' icon by UA, icons are likely be sub meter most of the times. Hence we chose to use 'mm'. Let me check with the immersive-web group and get back.

OK. The format could be changed to allow floats. Would you use meters or mm if floats are allowed?

Could be. But the selection from UA will likely be driven by where this 'icon' is being used. On our platform, we tend to pick up the biggest icon for use cases such as "Shortcut on Desktop" and use our the smallest version for "bookmarks" and "history" list representations. (Because they can run into 100s and with more complex 3D icons, they tend to slow the UX)

OK. I take it that UAs shouldn't factor in bandwidth constraints when deciding which resource to use.

Based on our discussions with immersive-web group, this was very much a desired feature.

How commonly is this desired? How should the UA choose which size to use?

I don't know of other implementation apart from Magic Leap or Holo Lens who use this. (may be Oculus browser). This is pretty new and it is difficult to gauge by usage.

If it's very uncommon to want to provide multiple 3D icons, then it seems better to avoid the complexity and only support providing one variant.

Yes. I need to figure out how other UAs would handle this though. On our platform though, we prioritize 3D icon over 2D based on mime-type or resource 'extension'.

My thinking was that developers may want to provide both SVG 2D icon and a 3D icon, so both would use 'any'. The type attribute can be used to distinguish them, though.

Will look into this.

See https://whatwg.org/working-mode#additions

In principal, everyone in the immersive-web group.

Who are interested in supporting 3D favicons?

[aarongustafson]

I wanted to bump this thread as it was recently bumped on the ImageResource repo as well. Any progress here?