Require `img` to be able to load the same video formats as `video` supports

[zcorpan]

Related to https://github.com/whatwg/html/issues/6363

Regardless of what changes are made or not made to video, the question of which formats img should support remains. That's this issue.

WebKit supports video formats in img, while Gecko and Chromium do not.

In https://bugs.chromium.org/p/chromium/issues/detail?id=791658 it was requested that Chromium support MP4/h.264 in img. The issue was closed as WontFix:

Closing as WontFix per c#35, due to the following:

• The widespread adoption of WebP (addresses CDN use case)
• Forthcoming AV1 based image formats (ditto).
• Memory inefficiency with allowing arbitrary video in image.
• Most sites have already switched to <video muted> now that autoplay is allowed.

For the last point, video today doesn't have the same capabilities as img (e.g. srcset, media).

For AVIF, I asked on Twitter if there are reasons why it would still make sense to support other video formats assuming img can load AVIF: https://twitter.com/zcorpan/status/1428459457407864836

Is animated AVIF good enough for the "autoplaying muted looping short video" use case? That is, is making <img> support video formats playable by <video> unnecessary given AVIF?

• @jonsneyers

  From a browser engineering pov: if you allow one video codec in an img tag, why not just allow all video codecs you support? From a web dev workflow pov: why require wrapping your AV1 in AVIF? Would be easier if you can use existing video containers, as they have better tooling.

  From a devops pov: AV1 encoding is computationally expensive, and h264 or vp9 would already give most of the bandwidth savings (compared to gif/awebp/apng). It's likely in many use cases a better trade-off to produce h264 than av1 atm.

  From a future-proofing pov: when AV2 arrives, will we still be stuck with AVIF in the img tag, or will we have to wait for AVIF2 to be defined? It would make more sense to allow AV2 in an img tag as soon as it's allowed in a video tag.

  From a web perf pov: AV1 decoding is computationally relatively heavy on low-end devices without hw support, compared to h264 decoding. Time to render is what matters, not just bandwidth. Also battery life is a consideration.

  From a legacy pov: many existing "GIFs" are in fact already h264 (on sites like Tenor or Giphy). Making them work in an img tag without needing to transcode to av1 would be convenient and would avoid generation loss.

  From a compression density pov: yes, AVIF is good enough (atm) since it is the best video codec (currently) available. If all browsers would support (animated) AVIF in an img tag, it would be a major step forward. But having alternative codecs also work would be even better.

• @kornelski

  I think treating AV1 in MP4 with "I'm an image" tag differently from AV1 in MP4 with "I'm a video" tag only shows how arbitrary and nonsensical the whole concept of "animation" is.

  AV1 encoding is painfully slow. Order of magnitude slower than anything else. This makes on-the-fly gif2avif conversion expensive to perform. gif2h264 is relatively easy. gif2vp9 doable.

• @jernoble

  IMHO, no. One of the benefits of allowing video-in-an-img is the dramatic reduction in power use due to broad hardware decoder availability. That benefit is largely absent for AVIF content on existing hardware.

  @zcorpan replied

  Is MP4 still the format with wide hardware decoder availability, or are there others?

  @othermaciej replied

  MP4 (really H.264/AVC) likely has the widest hardware decoder availability. VP8 and VP9 are also update. Neither H.265/HEVC nor AV1 have broad hardware support yet.

[zcorpan]

cc @whatwg/media

[kornelski]

As someone who works for CDN with animation conversion feature: animated WebP is bad. We would prefer not to use it.

The animated WebP format has been designed to be conceptually similar to GIF, and work for the same use-cases that GIF does, but this made it inherit GIF's weaknesses. Content that makes GIF sizes balloon typically also compresses poorly in WebP. Both are especially inefficient for short video clips with camera motion — a use-case handled better even by ancient codecs like MPEG-1. WebP often fails to meaningfully reduce GIF file sizes, and can even end up making files larger than GIF.

[kornelski]

@eeeps on why GIF is still used:

https://twitter.com/etportis/status/1442838357432700932

Many, many contexts that aren't built to accept video files and loop/mute/autoplay them, without displaying controls. e.g.: most CMSes, messaging platforms, and the <img> tag.

https://twitter.com/etportis/status/1443218535774253060

One perf use case for GIF over <video> is (I think) speculative preloading. Last I checked (a while ago...) a tiny GIF could render/autoplay faster than a tiny <video>.

[colinbendell]

From an end-user perspective: the human doesn't really know the difference between a gif and an mp4 except perhaps the length of play. They are all 'gifs' to most humans.

[colinbendell]

A few considerations that should be defined:

• should videos in img tags be assumed to be autoplayed ?
• should videos in img tags be assumed to be muted?
• should videos in img tags be assumed to be looped? (IIRC, only MOV has a flag for looping)
• are there reasonableness test that the UA could impose on an <img src=mp4> such as length of play? (eg: only auto play and loop a video up to 60s because of memory and bandwidth considerations)
• should the Accept: video/* header be a required as a signal from the UA? (will this require coordination with the IETF?)
• should the UA avoid range requests like most <video src=mp4 when requesting a resource that contains a file extension of a known video format?
• should the UA support streaming formats like HLS, DASH, CMAF? (currently Safari does)
• should the preloader parse and continue to preload from manifests if triggered from an <img src=hls?
• what are the equivalent dimX, dimY, resX, resY EXIF attributes in an ISOBMFF for intrinsic size calculations?
• what are the considerations for the use of thumbnail boxes in most ISOBMFF?
• are there general guidances or considerations for content providers the serve ISOBMFF based video formats? (eg: box organization, use of thumbnails, use of FourCC, etc)

[Yay295]

should the UA support streaming formats

gif's can technically be streamed, so I think video streams should be allowed as well.

what are the equivalent dimX, dimY, resX, resY EXIF attributes

This would be hard to set, because I know it's possible with with at least VP8 to actually change the size of the video with each frame.

[jonsneyers]

A concern I have about the current Chrome approach of allowing AV1 wrapped in AVIF in an img tag, but not AV1 wrapped in any other container, is the following:

What about Nokia's patent claims on HEIF? AV1 is designed to be royalty-free, but the HEIF container that AVIF uses may not actually be royalty-free — at least Nokia does not give a patent grant, except for non-commercial purposes. See also: https://news.ycombinator.com/item?id=19874121

If we could use AV1 in any video container (in particular, in a non-patent-encumbered one), and not be forced to wrap it specifically in AVIF (which is based on HEIF), I would consider that a lot safer regarding potential patent trolling when adoption becomes larger and Nokia smells money. It would make perfect sense for them to lay low now and wait for more adoption before they start demanding royalties. These patents will only expire around 2035-2036, so they could be a problem.

[Sheraff]

For accessibility purposes, there are probably scenarios where you'll have to offer a way to pause the video. Will <img> tags now have a JS .pause() method?

From WCAG 2.1, potential friction point if an image can't be paused: 2.2.2 Pause, Stop, Hide.

[kornelski]

@Sheraff This problem already exists for GIF. The difference is not in user-visible behavior (which I agree should be improved!), but in allowing smaller file sizes.

[Sheraff]

@kornelski this solution is being debated as

• an improvement over GIFs, in which case, we already have accessibility issues, and file size improvements would be better than nothing
• but also as a crutch for the removal of the media attribute in video <source> tags (which still exists in the picture <source> tags), in which case we would be losing accessibility features

[jonsneyers]

• but also as a crutch for the removal of the media attribute in video <source> tags (which still exists in the picture <source> tags), in which case we would be losing accessibility features

Why would <img> being able to load the same video formats as <video> be an argument to not have the media attribute in video <source> tags? I think it's clear that the video tag is still needed for actual video (with audio, typically not looping, with controls, etc), and picture/img will not make it redundant.

So I think all of the following would be good, and they are imo orthogonal issues:

• <img> being able to load the same video formats as <video>
• add a .pause() method to images (also for GIF/APNG/animated WebP), and possibly change UA behavior to allow pausing animated images
• have a media attribute in video <source> tags

[Sheraff]

@jonsneyers the media attribute has already been removed from video <source> tags. So at this point, we have no clean way of doing responsive videos for above-the-fold content. If <picture> starts to support video content, developers are going to flock to this solution for the use-case of responsive above-the-fold videos.

So in my opinion, this change should either land with accessibility in mind, or land at the same time as some changes to the <video> tag that solve the responsive above-the-fold videos use-case.

[zcorpan]

This issue is not about the media attribute on video's source element. Continue that discussion in https://github.com/whatwg/html/issues/6363 please.

[zcorpan]

@Sheraff can you file a new issue about the ability to pause animated images in img?