Embedded preferred-image-rendering metadata

Introduction

Raster images can be rescaled (in particular upscaled) in various ways. The image-rendering CSS property can be used to control this. For photographic images, generally some kind of smooth interpolation is preferable, while for pixel art, generally nearest-neighbor upsampling is preferable. Which type of rescaling algorithm to use is in reality a property that is tied to the image content, and controlling it externally through CSS is not convenient. In this sense, the property is somewhat similar to image orientation: it is a property of the image itself — which is why for orientation, the CSS property image-orientation can even no longer be used to adjust the orientation of an image, and embedded Exif metadata is used instead. For image-rendering though, there currently is no way to embed this information in an image file itself, and the image-rendering CSS property is the only way to control the choice of rescaling method.

Use Cases

Pixel art

When sharing pixel art images, the artist wants to ensure that the image will be viewed in the intended way, i.e. with image-rendering: pixelated. If they are sharing the image on their own website, they can do that using CSS. However this information gets lost when the image gets downloaded and cannot be conveyed when uploading the image to e.g. social media or a typical CMS.

To circumvent this problem, artists have resorted to various 'tricks', such as:

Manually upsampling the image with nearest neighbor, turning every pixel into e.g. a block of 10x10 pixels, to reduce the impact of the rescaling algorithm choice.
Vectorizing the pixel art into an SVG where every pixel becomes a square.

These methods to circumvent the problem are obviously bad for both compression and performance.

High-density displays

Displays with very high densities are becoming increasingly popular, e.g. DPR 3 or even higher. While in the past the rule of thumb was that images should be at least as large as the actual display size they will be rendered at (so the browser effectively only has to downscale, not upscale), so they don't look blurry, this is now no longer the case: when displays densities are exceeding 'retina' resolution, it makes sense to send images at a lower resolution (say DPR 2) and to rely on browser upscaling. This makes it more important than in the past to have convenient control over the way the upscaling is done.

Pinch-to-zoom

On mobile devices like phones, manually zooming in on an image is a commonly used action. This typically causes the browser to upscale the image. Having better image-specific control over the way this is done can help to ensure optimal viewing conditions.

Goal

Have a way to embed the preferred image-rendering choice in the image file itself, so the intended rendering will be used automatically and by default, rather than requiring custom CSS.

Proposed Solution

The most straightforward solution would be to add or repurpose an Exif or XMP metadata field to store the preferred-image-rendering information in a way that remains tied to the image and that works for any image format that supports Exif/XMP.

Browsers and image viewers would then inspect this field and use it in the case of image-rendering: auto. When image-rendering has a custom non-auto value, it would still override the preferred-image-rendering choice.

For an image that does not have this metadata, nothing changes compared to the current situation.

A modifier or-from-image could be added to the image-rendering CSS field to indicate that the setting only applies to images that do not have an embedded preferred-image-rendering choice, so for example image-rendering: smooth or-from-image would mean "use the smooth scaling algorithm by default, but if the image file itself indicates a different preferred-image-rendering, use that instead".

Examples

Alternate Approaches

An alternative approach is to set the image-rendering property individually on every image. This significantly complicates markup generation and does not solve the problem of keeping this information tied to the image file object, i.e. the information is lost when the image is downloaded or referenced from another page.

Privacy & Security Considerations

No considerable privacy or security concerns are expected, but we welcome community feedback.

Let’s Discuss

Some open questions for discussion:

How exactly to embed this metadata? Is there an existing Exif field we can use/repurpose for this?
Should the values for the preferred-image-rendering metadata simply correspond 1-to-1 to the values for the image-rendering CSS property, or should it be something else? For example, it could make sense to have different options for the case of downscaling than for the case of upscaling.
If this mechanism becomes available, do we even still need the image-rendering CSS property, or should it be deprecated/reduced in functionality, like image-orientation? (imo we still need it, since there might be cases where setting a global default or overriding the preferred-image-rendering is useful, unlike orientation where that doesn't make sense)

WICG / proposals