RFC: Enhance Image 1.0

[colepeters]

This RFC supersedes https://github.com/enhance-dev/enhance.dev/pull/115, as that proposal would require either WebKit to adopt customized builtins (which seems less and less likely to happen) or for us to ship a polyfill to WebKit clients (not ideal as server side output would be negatively impacted for those users, of which there are many).

Goals

• Make responsive images easy for end users to implement (using picture and source elements properly can be vexing given their fairly involved and nuanced API — same goes for img and srcset)
• Allow for multiple transformations and optimizations to be made from consumers' source images, while delivering great end user performance on a range of device capabilities and network speeds
• Achieve all of the above without client side rendering, polyfills, or nonstandard use of the platform; Enhance Image should function as a simplified abstraction of platform features in tandem with our own image transformation service

Core functionality

• Image transformation/optimization:
  • Resize (< source max dimensions)
  • Reformat (JPG, PNG, WEBP, AVIF)
  • Quality grading (for lossy formats)
  • Opt-in priority loading
  • Additional transformations (cropping/aspect ratio changes, colour mode, filters, etc) to be examined at a later date to keep component scope current to the existing image service (we do have cropping already, but I'm considering whether we should address it later as I'm not sure the current implementation w/ respect to the component API is ideal)
• Ability to request multiple variants of a single source image, scoped by media queries

Prior art

Enhance Image alpha

• 👍🏻 Enables all of the above
• 👎🏻 Component API is complex, easy to get wrong, and not scalable (this version was intended solely for internal use at Begin with these caveats in mind)
• 👎🏻 Currently introduces a substantial initial load time for uncached transformations of the source image which hits performance metrics hard; difficult to ship a builtin solution for this

next/image

• 👍🏻 Enables generation of multiple sizes scoped to media queries (recent addition via the sizes prop), setting of quality (one setting for all generated images), lazy loading
• 👍🏻 Offers generation of blurred placeholder, which can be customized by the consumer
• 👍🏻 Offers image preloading via <link rel='preload'>
• 👎🏻 Does not offer reformatting
• 👎🏻 Requires NextJS

Unpic Img

• 👍🏻 Automatically generates a batch of size variations for a given image and max width/height
• 👍🏻 Offers different layout modes (constrained, full width, fixed), priority and lazy loading
• 👍🏻 Delivers WEBP and AVIF if supported
• 👍🏻 Generates blurred placeholder + provides an option to pass your own base64 string for it
• 👎🏻 No native web component integration — frameworks only
• 👎🏻 No settings for quality or other transformations
• 🤔 Consumers need to bring their own image CDN or CMS

Proposal

The alpha version of Enhance Image was based on the picture element, where each variant would be rendered as a new source element within its picture parent. This presented some problems, primarily due to the aforementioned lack of full cross browser support for customized built in elements.

The version of Enhance Image proposed here is based on the use of the img element’s srcset and sizes attributes. Despite my initial dislike of these attributes’ APIs, they are in fact designed explicitly for serving different sizes of the same image, whereas picture and source are designed to solve problems of art direction (i.e., serving a completely different image under a given media condition). Using the srcset and sizes attributes gives the client a bit more control in choosing which image to serve, and can cut back on unnecessary network requests that would be unaddressed by the previous implementation of Enhance Image.

The main concern for Enhance Image, in this case, is simplifying the admittedly nuanced and potentially complex native API for responsive images. This ends up being quite feasible, as seen below.

Configuration

Enhance Image will generate a range of transformations for your images on sandbox start, so that these transformed images will be available to your application at runtime.

You can configure your image transformations as follows:

# .arc
@arc-image-plugin
widths
  2400
  1200
  800
format webp
quality 80

Configuration options:

widths

A list of widths in pixels, used to generate resized variants of your source image. Images will be resized with their aspect ratio preserved. Source images smaller than a given width will not be enlarged. Defaults to 2400, 1200 and 800px widths.

format

The image format to use for transformed images. Options are webp, avif, jpeg, png, and gif. Defaults to webp.

quality

Quality setting for lossy image formats, given as number between 0–100. Defaults to 80. This setting will not apply if using the png format.

Component API

<enhance-image
  src='/_public/pupper.jpg'
  sizes='(min-width: 48em) 50w, 100vw'
  width='5000'
  height='6250'
  priority
></enhance-image>

Attributes

src The path to the image to use as a base for transformations.

sizes A comma separated list of media conditions and source size values — just like the native img.sizes attribute. The final value in this list should not contain a media condition, as it will be used as the default when no given media conditions match. The client will use these values to determine the appropriate width of image to use for a given media condition. Note that source size values affect the intrinsic size of the image, but this can be easily overridden via CSS. Defaults to a single 100vw size, so the client will select the best image for use as a full width image on the current viewport.

width and height Unitless lengths for the image’s intrinsic width and height. These must be provided in order for the client to determine the image’s aspect ratio and thus prevent cumulative layout shift. (We can likely automate this for the user pretty easily; see roadmap below.)

priority Boolean property which will apply the fetchpriority=“high" attribute to the image. This attribute is still forthcoming for Firefox, Safari, and Opera. When this property is not declared, images will default to the loading=“lazy" attribute (and thus priority images in clients lacking fetchpriority will be treated the same as images with loading=“eager" declared). This is useful for images such as hero graphics that appear within the viewport’s initial scroll boundaries.

To solve for initial implementation

Image generation

Using the configuration in the .arc file, we need to pre-generate the transformed images on sandbox start (and possibly run a watcher for local dev?), to avoid the lag when transformed images aren’t already cached.

Post 1.0 roadmap (unsorted)

• Automatic insertion of image dimensions during SSR (potentially via image-size?)
• Handle placeholder images (default to inlined base64, and allow for passing of a custom placeholder)

[macdonst]

@colepeters

Enhance Image will generate a range of transformations for your images at deploy time, so that these transformed images will be available to your application at runtime.

But we aren't converting the images at deploy time right? It just makes a path available to you to request at run time. The first hit on the url would create the image?

[colepeters]

@macdonst We aren't yet… but we will be! 😈 (I mistyped here, it would be on sandbox start, but yeah.) We discussed this during yesterday's Enhance Sync, will loop you in!

[colepeters]

Moving forward with configuring generated image sizes, format, and quality in .arc file. arc-image-plugin will read this config + provides fallbacks, and write the resulting config to the consuming Enhance project. The Enhance Image component will consume that config via state.

For current status: https://github.com/orgs/enhance-dev/projects/13/views/1