Enable serving WebP/AVIF with JPEG fallback, for more complete browser support.
A good initial test case for this feature would be generating WebP images and JPEG images on upload, then using the picture element to specify the WebP with JPEG as the fallback. Although this scenario isn't likely to be that useful, it will set the path for the same approach with other formats.
Some additional details about my findings and the approach:
Use cases for implementing the <picture> element
The picture element makes sense when the image has more than one mime type, or when the upload type does not match the original type and the user wants to provide a fallback. Picture element could also be used for art directed images.
Use Case 1: uploaded type converted for sub sizes (one mime type, different than original)
Description: User uploads JPEG, sub sized images are WebP. For maximum compatibility, user wants to display WebP in a picture element, falling back to the uploaded JPEG for browsers that don’t support WebP.
Description: At some point in the future when a user uploads JPEG images, WordPress will be capable of converting these to AVIF images (AVIF support was added in PHP 8.1). To get the best performance and widest compatibility, the user wants to display a picture element showing first the AVIF format to supporting browsers, then falling back to JPEG images for other browsers.
Description: In this case the user wants to display a picture element with src sets for AVIF, falling back to WEBP when AVIF isn’t supported, and finally a fallback to JPEG when WEBP isn’t supported.
When users upload an image to WordPress, the system generates sub sized images for front end display (a sub sized image is created for each available size smaller than the uploaded image, in the same mime type format as the upload). Data about the original image as well as each sub-sized image created is stored in post meta in an array with the key _wp_attachment_metadata. This meta data includes an array keyed on sizes that contains the sub-sized image data, including the file, width, height and mime type. In the future the sub sized images might not be in the same mime format as the original (as is the case when a JPEG is uploaded and WebP is output as the default), or may contain multiple mime type sub sized images, for example AVIF and JPEG images.
The default data for a typical image would look like this after uploading(image_meta truncated):
WordPress automatically iterates through the post content and adds srcset attributes for images placed from the media library (images placed by URL don't work currently). It uses a regex to find media-library images, then calls wp_get_attachment_image to get each image source. wp_get_attachment_image in turn calls wp_get_attachment_image_srcset to generate the srcset.
The image source code for a typical image looks like this (line breaks for clarity):
The wp_calculate_image_srcset function, includes code to handle images changed by edits in the media library where each edited file name gets an edit hash appended (eg. “-e1640909072390”) to the filename. Sub sized images of the edited image get the same appended hash so they can be matched to the edited image.
For example if you rotate an image using the media library tools, the edit action creates a series of new files with hash extensions, and the post meta ‘sizes’ array is updated to reference the new files. The edited file urls will then be used for srcset generation. Note that the original_image meta data does not change, and the original image is never edited or removed, WordPress never changes your uploaded image!
Note that cropping an image directly in the block editor works differently - a new image is created in the media library with the name “-cropped” and cropped dimensions, all sub-sizes are re-generated for that image.
Enabling additional mime types, a cookbook
How can we enable additional mime types in WordPress to support uses cases 2 & 3?
For example, in use case 3, WordPress would create sub sized images in several formats when users upload images (in any supported format). Sub sized images would be created in AVIF, WEBP and JPEG formats.
Proposed approach
Internally, the sizes array keys can be any string, so when creating additional images beyond the first mime type - for each mime type we can prepend the mime type to the key names, so a WebP medium image would have the key images/webp-medium.
wp_calculate_image_srcset would be updated to accept an optional mime type.
When a mime type is not included, the function would work as is, except it would filter out any additional mime type image sizes from the meta data.
The function wp_get_attachment_image would be updated to automatically output a picture element when alternate mime type sub sized images are found in the images post meta.
Mime type order and availability for picture element support would be filterable on a per-image basis.
When the original image doesn’t match the sub sized images, it could be used as a fallback
When multiple mime types are provided, they can be used as source elements for the picture element with the final type used for the fallback image srcset.
Code must also consider image edits
As described above we need to ensure edited images are handled properly.
Filters should be enabled so developers can adjust each part of the output, especially the source type, srsset and sizes elements. Existing filters should work to enable developer fine tuning, we should validate that and ensure all context is passed (eg mime type).
Consider whether to require theme support of html5 eg with current_theme_supports( 'html5' )
For the purposes of the plugin we can test approaches by filter the post content.
New DOM element node may break theme or custom CSS selectors. How are you going to fix it?
Good point, we probably want to let themes opt into the behavior so they can adjust CSS accordingly. We could make opting in as simple as checking current_theme_supports( 'picture' ) (as suggested on the trac ticket). Maybe with core image blocks we can improve that by adjusting CSS?
Does element CSS class/style applies only to or to other tags or to tag?
Good question that needs more research; I'm guessing you would need to apply to the picture or both the img and picture elements, the element displayed may depend on browser support for the image format or for the picture element itself.
Chrome only seems to use responsive images correctly when sizes are provided as part of the media query (first example), the second/third examples use srcset with sizes, specifying the source with the type attribute and work in firefox and safari, in chrome dev the responsive image fail to work with this approach, maybe I have something incorrect in my markup? Appreciate any help here!
I created jpeg, webp and avif versions of images at various sizes.
I added a "WebP" tag to the WebP images so they are easy to identify. I don't have an editor that works with AVIF to add the tags there.
New DOM element node may break theme or custom CSS selectors. How are you going to fix it?
Good point, this has been raised before. One approach would be an opt-in approach for the progressive enhancement, themes would declare support for 'picture-element' to get the auto-feature for multiple mime types. Alternately, any theme or plugin could call wp_get_attachment_image with a parameter indicating they want a picture element.
Our main initial use case for this will be images with multiple mime types (for example AVIF with jpeg fallback), however we need to ensure other use cases (for example varying density or art direction) are also covered by the approach we choose.
Does element CSS class/style applies only to or to other tags or to tag?
I'm not sure, this needs more investigation and likely a developer note to show how to properly apply css when using the picture element.
How about 'img' element and loading=lazy attribute? Does it apply to picture-source elements?
Yes it does, according to this it is sufficient to set the attribute on the contained img tag.
@adamsilverstein is this something you are continuing to work on once back from the break? If so, could we assign the ticket to you please?
I have self assigned, however this is a large task and could use several developers working on it, I will think about how we can break out smaller tasks.
The selected image is then presented in the space occupied by the element.
So the <img> node remains the placeholder for the image as chosen by the browser from the available sources, meaning CSS should continue to target <img> and not <picture> or <source>?
@adamsilverstein @jjgrainger updating this issue to be a [Type] Overview as it will act as our main "epic" issue, and we will create sub-issues once we have finalised an approach. Does that work for you?
In relation to supporting the picture element in core, I wonder how many WordPress sites already use it? I'm sure some plugins or themes must already use the picture element.
Maybe @rviscomi can help here to build a query similar to the one we worked on for WebP usage by WordPress version [script], this time looking for picture element use instead. This would let us track adoption once/if we introduce the feature in core.
Query
# ⚠️ This query processes 1.4 TB
```sql
WITH picture AS (
SELECT
url
FROM
`httparchive.pages.2022_01_01_mobile`
WHERE
JSON_VALUE(JSON_VALUE(payload, '$._element_count'), '$.picture') IS NOT NULL
), wordpress AS (
SELECT DISTINCT
url
FROM
`httparchive.technologies.2022_01_01_mobile`
WHERE
app = 'WordPress'
), total AS (
SELECT
COUNT(0) AS total
FROM
wordpress
)
SELECT
COUNT(0) AS wp_picture,
(SELECT * FROM total) AS total,
COUNT(0) / (SELECT * FROM total) AS pct_wp_picture
FROM
picture
JOIN
wordpress
USING
(url)
```
## Results
```
wp_picture total pct_wp_picture
129,755 2,648,818 0.0490
```
Chrome only seems to use responsive images correctly when sizes are provided as part of the media query (first example), the second/third examples use srcset with sizes, specifying the source with the type attribute and work in firefox and safari, in chrome dev the responsive image fail to work with this approach, maybe I have something incorrect in my markup? Appreciate any help here!
@adamsilverstein I will take a look at this for you!
I created jpeg, webp and avif versions of images at various sizes.
I added a "WebP" tag to the WebP images so they are easy to identify. I don't have an editor that works with AVIF to add the tags there.
Hi folks! I've been working on a high-level proposal for images along with @adamsilverstein - Im hoping it covers some, if not all of the topics brought up in this thread. That being said we are most definitely looking for community feedback, insight and input. The <picture> element is a complex beast so we should hammer down as much detail as we can.
Summary
The proposal (attached below) provides high-level insight into the behaviour and possible implementation of the <picture> element into WordPress. There are some interesting challenges that I've uncovered:
How do we ensure this functionality is backwards compatible?
Styling of <picture> elements could lead to regression in themes
I've included markup examples as well as cross-browser tests, by far Safari is the weakest link in the chain.
I plan to add a section that looks at how other plugins handle <picture> element support cc @adamsilverstein
I've also included important core functions that could be extended.
One of the biggest drawbacks, with lots of questions is the ability to serve appropriately sized, art-directed images across device breakpoints - I don't see how we could do this easily at this point, but its something to at least think about.
Even here mentions using srcset for resolution/bandwidth cases.
Here's a use real use case I run into and wondering if picture element would help with this:
I work with a marketer who provides me images so I can upload them as a featured images for articles. The single article template has a blown up big version of the featured image while the recent news block or archives page template shows multiple articles with thumbnail sized images. Sometimes they are cut off/cropped on smaller screen sizes. Would using picture element help in these cases?
Also I'm wondering if anyone knows why the picture element is so underused, was it due to IE11? I checked some random sites to see if they use picture and none of them are using it:
@LukaszJaro - I think the biggest issue with the <picture> element in WP at the moment is the fact that there is no UI to enable consistent art-direction. The element itself was part of the new HTML5 spec, but seems to have lost a fair amount of traction lately. Im not too sure why that is, but it could just be because of how complex images especially images that now need to adapt to different devices behave.
The picture element would essentially allow you to create as many crops as you like that would meet different conditions. You could have portrait on mobile and landscape on desktop, without the need to feature detect or show/hide images with CSS. The browser makes the call based on the conditions provided in <source> - it is very powerful. But again we may also run into a situation where 1 image, could have 10 possible versions (considering the market of devices we're dealing with)
On reference to the article - I think its sound. WP already attempts to solve responsive images using srcset and sizes - but there is no space to art direct the images on different devices or screen widths. This is because srcset on an <img /> tag is only responsible for serving the "correctly" sized image based on a condition or width.
We've been using <picture> tags for many years now on dozens of sites and it has been great. For legacy browsers we used to include a common set of JS to boot up support for HTML5 features in general, including the <picture> tag, but we haven't included that in several years.
I've never been a fan of the srcset attribute on <img> for responsive, so we just stuck to media queries if needed, and it has worked without any issues as far as I remember, Worst case, someone gets the <img> tag.
For the most part we use this as an upgrade to WebP for users, but we occasionally use it for art direction and then almost always to crop hero images.
I think the hardest leap for people from a styling perspective is that they need to consider the <picture> tag as basically a wrapper <div> around an image. So switching from just an <img> to one wrapped in a <picture> tag needs to be treated the same. Sometimes it isn't a problem, but if you are in a grid or flex context, the <img> might not behave how you initially expected, because the container might be getting partially styled instead.
I just checked and Gutenberg is [explicitly selecting](https://github.com/WordPress/gutenberg/blob/8dc36f6d30cc163671bdaa33f0656fdfe91f1447/packages/block-library/src/image/style.scss#L1-L7) the `img` tag in the Image block:
```scss
.wp-block-image {
img {
height: auto;
max-width: 100%;
vertical-align: bottom;
box-sizing: border-box;
}
```
This is done commonly in Gutenberg:
https://github.com/search?q=repo%3AWordPress%2Fgutenberg+%2Fimg+%7B%2F+language%3ASCSS&type=code
https://github.com/search?q=repo%3AWordPress%2Fgutenberg+%2Fimg%2C%2F+language%3ASCSS&type=code
This is also super common in themes generally: https://wpdirectory.net/search/01HX83JEWYSX880A2CN2W8G5GA
This was a big headache for the AMP plugin when we converted `img` to `amp-img`, as we had to rewrite all of the selectors mentioning `img` to use `amp-img`. It is possible, but only if you are processing all the CSS.
adamsilverstein
commented
2 years ago 
ddur
commented
2 years ago 
eclarke1
commented
2 years ago 
futtta
commented
2 years ago 
rviscomi
commented
2 years ago 
dainemawer
commented
2 years ago 
LukaszJaro
commented
2 years ago 
FrankGalligan
commented
2 years ago 
cjhaas
commented
2 years ago 
westonruter
commented
4 months ago
Current in progress PR: https://github.com/WordPress/performance/pull/73
Related trac ticket: https://core.trac.wordpress.org/ticket/42920
See this comment for a detailed description: https://github.com/WordPress/performance/issues/21#issuecomment-1003404685
Original description:
Use picture element to enable modern formats like AVIF with fallback to more widely supported format like WebP or JPEG, including responsive srcset markup like current images. Issue - https://github.com/adamsilverstein/modern-images-wp/issues/1 . references: The anatomy of responsive images - JakeArchibald.com & markup: I just want an image on my page
A good initial test case for this feature would be generating WebP images and JPEG images on upload, then using the picture element to specify the WebP with JPEG as the fallback. Although this scenario isn't likely to be that useful, it will set the path for the same approach with other formats.