ramonjd
commented
1 month ago Firstly, thanks so much for the detail and all the background work!
Converting HEIC images to a web-safe format
This came up in a recent discussion I was having.
What's your take on the level of effort to integrate this feature? I was browsing the media experiments repo code and was wondering if there were any further considerations, beyond what you've explored already.
It would be a fantastic feature to have - happy to have a shot at it, or try to organize someone to have a shot at it 😄 if you think it's worth it at this stage. I'll take your guidance.
Thank you!
swissspidy
noisysocks
This is something that was previously mentioned in #55106 and has been heavily explored in Media Experiments already (see for example this blog post for an overview of the latter). I believe the core logic at the heart of Media Experiments is a great foundation to implement such a feature, and Gutenberg is the right place to further develop it and bring it to more people.
Overview
Current image processing in WordPress relies on server-side resources and older image libraries, leading to potential performance issues and limited support for modern image formats such as AVIF. This results in a subpar user experience, particularly with resource-intensive tasks like resizing and compressing images. Additionally, the lack of modern compression tools like MozJPEG further hinders optimization efforts.
Client-side media processing offers a solution by leveraging the browser's capabilities to handle tasks like image resizing and compression. This approach not only alleviates the strain on server resources but also enables the use of more advanced image formats and compression techniques, ultimately improving website performance and user experience. By tapping into technologies like WebAssembly, WordPress can provide a more efficient and seamless media handling process for both new and existing content.
Resources
Roadmap
Given the many possibilities client-side media processing unlocks, this project can be split into multiple phases, each with their dedicated goals and success criteria. For example:
Phase 1
Goals
Nice to have
Non-goals
Phase 2
Goals
Nice to have
Non-goals
Phase 3
Goals
Nice to have
Caveats and risks
Technical complexity
WASM-based image optimization requires
SharedArrayBuffersupport, which in turn requires cross-origin isolation. Implementing that in a robust way without breaking other parts of the editor is challenging. There are currently some known issues in Firefox and Safari due to these browsers not supportingcredentiallessiframe embeds. Embed previews in the editor currently do not work because of this, until those browsers add support for<iframe credentialless>.Image compression by itself is rather complex due to the vast amount of different combinations of codecs and encoding options available. While the existing server-side image handling in WordPress has been proven over many years, doing everything client-side is new territory. There will be many edge cases that need to be handled.
As for overall architecture design, the existing Media Experiments project contains a very solid foundation that already solves many challenging problems and could be built upon.
Confidence level
Reception in the community so far has been extremely positive, including from WordPress leadership. There is clearly a desire for this kind of solution.
The biggest risk is about compatibility with other parts of the editor (plugins, embed previews) and certain environments because of the cross-origin isolation required for
SharedArrayBuffer. Not breaking existing sites is very important. The risk can be mitigated through extensive testing, additional safeguards, and documentation. Lack of<iframe credentialless>support is not ideal as it degrades editor UX in some cases, but it does not break the authoring experience as a fallback exists.While image optimization could theoretically be done through other means such as the canvas API, said API is extremely limited across browsers. For example, there is a lack of support for compression options (e.g. lossless or lossy) and mime types (no AVIF everywhere, no WebP in Safari).