search

WordPress / gutenberg

The Block Editor project for WordPress and beyond. Plugin is available from the official repository.
https://wordpress.org/gutenberg/
Other
10.49k stars 4.19k forks source link

Tracking: Client-side media processing #61447

Open swissspidy opened 6 months ago

swissspidy commented 6 months ago

This is something that was previously mentioned in #55106 and has been heavily explored in Media Experiments already (see for example this blog post for an overview of the latter). I believe the core logic at the heart of Media Experiments is a great foundation to implement such a feature, and Gutenberg is the right place to further develop it and bring it to more people.

Overview

Current image processing in WordPress relies on server-side resources and older image libraries, leading to potential performance issues and limited support for modern image formats such as AVIF. This results in a subpar user experience, particularly with resource-intensive tasks like resizing and compressing images. Additionally, the lack of modern compression tools like MozJPEG further hinders optimization efforts.

Client-side media processing offers a solution by leveraging the browser's capabilities to handle tasks like image resizing and compression. This approach not only alleviates the strain on server resources but also enables the use of more advanced image formats and compression techniques, ultimately improving website performance and user experience. By tapping into technologies like WebAssembly, WordPress can provide a more efficient and seamless media handling process for both new and existing content.

Resources

Roadmap

Given the many possibilities client-side media processing unlocks, this project can be split into multiple phases, each with their dedicated goals and success criteria. For example:

Phase 1

Goals

Nice to have

Non-goals

Phase 2

Goals

Nice to have

Non-goals

Phase 3

Goals

Nice to have

Out of scope

Non-goals at the moment, or something for far out in the future:

Caveats and risks

Technical complexity

WASM-based image optimization requires SharedArrayBuffer support, which in turn requires cross-origin isolation. Implementing that in a robust way without breaking other parts of the editor is challenging. There are currently some known issues in Firefox and Safari due to these browsers not supporting credentialless iframe embeds. Embed previews in the editor currently do not work because of this, until those browsers add support for <iframe credentialless>.

Image compression by itself is rather complex due to the vast amount of different combinations of codecs and encoding options available. While the existing server-side image handling in WordPress has been proven over many years, doing everything client-side is new territory. There will be many edge cases that need to be handled.

As for overall architecture design, the existing Media Experiments project contains a very solid foundation that already solves many challenging problems and could be built upon.

Confidence level

Reception in the community so far has been extremely positive, including from WordPress leadership. There is clearly a desire for this kind of solution.

The biggest risk is about compatibility with other parts of the editor (plugins, embed previews) and certain environments because of the cross-origin isolation required for SharedArrayBuffer. Not breaking existing sites is very important. The risk can be mitigated through extensive testing, additional safeguards, and documentation. Lack of <iframe credentialless> support is not ideal as it degrades editor UX in some cases, but it does not break the authoring experience as a fallback exists.

While image optimization could theoretically be done through other means such as the canvas API, said API is extremely limited across browsers. For example, there is a lack of support for compression options (e.g. lossless or lossy) and mime types (no AVIF everywhere, no WebP in Safari).

Implementation status

ramonjd commented 5 months ago

Firstly, thanks so much for the detail and all the background work!

Converting HEIC images to a web-safe format

This came up in a recent discussion I was having.

What's your take on the level of effort to integrate this feature? I was browsing the media experiments repo code and was wondering if there were any further considerations, beyond what you've explored already.

It would be a fantastic feature to have - happy to have a shot at it, or try to organize someone to have a shot at it 😄 if you think it's worth it at this stage. I'll take your guidance.

Thank you!

swissspidy commented 5 months ago

I heard of similar discussions recently :)

Always great to hear interest in this kind of topic. And I am glad you already browsed the media experiments code a little bit.

If you are interested solely in HEIC conversion, it by itself could be implemented relatively easily in theory. See #61861 as a demonstration.

However, as with everything else mentioned in this ticket, there are some important aspects to consider. For instance:

And perhaps most importantly: libheif-js and the underlying libheif license is licensed under LGPL 3.0, which is incompatible with GPL v2, but is compatible with GPL v3. Given that WordPress is licensed under GPLv2 or later, it would mean that the final product would need to be distributed under GPL v3 only. IANAL, but I don't think this is gonna fly. So that means an alternative to libheif-js needs to be first evaluated and tested. There are some libraries built on libavif but I haven't had success using those in the past.

(Edit: dynamically linking via loading libheif-js from an externally hosted script might be OK, but again, not a lawyer)

➡️ Aside: Thanks to this POC PR I discovered that the current license check in this repo is broken 😬 I am fixing it in #61868

Bottom line:

I've deliberately split this project into multiple phases as it's more complex than what it might seem on the surface. I strongly suggest tackling this step by step.

Happy to discuss HEIC itself separately elsewhere.

noisysocks commented 5 months ago

Thanks for going into detail Pascal! I like your idea of a "Would you like to convert this to a web safe format?" modal that teaches the user what's going on. I've asked Automattic's legal team for some non-IANAL guidance about libheif.

I think supporting HEIC is the most important outcome here. You encounter these files more and more nowadays and it's very frustrating to users when they don't "just work" in WordPress. I'd push for starting with HEIC support and then expanding on that into other areas (compression, resizing, converting GIFs, etc.)

ramonjd commented 5 months ago

Given that WordPress is licensed under GPLv2 or later, it would mean that the final product would need to be distributed under GPL v3 only. IANAL, but I don't think this is gonna fly.

That's what I'm reading from https://www.gnu.org/licenses/gpl-faq.html#gpl-compat-matrix — if WordPress uses code under LGPLv3, then the "combined" work (WordPress + lib) would then fall under the terms of GPLv3

Bit of a wild suggestion, but is there a chance we could fallback to backend processing via Imagick: https://imagemagick.org/script/formats.php#supported

I say "wild" as I assume HEIC files can be pretty massive, and batching would be pricey in terms of performance.

So that means an alternative to libheif-js needs to be first evaluated and tested. There are some libraries built on libavif but I haven't had success using those in the past.

https://www.npmjs.com/package/heic2any or https://github.com/MaestroError/php-heic-to-jpg (both MIT) might also be worth a looksy 🤷🏻

I've asked Automattic's legal team for some non-IANAL guidance about libheif.

Best answer. 😄

swissspidy commented 5 months ago

Thanks for going into detail Pascal! I like your idea of a "Would you like to convert this to a web safe format?" modal that teaches the user what's going on.

Cool. I'm going to implement it in my plugin soon so we can see how well it works UX-wise.

For WCEU I plan on working on a POC for bringing parts of that plugin to Gutenberg.

I think supporting HEIC is the most important outcome here. You encounter these files more and more nowadays and it's very frustrating to users when they don't "just work" in WordPress. I'd push for starting with HEIC support and then expanding on that into other areas (compression, resizing, converting GIFs, etc.)

Not sure I agree with this one. Performance issues due to too large & heavy images are quite widespread. HEIC is mostly an iPhone thing and not everyone uses those.

As long as there are legal question marks anyway, I'd recommend focusing on server-side HEIC conversion in CORE-53645.

npmjs.com/package/heic2any or MaestroError/php-heic-to-jpg (both MIT) might also be worth a looksy 🤷🏻

heic2any uses the same libheif library under the hood and the usage would be exactly the same, so the MIT license there seems incorrect to me.

php-heic-to-jpg only works server-side and requires an executable file on the server, which is not suitable for WP. For server-side support I recommend focusing on ImageMagick support in CORE-53645.

noisysocks commented 5 months ago

As long as there are legal question marks anyway, I'd recommend focusing on server-side HEIC conversion in CORE-53645.

I agree. Thanks @swissspidy, sorry for derailing your issue here with talk of HEIC 😀

swissspidy commented 3 months ago

At WordCamp Europe I had a great conversation with @youknowriad and @gziolo where I was able to demo Media Experiments and elaborate on its inner workings and some design decisions. Now, I made a ton of (documentation) improvements so people can better understand how it works. The technical overview doc is a great entry point.

I would appreciate if y'all could check it out. All code is 100% TypeScript and well-documented (check out the readmes!), which should help with reviews as well.

cc @ntsekouras who previously showed interest in reviewing the codebase

As the plugin name implies, there are some experimental features in there right now, but the core part, the client-side media processing & compression is what matters most and is what I am proposing to gradually merge into Gutenberg.

That said, if we think this functionality is better tested at scale in its own plugin for now, we could also consider further testing Media Experiments separately as part of the Performance Lab family of plugins before bringing it into Gutenberg.

youknowriad commented 3 months ago

@swissspidy So I've been thinking about this more (specially the store related stuff). So I've come up to the realization that we might be able to actually bundle most of the package/behavior into the generic block-editor package and store (or as a dependency of block-editor package).

In other words, all block-editors using the block-editor package can gain support for:

But, to be able to do so, we need to split the "WP specific logic" from the "Media handling logic". For the "upload API" we already have that it's the mediaUpload setting, so nothing is needed there, no new setting or anything.

For the rest, I'm not sure I understand if there's any WP specific logic in the rest of the logic (the client side optimizations, the compression...). At first glance, there's none right?

I guess what I'm proposing is the following:

Splitting the MediaExperiment into two parts:

How feasible would that be?

mtias commented 3 months ago

I'd love to start testing some of these enhancements on the plugin, we can gate them with a feature flag if anything feels too experimental yet.

ntsekouras commented 3 months ago

Riad's suggestion for inclusion in block editor seems like a good plan. I can definitely help reviewing and with whatever needed.

swissspidy commented 3 months ago

So I've come up to the realization that we might be able to actually bundle most of the package/behavior into the generic block-editor package and store (or as a dependency of block-editor package). How feasible would that be?

Interesting 👀 Let me analyze the code base a bit more to provide a good answer.

Just so I understand correctly:

Right now, everything is built as sort of a drop-in replacement for the media-utils package, so everything behaves the same as in Gutenberg:

What you are suggesting:

Is that correct?

For the rest, I'm not sure I understand if there's any WP specific logic in the rest of the logic (the client side optimizations, the compression...). At first glance, there's none right?

As mentioned, everything was built as a drop-in replacement for media-utils, so there are some WordPress specifics/assumptions in there. The ones that stand out to me right now:

youknowriad commented 3 months ago

Is that correct?

Yes, that's correct.

As mentioned, everything was built as a drop-in replacement for media-utils, so there are some WordPress specifics/assumptions in there. The ones that stand out to me right now:

Yes, the problem with that is that media-utils has always been a stateless package and making it stateful means access to its store from other packages which would break existing assumptions: things like multiple editors for instance is one example of that.

After uploading a single image, it adds all the thumbnails to the queue and uploads those to the server (using a different API endpoint than wp/v2/media)

Updating post meta for attachments after upload (e.g. after generating a video poster image or compressing an existing image)

Yeah, I think my answer here would be to try to make these use mediaUpload setting (like adding things to the arguments to handle these cases) or provide alternative settings to the block editor (which also means potentially disable these behaviors if these settings are not present)

Note: I see that useSaveImage in block-editor currently incorrectly depends on api-fetch and is clearly WordPress-specific. Given that I plan on adding client-side image editing as well, this needs to be refactored.

💯 Yes, I discovered that too late, there's an ESlint rule that is ignored there because of this (see top of the file)

swissspidy commented 3 months ago

OK, understood! In that case I will go through my plugin once more and make the necessary changes so it's easier to reason about & eventually port over.

swissspidy commented 3 months ago

Thanks for going into detail Pascal! I like your idea of a "Would you like to convert this to a web safe format?" modal that teaches the user what's going on. I've asked Automattic's legal team for some non-IANAL guidance about libheif.

@noisysocks Curious, have you ever heard back on this?

noisysocks commented 3 months ago

I did. Your understanding was correct, we'd need to relicense WordPress or use a different library 😞

swissspidy commented 3 months ago

Good news, I found some possible libheif-js alternative in the form of ffmpeg. Will share details at https://github.com/swissspidy/media-experiments/issues/483

spacedmonkey commented 2 months ago

I have been thinking about poster generation for a while. Poster generation can be done without the need to FFMPeg. This can be done using canvas tag. @swissspidy and myself worked on something similar in web stories. See this example.

We could add some middleware to the api-fetch. This can defect if a video file is being uploaded and generate the post image, uploaded and set it as the feature image. There is already logic to detect if it is a media upload request, isMediaUploadRequest.

Poster image generate would be a good POC about handling media in the browser.

swissspidy commented 2 months ago

Poster generation like this is already implemented in my media-experiments plugin and can be ported over in a later stage, according to the roadmap.

spacedmonkey commented 2 months ago

Poster generation like this is already implemented in my media-experiments plugin and can be ported over in a later stage, according to the roadmap.

Where is the roadmap?

swissspidy commented 2 months ago

Literally in the issue description.

spacedmonkey commented 2 months ago

Literally in the issue description.

Thought there would a blog post on make with more detail. The above doesn't really give a tonne of detail about poster images, other than you mentioning that may generate poster.

Updating post meta for attachments after upload (e.g. after generating a video poster image or compressing an existing image)

Can you provide more detail how this might be archived, like using a canvas tag or ffmpeg? Will this be done as part of the upload process transparently to users or requires uses to press a button?

Regarding the roadmap, as I am trying to help, I am looking at other part of it, to see what others, like myself could work on. I also trying to understand why things are ordered how they are.

swissspidy commented 2 months ago

Can you provide more detail how this might be archived, like using a canvas tag or ffmpeg? Will this be done as part of the upload process transparently to users or requires uses to press a button?

You can find a working implementation using https://github.com/swissspidy/media-experiments. Right now it uses canvas but there's an ffmpeg version as well. It's all automatically behind the scenes during the upload, but there is also a button for generating a poster for existing older videos.

Regarding the roadmap, as I am trying to help, I am looking at other part of it, to see what others, like myself could work on. I also trying to understand why things are ordered how they are.

The first step is setting the foundation for client-side media processing, with the focus on images, providing a way to compress images and generate all sub-sizes in the browser. After that, other media formats like GIFs and videos can follow.

If you're interested in any of that, I recommend checking out https://github.com/swissspidy/media-experiments.

spacedmonkey commented 2 months ago

I have checked out the repo. I have been watching the repo since you created it. Not to mentioned much of work in the repo is based on work that you and I worked on in web stories. But what is in that repo are called experiments. It is unclear what is and isn't being ported over to Gutenberg.

My point is this, even through I have followed every step of this, I still think the above "roadmap" is unclear and needs more detail. This detail is required for a number of reasons

Again, just want to understand your reasoning and support here. This work is great and will be great benefit to everyone. Just want to make that benefit clear to everyone who read this ticket. 👍