Saliency detection

[swissspidy]

Detect the most relevant part of an image to improve thumbnail cropping. For example using MediaPipe’s object detection solution.

[swissspidy]

VIPs supports smart cropping (vips_smartcrop()) with the attention strategy (--smartcrop attention), which is also the default. "This one searches the image for features which might catch a human eye"

[johnbillion]

libvips is available in wasm which means we could use Sharp client-side and get some really powerful image processing capabilities without needing the PHP bindings.

Looks like StackBlitz has done some work on this. I've no clue what level of processing power it requires, whether it's feasible on a mobile device, etc.

[swissspidy]

libvips is already setup in this project, so I‘d probably start with that. Though it would be nice being able to use with other encoders as well.

[kleisauke]

Here's an online wasm-vips demo that uses the attention strategy: https://kleisauke.github.io/wasm-vips/playground/?modules=#smartcrop

Note that Wasm support in sharp v0.33.0 requires a Node.js compatible runtime, making it currently incompatible within web browsers.

[swissspidy]

Will probably simply turn this toggle into a "Thumbnail generation" dropdown:

Possible options:

• Client-side (default)
• Client-side (smart)
• Server-side

The attention strategy seems to work well, though sometimes the result is a bit odd at first.

Example input image:

Default crop:

Attention: