Automatic facial recognition & focal point setting

[E-VANCE]

Hey there,

this is somewhat related to #10 and would substitute the need specified there (at least for me)...

When working with the different image formats I regularly encounter 'chopped off' heads and less then ideal crops. In an ideal world each image size specified would (optionally) undergo a facial recognition and if one or more faces are recognized then the focal point for cropping would be set accordingly and – magically – the resulting images properly represent the people in them.

I would love to contribute to this and think this could be quite the valuable add-on but before loosing myself in the process I wanted to gather some feedback first...

Mainly for privacy and security reasons (but also with regards to cost when it comes to API usage) the image recognition should only happen locally IMO. That said one would have to implement a Python-based approach (f.ex. via face_recognition) since facial recognition can't (or at least shouldn't) be done in PHP, at least that's what I gathered from my quick research. Thus this would kinda 'break' the scope of the plugin (= any WordPress-plugin) and I suppose it could only be done in conjunction with a custom module in Trellis (or similar environment where we can access the server directly and install (python-)dependencies)...?

What are your thoughts on this? Would you still see this as beneficial? Or is there another, more suiting / easy approach I haven't considered?

Looking forward to hearing back, Thanks & regards

Henning

[Twansparant]

Hi Henning,

I agree, this would be neat add-on! For now, it would be a nice-to-have for us though, since we don't have a lot of time to work on the plugin at the moment.

In an ideal world each image size specified would (optionally) undergo a facial recognition

I think this will be too memory consuming, it will be better to do this per image and apply the focal point on all it's cropped variations (if that's even possible).

Mainly for privacy and security reasons (but also with regards to cost when it comes to API usage) the image recognition should only happen locally IMO

With locally you mean on the server and not in the browser? There are no privacy reasons in my opinion, no 'faces' are stored, it will just detect a face, not someone's face. It won't be connected to an actual person, at least that's not a feature we will be needing.

since facial recognition can't (or at least shouldn't) be done in PHP, at least that's what I gathered from my quick research.

Why shouldn't it happen in PHP I wonder? I found a few php wrappers for existing face recognition API's:

• Google Cloud Vision for PHP
• Kairos SDK (PHP)

What are your thoughts on this? Would you still see this as beneficial? Or is there another, more suiting / easy approach I haven't considered?

It would definitely break the scope of the plugin if the Face recognition will require a Python-based approach and to be honest I have no intentions going down that route...

If it could be done within php I'm open to it, but you'll probably need an api key for an external service, so the feature should definitely be optional.

Thanks for your time, I would love to hear any solutions you come up with! Cheers, Toine

[Elwood-P]

Something like this might be good and easier to implement than face recognition? Smart cropping using imagemagick: http://www.fmwconcepts.com/imagemagick/smartcrop/index.php

[E-VANCE]

I always like simpler / more clever approaches 😃 Thanks for chipping in! I see two issues though:

• Smartcrop probably won't work accurately in terms of facial recognition since there will be images with high details ≠ faces and thus one cannot rely on this in a feature-sense. But still this would be a valid option to offer, just needs to made clear that it is not a facial recognition tool (although a lot of the processed images will feel like it)
• Not sure how the licensing would work, that would have to be cleared before going forward. The developer states: If you: 1) redistribute, 2) incorporate any of these scripts into other free applications or 3) reprogram them in another scripting language, then you must contact me for permission, especially if the result might be used in a commercial or for-profit environment.

I probably would aim for "real" facial recognition though as smart cropping might be too generic...

Will still take some time before I can get to work on this.

[Twansparant]

I've been thinking about this for a while, in my opinion the only way this would work and would be feasible is to add an GUI option to set a focal point per image uploaded to the media library, something like this: https://github.com/wallabi-es/image-focus

Then you could use it as a f (focal) crop position in the syntax: <?= ResponsivePics::get_picture(1, 'xs:500/0.75|f'); ?>

[Elwood-P]

I've been thinking about this for a while, in my opinion the only way this would work and would be feasible is to add an GUI option to set a focal point per image uploaded to the media library, something like this: https://github.com/wallabi-es/image-focus

Then you could use it as a f (focal) crop position in the syntax: <?= ResponsivePics::get_picture(1, 'xs:500/0.75|f'); ?>

Yes I completely agree. Currently I've abandoned WP for native image processing and I'm using IMGix image CDN instead. Even their automatic focal point generation is not good enough though in practice though so I'm planning to pass them focal point coordinates set manually with something like this.

[Twansparant]

I added a new focal point picker interface to the wordpress media modal for selecting a focal point in an attachment image in the latest 1.4.0 release. Please let me know if this works out for you!

[E-VANCE]

Awesome! 🚀

Great work, many thanks for the implementation.

[Elwood-P]

Amazing, the update looks spot on – looking forward to giving it a test. Thanks for all your hard work.