Face Recognition

[andreasjacobsen93]

Feature request

User type: All

User level: All

Description

Implementation of OpenCV for face recognition to easily tag people in photos. This would make it easy for people to manage their photos. Their exists a wrapper for OpenCV in PHP, to allow the use of it (possibly v2 feature)

As a beginning (v1) we could integrate some sort of face detection on images, and allow users to tag people themselves. Then a menu link could bring up a face albums, as tagged by the user.

Benefit / value

This would bring the NextCloud gallery on level with similar solutions, from Google, Apple and Microsoft.

[Benfm2104]

I like the idea!

There is already a set of options for automated tagging, should OpenCV-features be added there to set the configuration (like people to tag)? https://github.com/nextcloud/files_automatedtagging

[matiasdelellis]

Maybe with this we can do something fast..

• https://github.com/ageitgey/face_recognition

At least write a cron to analyze the images, and add nextclooud tags or exif info.. The interface inside the gallery is another discussion, but I guess the important thing is simplify how to get the information, and do it transparent to the user ...

[sunjam]

What @matiasdelellis linked above works! It was added to an app called Ownphotos via a pull request.

Here is the pull request, which adds facial recognition via Nextcloud CarDAV Contacts.

[matiasdelellis]

Hi all, Although we are planning an important rewrite, we already have something functional directly in nextcloud. https://github.com/matiasdelellis/facerecognition

Soon news, :wink:

[stalker314314]

Hi everyone, Wanted to give small update and to also invite whomever wants to help:) FaceRecognition app that @matiasdelellis mentions is already working! However, at the same time, we are working on different approach. With this approach, we think we can get better end-user experience, which includes robustness (cluster stability), make everything "just works" (background tasks instead of commands) and reduce dependencies (no need for Python, just dlib).

Complete functional spec is currently here (plan is to move it to github wiki). We already implemented all PHP binding in dlib in this repo. There is now new branch in Matias' repo where we got background task crunching images, but it will still take some time before we got everything packaged up with cherry on top. Look at spec (add stuff if you think you have something to add), roam around repo, shout to two of us if you want to poke around! 😃

[SlavikCA]

Sounds very exciting!

Complete functional spec is currently here (plan is to move it to github wiki).

That link doesn't work. Is it in Wiki already? Which wiki?

[stalker314314]

It will end up here (but not there yet). For original link, you'll probably need LiveID. If you don't have one, I exported PDF here for convenience (I don't suspect it will change much in future, so we can tentatively move it to GH wiki, but waiting for @matiasdelellis to confirm)

[matiasdelellis]

Hi all, Of course, the documentation should be passed on to the wiki, but it's a lot of work to organize correctly to be really useful.. Meanwhile you can read it there, and give your opinion on the repo/issues of the application.

Thanks!

[stalker314314]

Functional spec (or design notes, call them what you want:D) are moved to wiki: https://github.com/matiasdelellis/facerecognition/wiki/Functional-spec. All comments are welcome!

[SlavikCA]

@stalker314314 Super exciting! Love to see that feature in the NextCloud.

That was the only NC shortcoming, preventing me from switching from GPhotos.

[emresaracoglu]

Is there any progress?

[stalker314314]

Well, @emresaracoglu, if you are refering to what we are doing in https://github.com/matiasdelellis/facerecognition, then certainly yes! First version (the one we plan to publish on app store) is almost done, we are now busy working on:

• producing binary packages for dependencies (we don't want people to compile pdlib from scratch, we want as smooth experience installing this app as humanely possible)
• documentation

Other than that, we think we can have app in app store this year (ideally spring/summer).

However... today - you need to fetch it from github, install stuff yourself...should be easy if you follow installation, but unfortunately, it is not for everyone. We need as many eyes as possible to try it out (before we release it on app store) and we need as many people as we can throwing ideas how we can improve experience using Nextcloud, now that people have their own set of face data...

If you want to help, start with wiki, and file an issue for ANY issue, typo, paper cut, problem, bug you encounter, this would help us big time!:) Otherwise, subscribe to this issue, we will notify here once it is out in app store;)

[emresaracoglu]

we don't want people to compile pdlib from scratch, we want as smooth experience installing this app as humanely possible

Great!

I discovered your repositories after this my message. It got me very excited this your app 👍

---

but I'm having a little problem, I'd appreciate it if you could help matiasdelellis/facerecognition/issues/127

[godfuture]

Is this still ongoing? I am very interested, because I used Google Picasa in the past which was able to automatically do facial recognition tagging. This feature would perfectly fit into the cloud.

[migueltg]

I am very interested too

[sunjam]

Did either of you read the post just above yours, linking directly to the app?? It is right there; give it a shot.

On Sun, Oct 27, 2019 at 11:49 AM Miguel notifications@github.com wrote:

I am very interested too

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nextcloud/gallery/issues/281?email_source=notifications&email_token=AANUKZVJGZV36GJPFIKDRITQQXIBBA5CNFSM4DN56QDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECLFFJQ#issuecomment-546722470, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANUKZVWBNTMCWIJ7W4YCYTQQXIBBANCNFSM4DN56QDA .

[migueltg]

Did either of you read the post just above yours, linking directly to the app?? It is right there; give it a shot. … Thanks 💪

[matiasdelellis]

Maybe with this we can do something fast..

* https://github.com/ageitgey/face_recognition

At least write a cron to analyze the images, and add nextclooud tags or exif info.. The interface inside the gallery is another discussion, but I guess the important thing is simplify how to get the information, and do it transparent to the user ...

Hi everyone, As the application was born with this simple message, I should tell here that we have a first release to test.. . :tada: :smile:

• https://apps.nextcloud.com/apps/facerecognition

Sure there is much to improve, but it is already useful enough to try... :wink: :smiley: As always, I invite you to try it, translate, complain, etc.. :wink:

[henri2h]

Hi everyone,

Small idea for the future face recognition feature.

Some software like Digicam are already using face recognition. Digikam allow you to save the face position in the metadata of the file. With this, one can tag a face on a picture and see the tagged picture on another computer.

(FEATURE) The nextcloud photos app could sync the recognized faces with the metadata which could allow users to tag a picture in Digikam for example and see it when using the photos app in nextcloud.

Thanks for all the work put in making this new app alive !! Can't wait trying the photo app once the nexcloud snap package will be updated to nextcloud 18 !!

[skjnldsv]

Can't wait trying the photo app once the nexcloud snap package will be updated to nextcloud 19 !!

nextcloud hub is shipped with photos (18) :)

[burk80]

is it possible to make this into a docker usable install.. i know there is problems with this atm .. but it seems like you need to compile the whole system with pdlib and dblib into photos or maybe even into nextcloud core .. which isnt desirable ... so if there is a possibility to make a docker image for facial_recognition so that you could link that in docker compose to nextcloud and or photos

[skjnldsv]

is it possible to make this into a docker usable install.. i know there is problems with this atm .. but it seems like you need to compile the whole system with pdlib and dblib into photos or maybe even into nextcloud core .. which isnt desirable ... so if there is a possibility to make a docker image for facial_recognition so that you could link that in docker compose to nextcloud and or photos

Would be awesome yes :)

[agboom]

@burk80 @skjnldsv You might be interested in following this thread: https://github.com/matiasdelellis/facerecognition/issues/221

There's also some documentation on how to get started with Dlib/Pdlib and Facerecognition in https://github.com/matiasdelellis/pdlib/wiki/Docker

Feedback is appreciated :smile:

[averdecia]

Hi, @matiasdelellis have you considered remove the image processing from the NC web server, and create a different application instead? Communicate them between APIs. We have an almost-working solution with this scheme. The main problem with the face recognition app is that it cant scale. For big instances, it increases the upload time and the heavy writing operation son the database. It may be a good idea if you change the core to an external source. Also, the models could learn from some user changes. A corn job is not a good solution for big instances, you may be able to see your grouped photos more than 1 week after you upload them

[matiasdelellis]

Hi @averdecia

Hi, @matiasdelellis have you considered remove the image processing from the NC web server, and create a different application instead?

In principle, I am always ready to listen to proposals.. 😉

Communicate them between APIs. We have an almost-working solution with this scheme.

Well.. Show me what API you are working on... My proposal is to add virtual folders to NC core.. an api similar to the Unified Searchs, where my application registers, informs the virtual folders, and when Files or Photos requires the content, I send the results.

The main problem with the face recognition app is that it cant scale. For big instances, it increases the upload time and the heavy writing operation son the database.

We have a large consumption for photo analysis, but I honestly think there is no better API than the one we are using.. However it's perfectly controlled.. It just depends on the size of the temporary image. You want quality?, use a large image and it will be slow and consume more memory. Do you want it to be fast? Minimize the images and you will lose some faces.

About how it scale, you may be right during clustering. But beyond the analysis (Which is the highest consumption but it is controlled), we have a single report which fails during clustering with 55,000 images consuming 2GB of ram. https://github.com/matiasdelellis/facerecognition/issues/339

We can improve, but I think it is still acceptable..

It may be a good idea if you change the core to an external source.

https://github.com/matiasdelellis/facerecognition-external-model ¿? 🤔

Also, the models could learn from some user changes.

Now we are focusing on automatic clustering, but of course we want to get there.

A corn job is not a good solution for big instances.

Why? Would you like it to be integrated into the nextcloud cron?

you may be able to see your grouped photos more than 1 week after you upload them

Please try uploading 55,000 photos to google photos, and tell me how long it takes.. 😅 Why do you think that the photos of your cell phone in google photos take time? Your cell phone does not do it, a server farm does the hard work and even takes time..

Please don't think we can do better than google .. 🙈 😅

[meichthys]

I think this issue may be resolved with the new "Recognize" app

Edit: or the FaceRecognition app!

[matiasdelellis]

Yup.. Touche.. 😉

[meichthys]

@matiasdelellis Could you clarify the primary differences between FaceRecognition and Recognize when it comes to face recognition? Can they be run together if i just disable facial recognition in the Recognize app settings?

[matiasdelellis]

Hi @meichthys I don't want to make any controversy about this.

Regarding this report, recognize was officially integrated as a facial recognition solution for Nextcloud Photos, and Nextcloud Face recognition is still a third-party app... So, it can be closed.

[meichthys]

@matiasdelellis Understood. As a user, it's confusing to see multiple apps that seem to serve the same purpose. I haven't seen any kind of comparison on how the two apps compare.

[matiasdelellis]

Ok.. Sorry, being politically correct, I'm not helping... 😅

The main difference, and the reason for the new development within Recognize, is that this application has a 100% automatic approach. Install and run as nextcloud cron.

The other application, Face Recognition, needs some external dependencies (pdlib) that some users find difficult to install.. and needs some initial configuration. Mainly how much memory you want to dedicate to the process, and the size of the images that basically defines the quality of the process.

Regarding the integration, recognize is integrated. There is no discussion about this. Regarding the quality of the results, we are discussing an application that just added this feature, so it is not fair to compare, and honestly I can not be impartial about it.

[nacho-domatix]

@meichthys Seems like we're seeking the same thing. Small world xD

[jakobroehrl]

Should be solved with NC25 and github.com/nextcloud/recognize