Move to better face recognition models

[florianwgnr]

Describe the feature you'd like to request

I have installed Recognize together with Memories and so far everything works: Recognize scans my images for faces and groups them together. Great!

Yesterday, just for testing, i set up an instance of immich and loaded the same photo library i have stored in my Nextcloud. Immich currently uses the buffalo_l model from InsightFace for facial recognition, and after scanning i was blown away by the accuracy: The model not only found way more pictures, it was also way better at assigning them to the right person. To give you some numbers: recognize found 706 pictures of me, immich with buffalo_l found 3780 pictures. All correctly assigned with sometimes mind-blowing accuracy.

Since the results were that good, i thought about suggesting replacing of the face recognition model, e.g. with buffalo_l or similar, to further promote Nextcloud as the main storage for photos. I am aware that this is not a one-click replacement and might require a lot of effort or additional work, but as this might also affect ongoing discussions, e.g. https://github.com/nextcloud/recognize/discussions/865 regarding undetected faces, i wanted to share my thoughts with you.

Describe the solution you'd like

Move to an improved face recognition model

Describe alternatives you've considered

--

[github-actions[bot]]

Hello :wave:

Thank you for taking the time to open this issue with recognize. I know it's frustrating when software causes problems. You have made the right choice to come here and open an issue to make sure your problem gets looked at and if possible solved. I try to answer all issues and if possible fix all bugs here, but it sometimes takes a while until I get to it. Until then, please be patient. Note also that GitHub is a place where people meet to make software better together. Nobody here is under any obligation to help you, solve your problems or deliver on any expectations or demands you may have, but if enough people come together we can collaborate to make this software better. For everyone. Thus, if you can, you could also look at other issues to see whether you can help other people with your knowledge and experience. If you have coding experience it would also be awesome if you could step up to dive into the code and try to fix the odd bug yourself. Everyone will be thankful for extra helping hands! One last word: If you feel, at any point, like you need to vent, this is not the place for it; you can go to the forum, to twitter or somewhere else. But this is a technical issue tracker, so please make sure to focus on the tech and keep your opinions to yourself. (Also see our Code of Conduct. Really.)

I look forward to working with you on this issue Cheers :blue_heart:

[bonswouar]

A better face recognition model would really be needed, current one isn't doing so great imo. It seems face detection is done thanks to https://github.com/justadudewhohacks/face-api.js , and there haven't been any new commit there for the last 4 years..

I guess a PR on face-api.js with new models would be doable, but as far as I understand it would need to be available for tensorflow-js (see my comment on image detection models) @florianwgnr There are some tensorflow implementations of InsightFace that you quoted, but unfortunately they're all much older than the release of the model buffalo_l (septembre 2021 I believe)

[tetebueno]

How about this fork that's being mentioned in project's issues?: https://github.com/vladmandic/face-api

[bonswouar]

How about this fork that's being mentioned in project's issues?: https://github.com/vladmandic/face-api

It doesn't look like the models have been updated much

[float3]

bumping this issue

[Handkep]

Is it feasible to create a wrapper for buffalo_l to substitute the face_api directly?

[bonswouar]

Is it feasible to create a wrapper for buffalo_l to substitute the face_api directly?

I guess it could be feasible, although the preferred way would probably be to implement https://github.com/nextcloud/recognize/issues/73 to have docker-based external app, which could theoretically load any model then...

[cyrill-b]

I believe the recently published SDK here might be helpful to this effort.