Face recognition

[mholt]

Find a model that's actually competitive with Google Photos' facial recognition that can run on-device.

[sumnerboy12]

Have you seen what the Immich devs have built? Theirs is a photo-only aggregator but the facial recognition and identification ML is very impressive - I am sure they are using open-source tools for it so you might be able to get some ideas for timelinize?

[mholt]

Yes! Immich is really great. I think they use a DBSCAN algorithm/model. I've also heard good things about CLIP.

Although I studied machine learning in grad school, this particular task and the production nuances of deploying a feature like it are not my forte. So I would happily accept a contribution here. I can definitely help with the actual integrating into the app, but we'd need to discuss technologies and models and how it would work together.

Anyone want to take it up?

[mholt]

Actually, I stumbled upon this: https://ai-photo-sorter.vercel.app/ -- this demo claims to run in the web browser which is pretty cool. That's not what we need in our case, but if it's that portable, then that might be promising for us. Like if it's WASM or something.

Also, the way I see this working is we'd run the face recognizer at import-time, to automatically add a relation with entities (the people in the picture).

[sumnerboy12]

I would love to help but don't have any ML skills or experience sorry!

[Goosegit11]

What is the purpose? To keep track of your face? lol That reminds me these YouTube videos when people do a photo everyday and create a timelapse of them

[mholt]

@Goosegit11 So we can make photos searchable by who's in it! :mag: And also relate photos with the entity in our database.

Also see my comment above:

we'd run the face recognizer at import-time, to automatically add a relation with entities (the people in the picture).

[Goosegit11]

Wouldn't it be better to just do an integration with a service that is already designed for this and already does it? Immich, Photoprism

Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

[mholt]

I'm not sure what you're quoting, but that definitely doesn't apply to any of Immich, Photoprism, or Timelinize!

No, face recognition needs to be bundled into the application, especially for the sensitivity of the task, it needs to be handled on-device and without a completely separate full-blown photo gallery app installed.

[Goosegit11]

What about just adding Immich/Photoprism as a data source in Timelinize?

I'm not sure what you're quoting, but that definitely doesn't apply to any of Immich, Photoprism, or Timelinize!

Well, why not? Imagine having one app that manages your movies, photos, calendar, contacts, geolocation, and so on. Essentially self-hostable Google services. I think that would be way worse than having programs like Jellyfin or Immich for every purpose.

I was quoting Unix philosophy.

[mholt]

What about just adding Immich/Photoprism as a data source in Timelinize?

That's part of the plan, eventually :smiley: I don't have an open issue for these, but it makes sense to support as many photo gallery programs as possible.

I was quoting Unix philosophy.

Ah, well, that makes sense for Unix utilities, but Timelinize is a complex web application.

Or, I guess, you could sum up Timelinize's "one thing" as personal archive tool. It's big and broad but hey in that sense it's "one thing" :sweat_smile:

[mholt]

I have an intelligent image search prototype working in #7, I just need to integrate it into Timelinize now. It may not actually be super difficult once I figure out how to "bundle" Python with our Go app...

Anyway, once that is working, face -- and pet -- recognition should be much more feasible:

• Extract face boundaries in images
• Run those segments through a clustering algorithm/model to group the same faces together
• Give the user a way to view these groups and give each one a name

It's still non-trivial, but the overall task seems more manageable to me now.