timelinize / timelinize

Store your data from all your accounts and devices in a single cohesive timeline on your own computer
https://timelinize.com
GNU Affero General Public License v3.0
641 stars 19 forks source link

Face recognition #6

Open mholt opened 2 months ago

mholt commented 2 months ago

Find a model that's actually competitive with Google Photos' facial recognition that can run on-device.

sumnerboy12 commented 2 months ago

Have you seen what the Immich devs have built? Theirs is a photo-only aggregator but the facial recognition and identification ML is very impressive - I am sure they are using open-source tools for it so you might be able to get some ideas for timelinize?

mholt commented 2 months ago

Yes! Immich is really great. I think they use a DBSCAN algorithm/model. I've also heard good things about CLIP.

Although I studied machine learning in grad school, this particular task and the production nuances of deploying a feature like it are not my forte. So I would happily accept a contribution here. I can definitely help with the actual integrating into the app, but we'd need to discuss technologies and models and how it would work together.

Anyone want to take it up?

mholt commented 2 months ago

Actually, I stumbled upon this: https://ai-photo-sorter.vercel.app/ -- this demo claims to run in the web browser which is pretty cool. That's not what we need in our case, but if it's that portable, then that might be promising for us. Like if it's WASM or something.

Also, the way I see this working is we'd run the face recognizer at import-time, to automatically add a relation with entities (the people in the picture).

sumnerboy12 commented 2 months ago

I would love to help but don't have any ML skills or experience sorry!

Goosegit11 commented 3 weeks ago

What is the purpose? To keep track of your face? lol That reminds me these YouTube videos when people do a photo everyday and create a timelapse of them

mholt commented 3 weeks ago

@Goosegit11 So we can make photos searchable by who's in it! :mag: And also relate photos with the entity in our database.

Also see my comment above:

we'd run the face recognizer at import-time, to automatically add a relation with entities (the people in the picture).

Goosegit11 commented 3 weeks ago

Wouldn't it be better to just do an integration with a service that is already designed for this and already does it? Immich, Photoprism

Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

mholt commented 3 weeks ago

I'm not sure what you're quoting, but that definitely doesn't apply to any of Immich, Photoprism, or Timelinize!

No, face recognition needs to be bundled into the application, especially for the sensitivity of the task, it needs to be handled on-device and without a completely separate full-blown photo gallery app installed.

Goosegit11 commented 3 weeks ago

What about just adding Immich/Photoprism as a data source in Timelinize?

I'm not sure what you're quoting, but that definitely doesn't apply to any of Immich, Photoprism, or Timelinize!

Well, why not? Imagine having one app that manages your movies, photos, calendar, contacts, geolocation, and so on. Essentially self-hostable Google services. I think that would be way worse than having programs like Jellyfin or Immich for every purpose.

I was quoting Unix philosophy.

mholt commented 3 weeks ago

What about just adding Immich/Photoprism as a data source in Timelinize?

That's part of the plan, eventually :smiley: I don't have an open issue for these, but it makes sense to support as many photo gallery programs as possible.

I was quoting Unix philosophy.

Ah, well, that makes sense for Unix utilities, but Timelinize is a complex web application.

Or, I guess, you could sum up Timelinize's "one thing" as personal archive tool. It's big and broad but hey in that sense it's "one thing" :sweat_smile:

mholt commented 3 weeks ago

I have an intelligent image search prototype working in #7, I just need to integrate it into Timelinize now. It may not actually be super difficult once I figure out how to "bundle" Python with our Go app...

Anyway, once that is working, face -- and pet -- recognition should be much more feasible:

It's still non-trivial, but the overall task seems more manageable to me now.