Share photos of a person with other users

jacksbox commented 1 year ago

Which version of recognize are you using?

3.4.1

Enabled Modes

Object recognition, Face recognition

TensorFlow mode

WASM mode

Downstream App

Files App

Which Nextcloud version do you have installed?

27.0.0

Which Operating system do you have installed?

Ubuntu 22.04

Which database are you running Nextcloud on?

postgres

Which Docker container are you using to run Nextcloud? (if applicable)

aio

How much RAM does your server have?

8GiB

What processor Architecture does your CPU have?

arm64

Describe the Bug

Not sure if it's a bug or a feature. I have two accounts which have access to a lot of shared pictures. The recognition works quite well and there are just a few Presons which needs to be merged. But if User A merges two Persons, or moves certain Faces to a Person, this only is applied for this user. If User B Logs in, the Persons are in the previouse, unmerged state. Is that intended?

Expected Behavior

Merged Persons and Moved Images should be applied to all users which have access to this images.

To Reproduce

Have 2 users Share all Pictures Recognise Merge some Persons for User A User B sees the unmerged Persons...

Debug log

No response

marcelklehr commented 1 year ago

Yes this is intended. At some point it may be possible to share "photos of person A" with another user, which is probably what you want, but not at the moment. see #881

jacksbox commented 1 year ago

hm, not exactly, no. I'll try to explain:

there are 1000 Images, which are shared between user 1 and 2 (both can access all)
classification and clustering finished
clustering resulted in 3 Clusters, of Person A, B, C
Now User 1 merges Cluster A and B, as this clusters are in fact the same Person
User 1 has now only 2 Clusters, A and C
User 2 (with access to the same images) still has Clusters A, B and C

I would expected - better wish ;) - that User 2 now also just has clusters A and C to prevent that user from doing the merge as well (which is easy in this example, but when we talk 10000s of images, there will be a lot of misclassifications which can be corrected)

It seems that for now each user uses individual clusters which makes this hard to achieve. But why not share clusters between all users? Show a user only pictures of a cluster to which the user has access to. If a cluster is empty for a user (because that user does not have access to any picture from it) it can be hidden / not shown. But I guess that could result in more misclassifications for setups where the users do not have many overlapping persons? Or not? Sorry, just rambling about...

marcelklehr commented 1 year ago

mmmh. Interesting thought. Let me loop in @jancborchardt here to see what he thinks.

marcelklehr commented 1 year ago

I think one thing that would be unexpected would be cluster names. Users should not implicitly share cluster names, even if they have photos of the same person, for privacy reasons. For implicitly sharing the cluster assignments, I don't know, it also feels sort of unexpected that someone else can change my cluster configuration. But then, that's only if a photo is shared already... But what if it's shared read-only?

Also, it might lead to weird problems when two users' photos get assigned to the same cluster that don't depict the same person, but the photos are not shared, then that cluster is kinda tainted without the ability to fix it.

Forza-tng commented 1 year ago

I can see both arguments. However, my personal opinion is to have a choice/setting to share metadata (people, objects, location, etc) with specific groups or people. My use case is family photos which I share with my partner. We have 100k photos from two decades stored and shared on our Nextcloud. It is great if we could jointly work on tagging photos and share the results, as doing duplicate work isn't really feasible.

rilcy75 commented 9 months ago

Hi, Great job! For me this feature is a must have. As many I use this NC app to share thousands of photo with my (huge) family and friend. Making recognizing and naming people cross NC users is a key features to address my objective. It could avoid people to use additionnal platform to host dedicated solution for photo sharing.

Olen commented 8 months ago

So, a challenge here is that not only the clusters are per user, but the detections as well, even if the files are shared:

This is an example of an image with 3 faces, shared in a groupfolder with 3 members:

  id   | fileid  |                        path                     |  user_id   | cluster_id                     
-------+---------+-------------------------------------------------+------------+------------
 18465 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | XXXX       |       1163
 18468 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | XXXX       |         -1
 18462 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | XXXX       |        134
 18461 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | YYYY       |        136
 18464 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | YYYY       |        238
 18467 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | YYYY       |         55
 18469 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | ZZZZ       |         -1
 18463 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | ZZZZ       |        140
 18466 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | ZZZZ       |          1
(9 rows)

Notice the same file_id and path, but different IDs in the face_detection table.

So each user has an entry in the table for each person (face) in each image. And each of these entries have a unique cluster_id. The person-identifier (face) is the x/y/vector, so there are some different aproaches possible:

Remove user_id from the table oc_recognize_face_clusters
Create a new table oc_recognize_face_clusters_users with two columns: cluster_id and user_id
By default continue adding one user per cluster as today.
Allow more user_ids to be added to the same cluster_id in the new table

This will need some additional code to update all the entries in the oc_recognize_face_detections table

Create a shared cluster
Add users A, B and C to the shared cluster
User A moves a "face" (actually combination of file_id, x/y/vector) to shared cluster
Magically - the same "face" (same combination of file_id, x/y/vector) is moved to the same shared cluster_id for user B and C

But we need to figure out how to handle different cases.

What if User B moves the "face" from the shared cluster to a private cluster - or removes the face?
What if User A moves a non-shared image to the shared cluster?
What if a shared image file in a shared cluster is moved to a non-shared folder?

Another option is to create what I would call "synced" clusters.

User A invites User B to sync cluster X
When the invite is accepted a new cluster Y is created for User B
All combinations of file_id+vector that exist in cluster X for User A is then duplicated for User B to cluster Y.
Any further modifications User A does to Cluster X is replicated for User B in Cluster Y
And any modifications User B does to Cluster Y is replicated to User A in Cluster X

This would need some additional columns in the oc_recognize_face_detections to add a timestamp or version id to ensure that the sync can be done properly, but is a slightly cleaner way as it can basically be done by an external script, and does not need much magic behind the scene.

Non-shared images are no problem here, because if the file_id is not found for User B, the image is just not added to User Bs synced cluster.

But it still have some challenges, like

What if User B moves a face from the synced cluster to a non-synced cluster. Where should this image be placed for User A?
There are possibilities for race-conditions where both users try to move a face to a different synced clusters at the same time.

So there is no easy way to do this as far as I can see, and it will require some work to really get it functional.

vbelloir commented 8 months ago

Hello folks, I give a try to recognize to see if it fits my needs. I am in the same situation as other. I used my self hosted nextcloud as a familly cloud, and I share 20 years of photo with my wife, and maybe later with childrens.

I give a try with a bunch of 1000 pictures. Face recognition is good, but with old photos, I have to help merging with already named cluster. If I spend hours, to parse the whole collection, I don't want that my wife and childrens have to do the same.

In my opinion, this is really a must have feature.

mkulich commented 8 months ago

Just came here to say this would be a great feature. I have the same use case where I have 10's of thousands of old family photos shared between users and it would be great if everyone could collaborate on them tagging faces.

rivendellstuff commented 7 months ago

just realized after letting it scan for awhile that its only doing it for my username and not my family for the 100k plus photos we have. this makes the feature completely useless for my use case. damn. at least tags persist. otherwise it would be completely useless to use nextcloud for photos at all

Chrispletsch commented 6 months ago

I´m facing the same use case. It would be great to have an option to decide how to handle this topic. For me it is the same as others before, me and my wife and familiy should not categorize faces two and more times without any additional benefit. For me this is a must have feature. On the otherhand memories ist just a great App!

tobiaseigen commented 4 months ago

For my use case this would be very helpful as well. My nextcloud is a family with just a few users, and we all access the same photos library.

marcelklehr commented 4 months ago

Please refrain from commenting without adding substantial information. If you wish to express support, use the emoji reactions, please. My team will look into solving issues with a large amount of support every once in a while.

marcelklehr commented 4 months ago

Note, that you do have a little influence over what Nextcloud GmbH works on: We don't promise anything, but every release cycle we try to work on enhancements that get a lot of upvotes, so you may express your support for this by giving this issue an upvote.

jancborchardt commented 1 month ago

@marcelklehr to me it sounds like a big possible privacy issue, much like the "shared tags" concept where private tags is something people want and need.

marcelklehr commented 2 weeks ago

Well, currently, even if you shared all your photos in nextcloud to the accounts of your family members, they will not see the same face clusters as you do and will have to do the same manual book keeping all over again to get clean face clusters. I guess the more pressing issue might be to improve automatic face clustering to obviate the need to do manual cleaning, still I think sharing faces is a legitimate use case.

jancborchardt commented 1 week ago

@marcelklehr right, so definitely UX-wise it’s nice, especially if it’s something that is done automatically. E.g. if you share some photos, also share the relevant face clusters. But if you set any custom names, they should of course not be shared?

That would improve the UX (since as you say, otherwise likely the same clusters would be found anyway), while not sharing the private info (the names you set manually).

marcelklehr commented 1 week ago

That's a nice idea! I like it. The only downside I see is that a users whom I have shared photos with can destroy the face clusters I have built. But that's maybe not so bad.

nextcloud / recognize