Recognize results are lost when transferring files to other user

andreaskoelsch commented 7 months ago

Which version of recognize are you using?

6.1.1

Enabled Modes

Object recognition, Face recognition

TensorFlow mode

Normal mode

Downstream App

Files App

Which Nextcloud version do you have installed?

28.0.4

Which Operating system do you have installed?

Debian 12

Which database are you running Nextcloud on?

MariaDB 10.11.6

Which Docker container are you using to run Nextcloud? (if applicable)

No response

How much RAM does your server have?

16GB

What processor Architecture does your CPU have?

x86_64

Describe the Bug

When files are transferred from one user to another user via transfer ownership, only the classification results are transferred, face detections are lost. Also, the transferred files are not added to the face detection queue for detecting the faces again.

Expected Behavior

Either keep the face recognition and run the clustering for the new user or re-run both face recognition and clustering after moving the files.

To Reproduce

Upload images to one user
let recognize do its thing
transfer ownership of these images to another user
see that face detections are lost and files are not added to face detection queue

Debug log

No response

github-actions[bot] commented 7 months ago

Hello :wave:

Thank you for taking the time to open this issue with recognize. I know it's frustrating when software causes problems. You have made the right choice to come here and open an issue to make sure your problem gets looked at and if possible solved. I try to answer all issues and if possible fix all bugs here, but it sometimes takes a while until I get to it. Until then, please be patient. Note also that GitHub is a place where people meet to make software better together. Nobody here is under any obligation to help you, solve your problems or deliver on any expectations or demands you may have, but if enough people come together we can collaborate to make this software better. For everyone. Thus, if you can, you could also look at other issues to see whether you can help other people with your knowledge and experience. If you have coding experience it would also be awesome if you could step up to dive into the code and try to fix the odd bug yourself. Everyone will be thankful for extra helping hands! One last word: If you feel, at any point, like you need to vent, this is not the place for it; you can go to the forum, to twitter or somewhere else. But this is a technical issue tracker, so please make sure to focus on the tech and keep your opinions to yourself. (Also see our Code of Conduct. Really.)

I look forward to working with you on this issue Cheers :blue_heart:

andreaskoelsch commented 7 months ago

PS: When moving files to a Group Folder, both face detection and object classification are re-triggered. Is this intended or worth a separate issue?

andreaskoelsch commented 7 months ago

It gets even more odd. I just realized, when I move separate files to Group Folders, they are triggered. If I move entire folders, on the other hand, the contained files are not triggered. Pretty sure this is not what's intended

andreaskoelsch commented 7 months ago

PPS: Even though the files which are moved separately to Group Folders are reprocessed by recognize, the faces do not show up for the other users, but only for the one who moved them there.

Current workaround: move the files/folders via the filesystem and re-scan the directories. That way, they are reprocessed and the results show up for all users in the Group Folder.

marcelklehr commented 7 months ago

Hello @andreaskoelsch

Thank you for this detailed analysis! File transfer to other users is a relatively new feature in nextcloud, so we haven't integrated with this feature yet. I'll also look into the other issues you found.

andreaskoelsch commented 7 months ago

Great, looking forward to testing it 😊

andreaskoelsch commented 6 months ago

I just found the time to test this today. Seems, it's still not fixed, as the face clustering is not triggered. When I move a file from my personal space to a group folder, it gets added to the classify and face recognitiion queues. However, after the predictions, the detected faces are not added to the clustering. In my own profile, the face recognition works, because it was already detected before moving the file to the group folder. However, other users of the group folder do not get a face recognition

marcelklehr commented 6 months ago

Mh, that may be because group folders are not technically shares, while I've been using the share manager to enumerate users with access. I'll change that to use the mount cache instead...

marcelklehr commented 5 months ago

Hey @andreaskoelsch I've prepared a beta version with a potential fix. Could you try this https://github.com/nextcloud/recognize/releases/tag/v7.0.2-beta.1 and report back here?

andreaskoelsch commented 5 months ago

Yes, we're almost there ;-) Thanks for the beta, I just tested it by uploading a picture of myself and moving it to a group folder. What happened after moving the file:

re-triggered face and imagenet classification
found face
1 face left for clustering, planned tasks: 0

So, what's left is that the clustering needs to trigger a new task after the face detection.

Also, what I found not ideal: After moving the file to the group folder, the classification labels are already available for both users before the imagenet executes again. It seems, they are already transferred with the file and another imagenet execution could be avoided for energy efficiency. Also, this makes me wonder if it would maybe be possible to transfer the face detection in a similar way, so the other group users would only need to run the clustering of the newly available faces instead of having to re-run the face detection. However, face detection is cheap, so if this is hard to implement, I won't complain

marcelklehr commented 5 months ago

No new classification should be run when moving files :/ Are you sure that another face classification is run on the file? The face detection should just be copied.

andreaskoelsch commented 5 months ago

Yes, absolutely. That was already the case in the comment above (https://github.com/nextcloud/recognize/issues/1118#issuecomment-2037712967)

marcelklehr commented 5 months ago

Can you try moving a folder with a file in it, to check if the face detections get copied to the groupfolder users? Then all i have to fix is the re-classification.

andreaskoelsch commented 5 months ago

I just tested this, i.e.moving a folder containing a file with a face detection to a group folder. After moving the entire folder, neither face detection nor clustering nor imagenet classification are triggered for this file.

The original user of the file still sees the face detection and classification labels of the file. Other users only see the classification labels, but no face detection. I also checked the unassigned faces tab, but nothing there.

marcelklehr commented 5 months ago

And just as a sanity check, when moving the folder to a shared folder, does the face detection show up for the sharees of that share?

andreaskoelsch commented 5 months ago

Is that question answered with the second paragraph of my last comment? otherwise, I don't understand it.. To repeat, only the original user sees the face detection of that file.

Btw, thanks for the quick replies ❤️

marcelklehr commented 5 months ago

Sorry, I will explain. There's groupfolders and there is normal shared folders (ie. a folder that a user shared with some other users). Could you repeat your experiment but instead of moving the folder with face pictures into a group folder, could you move it into a normal folder that is simply shared with some users? The question is again, does this copy the face detections to the other users. (From a technical perspective, groupfolders and shared folders are vastly different beasts)

andreaskoelsch commented 5 months ago

Ohh, gotcha. So, I tried sharing this folder with one image inside with another user. Interestingly, recognize then showed 2 faces left for clustering, 0 planned tasks. Not sure, why two faces.. anyways, I ran occ recognize:cluster-faces manually and the second user also shows the face.

ClusterDebug: Retrieving face detections for user testuser
ClusterDebug: Found 2 fresh detections. Adding 0 old detections and 2078 sampled detections from already existing clusters. Calculating clusters on 2080 detections.
ClusterDebug: Clustering complete. Total num of clustered detections: 2

marcelklehr commented 5 months ago

Ok. So, to summarize:

groupfolders still don't work
moving files triggers new classifications

andreaskoelsch commented 5 months ago

Not quite. Let me give a more detailed summary. Given a selfie with one face detection which I deleted and re-uploaded for each test scenario. All tests done with the most recent beta.

Shared Folder:
- Sharing a folder with a selfie inside says 2 new faces in the recognize admin panel, even though there's just one detection there. Also, this requires a manual call of clustering, then it works for all sharees
- Moving the selfie to an already existing shared folder works (only one new face waiting for clustering), but requires manual call of clustering. Afterwards, both sharees see the detection
- Moving a folder with the selfie inside to an already exising shared folder does not work. No new detections for the sharees.
Group folders:
- Moving the selfie to the group folder retriggers face and imagenet classifications. Afterwards, it says 1 new detections, 0 planned tasks. After manual clustering call, all group folder users see the detection.
- Moving the folder with the selfie inside to a group folder neither triggers a classification nor are there new detections. Very much like in the third case above.

So moving individual files works for both sharing and group folders, but requires manual clustering call and causes unnecessary classifications in case of group folders. Moving folders with files inside does not work.

PS: Classification labels work in all cases!

marcelklehr commented 5 months ago

So, I believe I've found the culprit(s). I'll make available a new beta later today, so you can test :)

andreaskoelsch commented 5 months ago

Was this last commit supposed to be the new beta, or will you publish a beta release? I don't want to stress you, just want to know whether I should already test or still wait.

marcelklehr commented 5 months ago

Nope, I still haven't published the beta, sorry.

andreaskoelsch commented 5 months ago

no worries, it's not urgent

marcelklehr commented 5 months ago

here you go :) https://github.com/nextcloud/recognize/releases/tag/v7.0.2-beta.2

andreaskoelsch commented 5 months ago

Thanks! So far, I've only tested one use case: Moving files to a group folder. Unfortunately, the moved files are still added to both face recognition and imagenet queues. However, Face Clustering for the group folder users is now triggered and working.

marcelklehr commented 5 months ago

Unfortunately, the moved files are still added to both face recognition and imagenet queues

I think I've fixed that now, too.

andreaskoelsch commented 5 months ago

Locally? At least not in the current beta

marcelklehr commented 5 months ago

Locally? At least not in the current beta

Yes, not in the beta, sorry, in the repo :)

andreaskoelsch commented 4 months ago

Today, I had a case of a moved folder. Face detections were not copied to other users. So, this is still not working. I can do some more testing with the next version

marcelklehr commented 4 months ago

I've published a new release just now with the fix

andreaskoelsch commented 4 months ago

Thanks for the effort. I just tested it and it seems the classification is now broken completely. I created a new ticket for this, as it's probably more urgent (https://github.com/nextcloud/recognize/issues/1154)

andreaskoelsch commented 4 months ago

Thanks for the quick fix to above issue. I tested again now. Moving files works as expected now, moving folders with files inside them still not.

marcelklehr commented 3 months ago

I'll look into this soon.

marcelklehr commented 2 months ago

Hi @andreaskoelsch the priority of this issue was reduced to "normal". This means Nextcloud GmbH will not investigate this actively anymore. We will however, happily accept pull requests with fixes.

andreaskoelsch commented 2 months ago

Thanks for the message, it's alright for me. At least it works now by moving files instead of folders, so there is a simple workaround for the issue

nextcloud / recognize