Handling the case when there is no face, but we need to assign Person to the Asset

[Gorbush]

How this should be handled? I see two options:

1. create an empty Face and use it as junction
2. create a new junction entity between Asset and Person

I am afraid that I do not understand how the current logic works - and I do not want to spoil the model by introducing the faces without actual data.

Any suggestions?

[hulk66]

Not sure I understand. In which case should we assign a Person to an Asset without having a face? Do mean to situation when there was no face detected but in fact there is a face, and the user wants to relate Person and Asset?

[Gorbush]

Several different scenarios:

1. Face is there, but it is covered by hand, or hood or other person. So we would like to tag that person so we can search by it.
2. Face is there, and visible - but for some reason VGFace has not detected it. And I need to find a way to enforce that face detection. May be by making a crop of the face end enforcing the face detection in that cut piece of photo and then create a face in the asset global coordinates...

[Gorbush]

And in general - do I understand right that each time I confirm the face there is some database updated with face detection logic which affects the future face detection? If it is true - then I do not want to spoil it by simply creating a Face entity without actual pixel data.

[hulk66]

To be honest I wouldn't mess with this edge case. The face detection itself is rather good. Also, no other service (Google Photos, Amazon Photos) or application (Photoprisim) is offering this. Seems to me it is no good relation between effort and benefit.

For your question: Yes, very limited though. If you confirm a face manually, then this is noted in the database. This in turn is used then to be "more confident" when finding similar faces. In general, a face is transformed in a so called embedding (a 2048-dimensional vector). The similarity between faces is calculated by the Euclidian distance between two of those vectors.

In the beginning I had the situation, that faces where assigned to a person because they were close to a face which was close to a face (and so forth ...) which eventually was close to a manually assigned face (transitive hull). This was especially happening to faces of my sons when they were very young (<5 years). To prevent this, I introduced a "barrier" to only assign a face to a person if there are only a limited number of face in-between. So, I count the number of "hops" I'm away from a manually confirmed face and the farer I'm away, the higher the confidence must be.

I hope this clumsy explanation makes it a bit clearer.

[Gorbush]

I have a lot of cases when out of 10 faces in the picture only 3 were detected. And some people are facing to other direction and not detected at all. I know who they are basing on the clothes, and I want to assign them to these pictures, because of they are doing some activity together...

To be honest I wouldn't mess with this edge case.

Not sure I understand what you mean. If you mean that there is no good way to create faces without embedding - to assign people - then the only way I see is to add another relationship people and assign people directly without faces. Are you good with that? It complicates search logic a bit, but will not mess with the current auto-detection logic.

[hulk66]

Feel free to change it. But then I would probably go around and just create “empty” faces for the ones not detected. By empty I mean without embedding/encoding, x,y,h,w … But you should set the confidence level to none for those faces, then they will not be taken into consideration for matching new faces.

Should be relatively easy and would not change any other logic.

Make sense?

Von: Sergii Puliaiev @.> Datum: Donnerstag, 5. Oktober 2023 um 20:32 An: hulk66/timeline @.> Cc: Tobias Himstedt @.>, Comment @.> Betreff: Re: [hulk66/timeline] Handling the case when there is no face, but we need to assign Person to the Asset (Issue #23)

I have a lot of cases when out of 10 faces in the picture only 3 were detected. And some people are facing to other direction and not detected at all. I know who they are basing on the clothes, and I want to assign them to these pictures, because of they are doing some activity together...

To be honest I wouldn't mess with this edge case.

Not sure I understand what you mean. If you mean that there is no good way to create faces without embedding - to assign people - then the only way I see is to add another relationship people and assign people directly without faces. Are you good with that? It complicates search logic a bit, but will not mess with the current auto-detection logic.

— Reply to this email directly, view it on GitHubhttps://github.com/hulk66/timeline/issues/23#issuecomment-1749439974, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AREMBXHUXFUHTSLE3ADA5YDX534MTAVCNFSM6AAAAAA5RR35R6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBZGQZTSOJXGQ. You are receiving this because you commented.Message ID: @.***>

[Gorbush]

But you should set the confidence level to none for those faces, then they will not be taken into consideration for matching new faces.

That's exactly what I was looking for - how to create face and do not mess with the learning logic :-) Thank you!