search

diablodale / dp.kinect2

Kinect v2 (XBox One) plugin for Cycling '74 Max
21 stars 4 forks source link

use face recognition to differentiate users? / human interaction scenarios #52

Closed pantracols closed 5 years ago

pantracols commented 5 years ago

hi

I'm using dp.kinect2 to track the joints of two users. then i use the position of the joints to process sound in real time. i'm very happy with the results and now i want to do concerts with two users. the problem is that everytime one of the users leaves the area that kinect sees and comes back, kinect assigns this user a new user number, different to the one it had previously, so i need to re-patch in max.

is there a way to use facial recognition to tell kinnect who is who at each moment? or is there a better way of doing what i need? i thought facial recognition would be a good way beacuse it would also prevent the system from reacting to the movement of possible audience or technitians that could be inside the area that the kinnect sees

thanks a lot!! best santiago

diablodale commented 5 years ago

Hi. The topic you are inquiring into is higher-level app related rules and logic. Something that the Kinect itself nor dp.kinect2 can directly solve for you.

In general, you are asking about human/player management and interaction/engagement scenarios. I recommend you think about your project and what it means to be a player...or to be a human that engages with your project. With clear rules, then you can see what technologies or processes can help you enforce those rules.

Just as an example...maybe one of your rules is the human closest to the Kinect is the one that is engaging with the project. Do you want the foremost to always be immediately chosen? In my experience, it is usual for multiple people to try and interact at the same time with the Kinect. If they are side-by-side or moving around as a group, they will vary their distance from the Kinect by centimillimeters (or millimeters). Even if two people stand "still" they tend to lean and sway and therefore their distance changes. If two (or more) people are standing side-by-side, this could result in each their distance values to "compete". The Kinect will see this and a simple algorithm always picking the closest will switch to the people leaning closest. Perhaps this is an affect you want. And often people love trying to use/interact with the Kinect in unexpected ways. Thinking about and handling these interaction scenarios is part of the creativity and the unique way you will create your solutions. :-)

Could face recognition help implement one of your rules? I don't know as I don't know your project well enough. I will caution you that the face shape recognition on the v2 Kinect you are using has two sets of data. You can get a somewhat quick general shape and major points. The full face shape takes a very long time and I doubt anyone could use it in a fast real-time project. (The Kinect v1 sensor is actually faster in creating its face model...less accurate...but faster...the v1 is still a good piece of hardware)

Again, I don't know your rules...but maybe something could be done with color of clothing, or IR leds that only the Kinect sensor can "see". It is possible to create a mask of the RGB image based on the humans identified. Then perhaps you can see the color of clothing for a person and use that for identification. Examples of how to mask are on the "tips" tab in the help file provided with dp.kinect2.

Happy patching! :-)

pantracols commented 5 years ago

hi dale! thank you so much for taking the time to write such an long and caring reply to my question. i have been able to program the patch in a way that serves the purposes of my performance following your advice. i have to say that kinect + dp.kinect + max are a dream come true of possibilities! all the best, santiago

On 11 Jun 2019, at 20:05, Dale Phurrough notifications@github.com wrote:

Hi. The topic you are inquiring into is higher-level app related rules and logic. Something that the Kinect itself nor dp.kinect2 can directly solve for you.

In general, you are asking about human/player management and interaction/engagement scenarios. I recommend you think about your project and what it means to be a player...or to be a human that engages with your project. With clear rules, then you can see what technologies or processes can help you enforce those rules.

Just as an example...maybe one of your rules is the human closest to the Kinect is the one that is engaging with the project. Do you want the foremost to always be immediately chosen? In my experience, it is usual for multiple people to try and interact at the same time with the Kinect. If they are side-by-side or moving around as a group, they will vary their distance from the Kinect by centimillimeters (or millimeters). Even if two people stand "still" they tend to lean and sway and therefore their distance changes. If two (or more) people are standing side-by-side, this could result in each their distance values to "compete". The Kinect will see this and a simple algorithm always picking the closest will switch to the people leaning closest. Perhaps this is an affect you want. And often people love trying to use/interact with the Kinect in unexpected ways. Thinking about and handling these interaction scenarios is part of the creativity and the unique way you will create your solutions. :-)

Could face recognition help implement one of your rules? I don't know as I don't know your project well enough. I will caution you that the face shape recognition on the v2 Kinect you are using has two sets of data. You can get a somewhat quick general shape and major points. The full face shape takes a very long time and I doubt anyone could use it in a fast real-time project. (The Kinect v1 sensor is actually faster in creating its face model...less accurate...but faster...the v1 is still a good piece of hardware)

Again, I don't know your rules...but maybe something could be done with color of clothing, or IR leds that only the Kinect sensor can "see". It is possible to create a mask of the RGB image based on the humans identified. Then perhaps you can see the color of clothing for a person and use that for identification. Examples of how to mask are on the "tips" tab in the help file provided with dp.kinect2.

Happy patching! :-)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

diablodale commented 5 years ago

🙂