get skeleton joints to color coordinates (x,y)

silicatproject commented 4 years ago

Hi,

I've some problems to properly project datas (Skeleton joints) on top of the color and depth streams One solution i guess could be to convert the skeleton joints to color coordinates (x,y) Is there any technic to do this?

i've seen Microsoft sdk provide CoordinateMapper for that : CoordinateMapper’s job is to identify whether a point from the 3D space corresponds to a point in the color or depth 2D space

Sadly i'm only a maxer and can't use this CoordinateMapper ... Maybe you've a solution? Thank you very much Mathieu

diablodale commented 4 years ago

It's technically possible. However, the current version of dp.kinect2 doesn't expose that particular coordinate mapping for the skeleton. The current version does...

pixeltoskel depth coordinate -> real-world skeleton-space https://github.com/diablodale/dp.kinect2/wiki#messages-supported
face tracking can output 3D face points in depth/colorspace

You have a lot of knowledge in this topic area. So please ignore the following if you already tried it. You could simulate/approx this by configuring your GL camera to match the physical/lens attributes of the Kinect. Then when you draw anything using skeleton coordinates in GL, it could align with a depth/colormap you also project in GL with something like (jit.gl.videoplane) or (jit.gl.cornerpin). I did something similar once when I aligned a GL camera to match a real-world HD projector that was projecting its image on a dance stage.

If ideas like the above don't work for your needs...what functionality do you need?

A new method skeltodepth so that you can send arbitrary x,y,z coordinates that map into depth/color?
Is it new data appended on the existing skel message?
It is a new message skeldepth sent on every frame?
It is a 2x20 matrix that contains this new skeleton data?

Given time and understanding your specific needs, e.g. questions above, its technically possible for me to make an incremental update to dp.kinect2 to add feature(s) to map the skeleton coordinates to depth/color. I have some concern in testing the boundary cases and how the feature would interact with all the transform possibilities @distmeter, @flipx, @align, @position, @quat, @rotate, @rotatexyz, and @scale. For example, if some of these transform attributes are used, does a new skeltodepth expect the parameter values given to be the original or transformed x,y,z coordinates.

silicatproject commented 4 years ago

Hi, Thanks for the tricks, but on my side the result is not perfect when i configure the GL camera to match the physical/lens attributes of the camera... i would love to have something like "@skeleton 3" every frame /skel/userid/jointname x y z confidence qx qy qz qw colorX colorY (colorX&colorY are the coordinate in color space in pixel format or 0. to 1.) or the depthX/depthY could be useful too instead of colorX&colorY if it's easier... best Mathieu

diablodale commented 4 years ago

Continuing thinking, exploring ideas, so nothing is decided or bad... :-) What are your thoughts? Can you share more details on your specific use-cases/scenarios?

New methods `skeltocolor` and `skeltodepth`

I think two new methods skeltocolor and skeltodepth that accept x,y,z coordinates makes sense. I think those x,y,z coordinates have to be in the transformed @distmeter, @flipx, @align, @position, @quat, @rotate, @rotatexyz, and @scale space. This enables a patch to use any x,y,z coordinates given on the skel message, pointcloud, face, etc. directly with skeltocolor/depth.

I'm unsure what to do about invalid coordinates. It is possible for a skeleton x,y,z coordinate to be outside the view of the depth or colormap. My instinct is to return potentially impossible values and let the patch do its own boundary checking, because it might be true that the Kinect algorithms successfully predict where that point would be if it had larger depth/colormap. This would allow the patch to make their own choices.

Here is a simulation...

----------begin_max5_patcher----------
993.3oc6YlsabaCE.84I.4efPOaOfKZaB5Cs+EIsnvgiFFaYKQJPRkNNA8eu
bYj8DGZIZa5rUaXHMhRhWdO2MRpO+5WsJaqXOSkAdC3u.qV8YSKqbsYaY0TC
qx5o6a5nJ2Cl0H56Ybc1IGtolsW6twuc5ofQEaGnq8JFPeACP6jL5tqAr8sJ
cK+bv6GZ2y5zB0Urt2eSOz0xYMhQtqavSs1ty0qhsWdZN5lmcfpatvzUmIYM
Z+.OOubM7D.tnxdBU.smH4qgf+d5sT5q6Xtt6lNhO12x6XZmRgNpUwndpYnq
0+80uxd1b5jmNjdK3cfFQmPZNJj6TYATWR9bpKAQrJXNBttvn0UaVWAgPD17
6MOJc9CBtV09I2yhsv6n14zdee7GxVZWVpwzG5DltJHCplkA0abF4xBKCpbW
jSNV68CO80CL+aXf.HaKked1ihPxdpSzk2NnjFznYxyXb5Ve2.CQTxign3mC
hVGgW0DQQuPzIhxY+iAeecj7Hef1bEPccO.t1+ePruYdr6HMA6vNBU3xjgWj
6FyrAee8Odhg+H7i.s4oOKopseripM0RrUJzBeFSISMH3JVHLim06FWhu0sF
aXtMyI7GyJDeotC5YJE87v5b4r5rqb3AUFd3z2UU9tpxsp7QF4ZH.UBCptyW
VD5lE.BR7mpiIRJdXfCCCTxSq7oNfpqsgAHAY.bYSNl3xh3mNTD4Rd5T.+73
R.vftytfx2AN0LWG.ZMfr1bLDWPaVlKStF4E+j5ZLHYCL9WjSLHKJinhyzrk
QIFFnuUvPZ5c1A+if0ciHPw3PXqJ7bFn7jqthe9xvfBws5hHhjJqrfq3+Yfy
6xYi9BBtH73PDWQpRxulfa1U89m94yb+K6c9T30GunWyJSJ2X9q5WqE8hlce
NP04urDsGJQmMaFB9BQevDMe4vzW.5CcWDlaGDPUQLUFOw8aK1BEV9QZuCHI
Gn6FVekc+j0XvuaK3vzBt2OAftskvy8gDwBpy8GyQQP5Ka0mYDrrcukxwe0C
rze5LGkI2bbtjtMLrQwrJkM9THEoe9Rn3hv8cn6STb2uShaXauwc3hRLJalF
UGzTvQi7cL6GCgpaE7idH69DZep62BDs7bcEZQAVmVAhWTfUISfnXTP69EkJ
4UFiIDkL4Y2lmHj2cFUlIVyjKL2z3sowLBlFme6T4p6QkQovJuIlgPMIY5rc
s1QDnlN4EkQ0MpRTbSUTF07zJvEyDgPoUfKZBQgQ5ghKzggOxjpCuhWVlpqW
Jbd1Um3utk6ul3uVx9X6zqT3ahJMkQ0lZniReku8kGlrdVuvDmvGamBUbiAy
g+C1gyAH
-----------end_max5_patcher-----------

Color and Depth coordinates for skeleton joints on every frame

I hesitate putting more data on the skel message that is not in skeleton-space. At the same time, I don't have a clear enough view on the use cases for this feature.

What are the good and bad things about these options???

single message with all coordinates on it
separate messages for skel, skeldepth, and skelcolor

Single message with all coordinates on it

Already skel can have two formats.

standard
standard with orientation data in quat (10 values) or matrix (22 values)

It is possible a patch wants any combination of orientation, color, or depth. This means the message formats would increase:

standard (6 values)
standard with color coordinates (8 values)
standard with depth coordinates (8 values)
standard with color+depth coordinates (10 values)
standard with orientation (10 or 22 values)
standard with orientation, color coordinates (12 or 24 values)
standard with orientation, depth coordinates (12 or 24 values)
standard with orientation, color+depth coordinates (14 or 26 values)

When I see message formats have this many combinations, I hesitate that is a good approach. To control the format, the @skeleton attribute would now accept [0-8] as values. Probably good to make it a bitmap similar to @depthonlyplayers.

With any of the above formats, a patch needs to receive the skel message and slice/route all the data values to eventually get the colorX and colorY to manipulate/display it. Is this single message and the work to parse it better than having a separate skelcolor message?

Three separate messages for joints in skeleton, color, and depth-space

I could image a new attribute on dp.kinect2 @skeletoncolor that could be enabled if a patch wants to receive skelcolor messages. And @skeletondepth for the depth coordinate message.

skelcolor userid jointname colorX colorY confidence
skeldepth userid jointname depthX depthY confidence
skel userid jointname x y z confidence

This enables a patch to set @skeleton=0 and @skelcolor=1 so that dp.kinect2 only outputs a skelcolor message which can be quickly used to paint/index into the colormap.

The order of these three messageswould be guaranteed; probably skelcolor, skeldepth, and finally skel. Guaranteed order allows a patch to plan its messages and connectors. It could even construct its own all-coordinates-in-one-message using simple objects like (pack), (buddy), (join), etc.

Other Options?

Any other ideas or comments?

silicatproject commented 4 years ago

Hi, My idea was to add some video fx on the kinect color image depending of the skeleton. For example playing with blurred circles around the hands ... One other idea was to add some graphic elements on the color image, when two persons are close, some lines are connected between them (hands, shoulders, hips...) . Actually being able to create some more accurate interaction with the real color image on the background ... We could imagine many fun "AR" things with these features!

On my side my feeling go to "Three separate messages for joints in skeleton, color, and depth-space" maybe easier to manage in max ...

The "New methods skeltocolor and skeltodepth" seems good but i don't know the latency if we ask calculation with 4 characters&24 joints....

For the "impossible values" i completly agree with you to let the max patch manage the datas ;-)

Thank you so much for the discussion!

Best Mathieu

diablodale commented 4 years ago

Hi. I think it is possible to include some of these enhancements to dp.kinect2. Would you be willing to try and test some early builds? And could you help exercise/test it so we can be sure it meets your needs and performs well?

silicatproject commented 4 years ago

Hi, So cool! For sure i can test it! Here a nice work in progress i made with your external :-) https://www.youtube.com/watch?v=ZQDWquKLCzc&t=16s

Le jeu. 23 avr. 2020 à 16:33, Dale Phurrough notifications@github.com a écrit :

Hi. I think it is possible to include some of these enhancements to dp.kinect2. Would you be willing to try and test some early builds? And could you help exercise/test it so we can be sure it meets your needs and performs well?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/diablodale/dp.kinect2/issues/56#issuecomment-618430079, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADR3XULICGKMSIUTUOFAKHLROBGVJANCNFSM4LPMNY7A .

diablodale commented 4 years ago

I'm unsure of the guaranteed order of the three skeleton coordinate messages. These messages (0, 1, 2, or all 3) would be output before any of the matrices (pointcloud, player, ir, color, depth) since outlet order is right->left.

Approach 1 would use skel as the änchor"and consistently final message. The other two could be either order. Perhaps skelcolor, skeldepth, skel

Approach 2 would be somewhat the same as outlet output (right->left). Meaning skel, skelcolor, skeldepth.

My instinct tells me approach 1, but I don't have any facts to support it. Do you have an opinion on this topic?

silicatproject commented 4 years ago

I don't see any problems with these 2 scenarios . I like too the concept of skel as "anchor" easier to understand . But we can easily manage the datas with max so the 2 scenarios are perfect for me! Best,

Le ven. 24 avr. 2020 à 01:32, Dale Phurrough notifications@github.com a écrit :

I'm unsure of the guaranteed order of the three skeleton coordinate messages. These messages (0, 1, 2, or all 3) would be output before any of the matrices (pointcloud, player, ir, color, depth) since outlet order is right->left.

Approach 1 would use skel as the änchor"and consistently final message. The other two could be either order. Perhaps skelcolor, skeldepth, skel

Approach 2 would be somewhat the same as outlet output (right->left). Meaning skel, skelcolor, skeldepth.

My instinct tells me approach 1, but I don't have any facts to support it. Do you have an opinion on this topic?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/diablodale/dp.kinect2/issues/56#issuecomment-618723219, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADR3XUKYYVMEZIIZ6BGWIVDRODFYXANCNFSM4LPMNY7A .

diablodale commented 4 years ago

Hi. I have two questions.

I have a early build you can try. Would you contact me by email @diablodale and ask for it? It has the depthtoskel, depthtocolor, skeltocolor, skeltodepth methods that you can give one or more coordinates and it will return answers for them like pixeltoskel does today.
I am unsure how skelcolor and skeldepth messages on the 5th outlet output relate to @distmeter, @position, @quat, @rotate, @rotatexyz, and @scale. I do understand how to use @flipx and @align.

Here is the scenario.

A patch tells dp.kinect2 to transform joint data using one or more of the attributes @distmeter, @position, @quat, @rotate, @rotatexyz, and @scale.
A patch turns on output of the skelcolor messages for outlet 5.

Does the message skelcolor 1 l_hand give color coordinates of the un-transformed joint? Or does it apply the joint transform(s) and then calculate/output the color coordinate?

I can code either approach. But I don't yet see the benefit of the 2nd (transform+color). Because the joint's color coordinate that would be output would not align with the colormap matrix being output on outlet 2. So while I could do it...is that useful? I guess you could use @position or @quat to make the skeleton joints "beside" you...like a twin. And then use the output of skelcolor to draw the skeleton "twin" on the colormap matrix.

I'm unsure. What is your thinking on this?

silicatproject commented 4 years ago

Hi, I would prefer the message skelcolor 1 l_hand give color coordinates of the un-transformed joint . I don't see any advantage to apply the joint transforms and output the color coordinate for the moment... Maybe i need some practices of these features in order to see the possibilities... mathieu-

Le mar. 5 mai 2020 à 19:56, Dale Phurrough notifications@github.com a écrit :

Hi. I have two questions.

I have a early build you can try. Would you contact me by email @diablodale https://github.com/diablodale and ask for it? It has the depthtoskel, depthtocolor, skeltocolor, skeltodepth methods that you can give one or more coordinates and it will return answers for them like pixeltoskel does today.

I am unsure how skelcolor and skeldepth messages on the 5th outlet output relate to @distmeter, @position, @quat, @rotate, @rotatexyz, and @scale. I do understand how to use @flipx and @align.

Here is the scenario.

A patch tells dp.kinect2 to transform joint data using one or more of the attributes @distmeter, @position, @quat, @rotate, @rotatexyz, and @scale.

A patch turns on output of the skelcolor messages for outlet 5.

Does the message skelcolor 1 l_hand give color coordinates of the un-transformed joint? Or does it apply the joint transform(s) and then calculate/output the color coordinate?

I can code either approach. But I don't yet see the benefit of the 2nd (transform+color). Because the joint's color coordinate that would be output would not align with the colormap matrix being output on outlet 2. So while I could do it...is that useful? I guess you could use @position or @quat to make the skeleton joints "beside" you...like a twin. And then use the output of skelcolor to draw the skeleton "twin" on the colormap matrix.

I'm unsure. What is your thinking on this?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/diablodale/dp.kinect2/issues/56#issuecomment-624211904, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADR3XUM5ZYX3LSVM5HKBC3TRQBHOFANCNFSM4LPMNY7A .

diablodale commented 4 years ago

I've released the v1.2 feature update to include these ideas. Available to download at the usual locations like https://hidale.com/shop/dp-kinect2/

diablodale / dp.kinect2