Add some examples to PyKinectV2 to display 2D and 3D information using OpenCV and Open3D

[limgm]

Added some examples of 1) Displaying body, body index, color, align color, depth and IR images in 2D using OpenCV 2) Displaying coloured point cloud, joint and joint orientation in 3D using Open3D

Refer to the link below: https://github.com/limgm/PyKinect2

[avpai]

Hi @limgm ,

I wanted to create a mask and then remove the background. I used this code from your example:

       kinect.has_new_body_index_frame() and \
       kinect.has_new_color_frame():

              body_frame       = kinect.get_last_body_frame()
              body_index_frame = kinect.get_last_body_index_frame()                               
              color_frame      = kinect.get_last_color_frame()

              body_index_img   = body_index_frame.reshape(((depth_height, depth_width))).astype(np.uint8)
              #body_index_img  =  cv2.resize(body_index_img,(960, 540),None, fx=0.5, fy=0.5)
              color_img        = color_frame.reshape(((color_height, color_width, 4))).astype(np.uint8)
              color_img_resize = cv2.resize(color_img, None, fx=0.5, fy=0.5) # Resize (1080, 1920, 4) into half (540, 960, 4)
              #body_index_img  =  utils.color_body_index(kinect, body_index_img) # Add color to body_index_img

              body_index_img= cv2.bitwise_not(body_index_img)
              cv2.imshow('Bi',body_index_img)

              res = cv2.bitwise_and(color_img_resize,color_img_resize,mask = body_index_img)
              cv2.imshow('Res',res)

But I am getting this:

I know it has to do something with sizing, I am trying to figure it out, so any help would be appreciated. Thank you

[limgm]

Hi @avpai

Yes the sizing between body_index_img (424, 512) is different from your color_img_resize (540, 960)

Maybe you can try the modification below that uses align_color_img that is aligned to match the size of body_index_img (424,512). Note: It uses a utility function get_align_color_image from the utils_PyKinectV2.py

body_index_frame = kinect.get_last_body_index_frame()                               
color_frame      = kinect.get_last_color_frame()

body_index_img   = body_index_frame.reshape(((depth_height, depth_width))).astype(np.uint8)
color_img        = color_frame.reshape(((color_height, color_width, 4))).astype(np.uint8)

body_index_img = cv2.bitwise_not(body_index_img)
cv2.imshow('Bi',body_index_img)

align_color_img  = utils.get_align_color_image(kinect, color_img) # Add this to get the color image that is aligned to depth frame size
# res = cv2.bitwise_and(color_img_resize,color_img_resize,mask = body_index_img) # Replace this line with the below
res = cv2.bitwise_and(align_color_img,align_color_img,mask = body_index_img)
cv2.imshow('Res',res)

[avpai]

Got it, Thanks!!!

body_index_frame = kinect.get_last_body_index_frame()
color_frame      = kinect.get_last_color_frame()

body_index_img   = body_index_frame.reshape(((depth_height, depth_width))).astype(np.uint8) 
color_img        = color_frame.reshape(((color_height, color_width, 4))).astype(np.uint8)

align_color_img = utils.get_align_color_image(kinect, color_img)                               
body_index_img= cv2.bitwise_not(body_index_img)                             

#cv2.imshow('body index', body_index_img)                    # (424, 512)                               
#cv2.imshow('align color with body joints', align_color_img) # (424, 512)                               
res = cv2.bitwise_and(align_color_img,align_color_img,mask = body_index_img)
cv2.imshow('Res',res)

Any Idea on how I can use the pygame display and OpenCV? Like use a binary mask on the below image?

The frame is from the pykinect2 library, I wanted to use OpenCV on that frame since I have used a major chunk of my codes and program on that frame. Is there a way where I can open that frame in OpenCV and then use Binary masks on it?

[avpai]

Hi @limgm ,

I want to get only body index frame of only one body. I identified the body as : body = self._bodies.bodies[l] where i is the index of the body I want to track. Is there any way to do this? I tried : bodybody_index_frame = self._kinect.get_last_body_index_frame(l) but its not working. any suggestions?

[limgm]

Hi @avpai

Yes simply putting the index i into the function self._kinect.get_last_body_index_frame(i) doesn't work as you have to refer to the original code in pykinect2/PyKinectRuntime.py where the get_last_body_index_frame(self) doesn't take in extra argument.

Perhaps you can modify the draw_bodyframe function in utils_PyKinectV2.py to break the for loop when if body.is_tracked returns true so that you will only track maximum of one body.

To specify the index of the body you want to track will be difficult as each time a new body is detected the index may not be the same

Note: the index of body can be specified in body_frame.bodies[i]

def draw_bodyframe(body_frame, kinect, img):
    if body_frame is not None: 
        for i in range(0, kinect.max_body_count):
            body = body_frame.bodies[i]
            if body.is_tracked: 
                joints = body.joints
                joint_points = kinect.body_joints_to_depth_space(joints) # Convert joint coordinates to depth space 
                joint2D = get_joint2D(joints, joint_points) # Convert to numpy array format
                img = draw_joint2D(img, joint2D, colors_order[i])
                img = draw_bone2D(img, joint2D, colors_order[i])
                break # Add a break here

    return img

[avpai]

Thank you for the answer but I am trying to get the body frame not the skeletal drawing. Something Like this, possibly binary

[microprocessor6]

i hope you have got solution already but to provide solution as i struggle (and for assisting new searchers) :

. . . other things

self._kinect = PyKinectRuntime.PyKinectRuntime(

PyKinectV2.FrameSourceTypes_BodyIndex)

. . . other things

if self._kinect.has_new_body_index_frame(): bi_frame = self._kinect.get_last_body_index_frame() body_index_img = bi_frame.reshape(((424, 512))).astype(np.uint8) cv2.imshow('BI img', body_index_img)

. . . . .other things

PyKinectCV_bodyIndexWork.zip

[mrsahabu]

Any link for simple pykinect windows SDK 1.8?

[mohaddeseh67]

hi every one I am trying to collect skeleton data from kinect xbox1 by python kinect sensor is working properly on sdk but in code it printed nothing i just wrote a simple code to check if there is any tracking or not i am expecting to print hi when i am infront of sensor bur nothing i have been in front of it whit many different distances

import ctypes import pygame from pykinect2 import PyKinectV2, PyKinectRuntime

def get_3d_joint_data(kinect):

if kinect.has_new_body_frame():
    print('hi')
    bodies = kinect.get_last_body_frame()

    joint_data = []

    if bodies is not None:
        for body in bodies:

            print('YEEEEEEEEEEES')

kinect = PyKinectRuntime.PyKinectRuntime(PykinectV2.FrameSourceTypes_Body) while True:

get_3d_joint_data(kinect)