Pykinect2 + OpenGL?

[avpai]

Has anyone tried integrating OpenGL to Pykinect2 to get a 3D view? Is it possible since Pykinect2 doesn't provide point cloud data?

[nitzel]

It should be possible. I've plotted my point clouds that I got from PyKinect2 via matplotlib (which has some basic support for 3d plots).

PyKinect can indeed give you 3D-coordinates for every pixel of the depth image, so I would call that a point cloud.

3D coordinates are called camera space points (because they relative to the camera - the camera being at x/y/z = 0/0/0 iirc)

class Kinect():
    """ Offers some more functionality and access via the
        PyKinect2 interface """
    def __init__(self):
        # From the kinect we only need the depth frame
        kinect_frame_types = PyKinectV2.FrameSourceTypes_Depth
        self._kinect = PyKinectRuntime.PyKinectRuntime(kinect_frame_types)

        # store height widht of kinect camera
        self.WIDTH = self._kinect.depth_frame_desc.Width
        self.HEIGHT = self._kinect.depth_frame_desc.Height

        self.has_new_depth_frame = self._kinect.has_new_depth_frame
        self.get_last_depth_frame = self._kinect.get_last_depth_frame

        # create C-style Type
        TYPE_CameraSpacePoint_Array = PyKinectV2._CameraSpacePoint * (self.WIDTH * self.HEIGHT)
        # create a buffer to reuse when mapping to CameraSpacePoint
        self.depth_to_csp_buffer = TYPE_CameraSpacePoint_Array()

    def close(self):
        self._kinect.close()

    def __enter__(self):
        return self

    def __exit__(self, type, value, traceback):
        self.close()

    def __del__(self):
        self.close()

    def map_depth_frame_to_camera_space_points(self, depth_frame=None):
        # numpy.ndarray(424*512) -> numpy.ndarray(424x512x3)
        """ From a depth-frame (numpy.ndarray) it calculates the
            coordinates in cameraspace. The returned array
            holds 3 floats for each coordinate.
        """
        L = self.WIDTH * self.HEIGHT
        if depth_frame is None:
            depth_frame = self.get_last_depth_frame()
        # size must be what the kinect offers as depth frame
        assert L == depth_frame.size
        # cstyle pointer to our 1d depth_frame data
        ptr_depth = np.ctypeslib.as_ctypes(depth_frame.flatten())
        # calculate cameraspace coordinates
        error_state = self._kinect._mapper.MapDepthFrameToCameraSpace(
                                                                L, ptr_depth,
                                                                L, self.depth_to_csp_buffer)
        # 0 means everythings ok, otherwise failed!
        if error_state:
            raise "Could not map depth frame to camera space! "
            + str(error_state)

        # convert back from ctype to numpy.ndarray
        pf_csps = ctypes.cast(self.depth_to_csp_buffer, ctypes.POINTER(ctypes.c_float))
        data = np.ctypeslib.as_array(pf_csps, shape=(self.HEIGHT, self.WIDTH,
                                                     3))
        del pf_csps, ptr_depth, depth_frame
        return np.copy(data)

[avpai]

So use something like:

        # numpy.ndarray(424*512) -> numpy.ndarray(424x512x3)
        """ From a depth-frame (numpy.ndarray) it calculates the coordinates in cameraspace. The returned array holds 3 floats for each coordinate. """

        L = self._kinect.depth_frame_desc.Width*self._kinect.depth_frame_desc.Height
        if depth_frame is None:
            depth_frame = self._kinect.get_last_depth_frame()     
        # size must be what the kinect offers as depth frame
        assert L == depth_frame.size
        # cstyle pointer to our 1d depth_frame data
        ptr_depth = np.ctypeslib.as_ctypes(depth_frame.flatten())
        # calculate cameraspace coordinates
        error_state = self._kinect._mapper.MapDepthFrameToCameraSpace(L, ptr_depth,L, self._kinect.depth_to_csp_buffer)

        # 0 means everythings ok, otherwise failed!
        if error_state:
            raise "Could not map depth frame to camera space! "
            + str(error_state)

        # convert back from ctype to numpy.ndarray
        pf_csps = ctypes.cast(self.depth_to_csp_buffer, ctypes.POINTER(ctypes.c_float))
        data = np.ctypeslib.as_array(pf_csps, shape=(self.HEIGHT, self.WIDTH,
                                                     3))
        del pf_csps, ptr_depth, depth_frame
        return np.copy(data)

and

 ax = plt.axes(projection='3d')
  X,Y,Z = self.map_depth_frame_to_camera_space_points                        
  ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap='viridis', edgecolor='none');

Am I doing it right? But I keep getting attribute error for "self._kinect.depth_to_csp_buffer". says that there is no such attribute.

[nitzel]

depth_to_csp_buffer is not part of the MS Kinect class, I've added that to my class manually. (see the __init__ constructor).

You can define TYPE_CameraSpacePoint_Array = PyKinectV2._CameraSpacePoint * (self.WIDTH * self.HEIGHT) and use TYPE_CameraSpacePoint_Array instead of self._kinect.depth_to_csp_buffer.

[avpai]

hi, I removed the errors, but I could'nt get it to plot. Is there a sample code or code snippet that you could share for the plotting array?

[nitzel]

Have you tried plotting the 2D projection? So only the X/Y coordinates of each pixel. Then you should already be able to recognise what's in front of your camera. If that does work, move on the the 3D plotting which is a whole lot tricker as you have to get the viewport, angle, etc right.

...
csps = my_kinect.map_depth_frame_to_camera_space_points()

These csps are basically x1,y1,z1,x2,y2,z2,... not [[x1, x2,...][y1, y2, ...][z1, z2, ...]]. So X,Y,Z = csps doesn't work.

I'm sorry, I haven't touched this code since 2017 and no longer have a Kinect to test code against - so no examples and no recently tested code from me.

[KonstantinosAng]

I just created a repository that uses the PyQtGraph, OpenGL and PyKinect2 to create real - time point clouds like the KinectSDK. It is fully optimized, using numpy that runs in C and can create point clouds up to 60+ frames. It can create point clouds using the depth camera, the rgb camera, only the body index frame or the skeleton track joints. Also it can simultaneously create all of the above point clouds. In addition, there are 4 track bars that can be used to change the point size and color.

Check it out, and if you like it maybe give it a star to keep track of the updates.

https://github.com/KonstantinosAng/PyKinect2-PyQtGraph-PointClouds

If you cannot understand how to use it or if you run into any problems feel free to ask me.

[lixryjz]

when I use MapDepthFrameToCameraSpace，I find it get points all is -inf？How to salve it？

[nitzel]

@lixryjz do the values you get look reasonable before you do MapDepthFrameToCameraSpace? If so, check you're calling MapDepthFrameToCameraSpace with the right arguments. If not, maybe your Kinect is broken.

However, I haven't touched this project since 2017, so I'll be unsubscribing from this thread as I don't remember enough details to help ^^