Receiving only color_frame when using PyQt5 threads

[MetalStrikerXLR]

I have used this library for my project and it works beautifully. I then proceeded to use PyQt5 to design a GUI and display the output video in it. I created a thread and placed the kinect code inside. On execution, the kinect code enters the while loop but doe not cross the first if statement which checks for new frames. On further investigation i found out that that the if statement fails because only new color_frame has arrived. without PyQt5 all new frames are detected. Has anyone faced a similar problem or can anyone point out what might be going wrong?

[KonstantinosAng]

It would be helpful if you pasted the code, so we can help you. It seems that you have not initialised depth source in the kinect initialization. Check for the following line:

kinect = PyKinectRuntime.PyKinectRuntime(PyKinectV2.FrameSourceTypes_Color | PyKinectV2.FrameSourceTypes_Body)

The code above only initializes color and body frame data. You need to open the Depth or IR frame option like:

kinect = PyKinectRuntime.PyKinectRuntime(PyKinectV2.FrameSourceTypes_Color | PyKinectV2.FrameSourceTypes_Body | PyKinectV2.FrameSourceTypes_Depth | PyKinectV2.FrameSourceTypes_Infrared)

and there are also the body index data and sound data like:

kinect = PyKinectRuntime.PyKinectRuntime(PyKinectV2.FrameSourceTypes_Color | PyKinectV2.FrameSourceTypes_Body | PyKinectV2.FrameSourceTypes_Depth | PyKinectV2.FrameSourceTypes_BodyIndex | PyKinectV2.FrameSourceTypes_Infrared | PyKinectV2.FrameSourceTypes_Audio)

Also check my github repository that uses PyQt5 to create real time Pointclouds to see if it can solve your problem: https://github.com/KonstantinosAng/PyKinect2-PyQtGraph-PointClouds

[MetalStrikerXLR]

@KonstantinosAng Thank you for the quick reply. As you pointed out, i checked if I made a mistake in kinect initialization. Alas, there was no mistake there and the kinect has been initialized properly. Thank you for linking your repository on point clouds. I am currently going through it to find any solution that might be relevent to my problem. Just for reference here is the base code for my project:

GUI Code: `from PyQt5 import QtCore, QtGui, QtWidgets from PyQt5.QtCore import * import MainCode as MC

class CaptureKinect(QRunnable):

@pyqtSlot()
def run(self):
    print("Thread 1 start")
    MC.initKinect()
    MC.disconnectStatus = 0
    MC.initialization = 0
    print("Thread 1 complete")

class ProcessKinect(QRunnable):

@pyqtSlot()
def run(self):
    print("Thread 2 start")
    MC.processKinect()
    print("Thread 2 complete")

class Ui_MainWindow(object): def setupUi(self, MainWindow): MainWindow.setObjectName("MainWindow") MainWindow.resize(800, 600)

    # Worker thread setup
    self.threadpool = QThreadPool()
    print("Multi-Threading with maximum %d threads" % self.threadpool.maxThreadCount())

    # GUI Design Code Goes here:
    self.centralwidget = QtWidgets.QWidget(MainWindow)
    self.centralwidget.setObjectName("centralwidget")
    self.verticalLayout_2 = QtWidgets.QVBoxLayout(self.centralwidget)
    self.verticalLayout_2.setObjectName("verticalLayout_2")
    self.verticalLayout = QtWidgets.QVBoxLayout()
    self.verticalLayout.setObjectName("verticalLayout")
    self.imageDisplay = QtWidgets.QLabel(self.centralwidget)
    self.imageDisplay.setText("")
    self.imageDisplay.setScaledContents(True)
    self.imageDisplay.setObjectName("imageDisplay")
    self.verticalLayout.addWidget(self.imageDisplay)
    self.horizontalLayout = QtWidgets.QHBoxLayout()
    self.horizontalLayout.setObjectName("horizontalLayout")
    self.conButton = QtWidgets.QPushButton(self.centralwidget)
    self.conButton.setObjectName("conButton")
    self.horizontalLayout.addWidget(self.conButton)
    self.disButton = QtWidgets.QPushButton(self.centralwidget)
    self.disButton.setObjectName("disButton")
    self.horizontalLayout.addWidget(self.disButton)
    self.verticalLayout.addLayout(self.horizontalLayout)
    self.verticalLayout_2.addLayout(self.verticalLayout)
    MainWindow.setCentralWidget(self.centralwidget)

    # Self Design Editions
    self.conButton.setEnabled(True)
    self.disButton.setEnabled(False)

    self.timer = QtCore.QTimer()
    self.timer.timeout.connect(self.updateGUI)
    self.timer.start(250)

    #### Signal Connection ####
    self.conButton.pressed.connect(self.connectKinect)
    self.disButton.pressed.connect(self.disconnectKinect)

    ###########################

    self.retranslateUi(MainWindow)
    QtCore.QMetaObject.connectSlotsByName(MainWindow)

def retranslateUi(self, MainWindow):
    _translate = QtCore.QCoreApplication.translate

    # Also add design code here
    MainWindow.setWindowTitle(_translate("MainWindow", "Kinect GUI v1"))
    self.conButton.setText(_translate("MainWindow", "Connect"))
    self.disButton.setText(_translate("MainWindow", "Disconnect"))

def connectKinect(self):
    # Worker thread handles Kinect Code
    thread1 = CaptureKinect()
    thread2 = ProcessKinect()
    self.threadpool.start(thread1)
    self.threadpool.start(thread2)
    self.conButton.setEnabled(False)
    self.disButton.setEnabled(True)

def disconnectKinect(self):
    print("Disconnected")
    self.conButton.setEnabled(True)
    self.disButton.setEnabled(False)
    MC.disconnectStatus = 1

def updateGUI(self):

    # Add repeating GUI code here
    if MC.initialization == 1:
        self.imageDisplay.setPixmap(QtGui.QPixmap("Kinect_Image.jpg"))
    else:
        self.imageDisplay.setPixmap(QtGui.QPixmap("logo.png"))

GUI Execution:

if name == "main": import sys

app = QtWidgets.QApplication(sys.argv)
MainWindow = QtWidgets.QMainWindow()
ui = Ui_MainWindow()
ui.setupUi(MainWindow)
MainWindow.show()
sys.exit(app.exec_())

` Main Kinect Code:

`import cv2 import math as mt import Kinect as Kv2 import ImageProcessing as IP import numpy as np import glob from pykinect2 import PyKinectV2 from pykinect2 import PyKinectRuntime import utils_PyKinectV2 as utils from open3d import *

disconnectStatus = 0 kinect = 0 initialization = 0 color_img = 0 joint = 0 jointPoints = 0

def initKinect(): #############################

Kinect runtime object

#############################
global disconnectStatus
global kinect
global initialization
global color_img
global joint
global jointPoints

kinect = PyKinectRuntime.PyKinectRuntime(PyKinectV2.FrameSourceTypes_Body |
                                         PyKinectV2.FrameSourceTypes_BodyIndex |
                                         PyKinectV2.FrameSourceTypes_Color |
                                         PyKinectV2.FrameSourceTypes_Depth |
                                         PyKinectV2.FrameSourceTypes_Infrared)
color_width, color_height = Kv2.getColorDimension(kinect)

########################
### Main Kinect Loop ###
########################

while True:
    if kinect.has_new_body_frame() and \
            kinect.has_new_body_index_frame() and \
            kinect.has_new_color_frame() and \
            kinect.has_new_depth_frame() and \
            kinect.has_new_infrared_frame():

        ##############################
        ### Get images from camera ###
        ##############################
        color_img, joint, jointPoints = Kv2.getKinectFrames(kinect, color_height, color_width)

        cv2.imwrite("Kinect_Image.jpg", color_img)
        initialization = 1

    if disconnectStatus == 1:
        break

kinect.close()
print("Kinect Shutdown!")

def processKinect(): global disconnectStatus global kinect global initialization global color_img global joint global jointPoints

shoulder_distance = 300
waist_distance = 300

SJR_x = 0
SJR_y = 0
SJL_x = 0
SJL_y = 0
Nk_x = 0
Nk_y = 0
HPL_x = 0
HPL_y = 0
HPR_x = 0
HPR_y = 0
SPB_x = 0
SPB_y = 0

# load cloth images
shirts = [cv2.imread(file) for file in glob.glob('Datasets/Shirts/*.png')]
pants = [cv2.imread(file) for file in glob.glob('Datasets/Pants/*.png')]
cropped_shirts = []
cropped_pants = []

# process and store cloth images
for items in shirts:
    shirt_processed = IP.removeClothBG(items)
    cropped_shirts.append(shirt_processed)

for items in pants:
    pant_processed = IP.removeClothBG(items)
    cropped_pants.append(pant_processed)

while True:

    if initialization == 1:

        #################################################
        ### Extract and Draw 2D joints on single body ###
        #################################################

        if joint != 0:

            joint2D = Kv2.getBodyJoints(joint, jointPoints)
            SJR_x = int(joint2D[8, 0] * 3.75) + 30
            SJR_y = int(joint2D[8, 1] * 2.547) - 50
            SJL_x = int(joint2D[4, 0] * 3.75) + 30
            SJL_y = int(joint2D[4, 1] * 2.547) - 50
            Nk_x = int(joint2D[2, 0] * 3.75) + 30
            Nk_y = int(joint2D[2, 1] * 2.547) - 50
            HPL_x = int(joint2D[12, 0] * 3.75)
            HPL_y = int(joint2D[12, 1] * 2.547) - 50
            HPR_x = int(joint2D[16, 0] * 3.75) + 60
            HPR_y = int(joint2D[16, 1] * 2.547) - 50
            SPB_x = int(joint2D[0, 0] * 3.75) + 30
            SPB_y = int(joint2D[0, 1] * 2.547) - 50

            print(SPB_x)

            color_img = Kv2.drawJointsFull(color_img, joint2D)

            distance1 = mt.sqrt(mt.pow(SJL_x - SJR_x, 2))
            distance2 = mt.sqrt(mt.pow(HPL_x - HPR_x, 2))

        else:
            distance1 = 300
            distance2 = 300

        if distance1 <= 0:
            shoulder_distance = shoulder_distance
        else:
            shoulder_distance = distance1

        if distance2 <= 0:
            waist_distance = waist_distance
        else:
            waist_distance = distance2

        ##############################################
        ### Resizing Cloth image according to user ###
        ##############################################

        resized_shirt = IP.image_resize(cropped_shirts[1], width=int(shoulder_distance + 60))
        resized_pant = IP.image_resize(cropped_pants[3], width=int(waist_distance))

        shirt_y = np.shape(resized_shirt)[0]
        shirt_x = np.shape(resized_shirt)[1]
        pant_y = np.shape(resized_pant)[0]
        pant_x = np.shape(resized_pant)[1]

        #####################################################
        ### Add resized image to Background removed Image ###
        #####################################################

        # Offsets:
        overlay = np.zeros_like(color_img)
        overlay_y = np.shape(overlay)[0]
        overlay_x = np.shape(overlay)[1]

        # Pant addition
        cutoff_pants = (SPB_y + pant_y) - overlay_y
        if cutoff_pants <= 0:
            cutoff_pants = 0
        else:
            cutoff_pants = cutoff_pants

        if (HPL_x - 20) > 0 and SPB_y > 0 and (HPR_x + 70) < overlay_x:
            overlay[SPB_y:SPB_y + pant_y - cutoff_pants,
            int(SPB_x - (waist_distance / 2)):int(SPB_x - (waist_distance / 2)) + pant_x] = resized_pant[
                                                                                            0:pant_y - cutoff_pants,
                                                                                            0:pant_x]

        # Shirt addition
        if (SJL_x - 20) > 0 and Nk_y > 0 and (SJR_x + 70) < overlay_x:
            overlay[Nk_y:Nk_y + shirt_y, int(Nk_x - (shoulder_distance / 2) - 30):int(
                Nk_x - (shoulder_distance / 2) - 30) + shirt_x] = resized_shirt[0:shirt_y, 0:shirt_x]

        Output = IP.mergeImages(overlay, color_img)

        #####################################
        ## Display 2D images using OpenCV ###
        #####################################

        # cv2.imshow('Color + Depth Image', color_img)
        cv2.imwrite("Kinect_Image.jpg", Output)

    if disconnectStatus == 1:
        break

`

[KonstantinosAng]

I see that you only use Color, Depth and Body data so you should disable the body index and infrared data in kinect for better performance as:

kinect = PyKinectRuntime.PyKinectRuntime(PyKinectV2.FrameSourceTypes_Color |
PyKinectV2.FrameSourceTypes_Depth | PyKinectV2.FrameSourceTypes_Body)

Also I believe that the Kinect requires some time to obtain the Frames. I think that the first frames arriving are the depth and then the color. Thus, give it a little time and change the if statement to:

if kinect.has_new_depth_frame()  and \
            kinect.has_new_color_frame() and \
            kinect.has_new_body_frame():

[MetalStrikerXLR]

@KonstantinosAng Thank you for pointing this out. This has fixed the issue as well improved the output video fps.

Though, I am still curious as to why when executing the code in a separate thread, the kinect fails to obtain all frames together. Is the data too much to be handled when the code is executed in a thread? because if i run the code independently without a thread the code executes perfectly and the kinect is able to obtain all frames together.

[KonstantinosAng]

Great, glad to hear that it is working.

If you take a look inside the pykinect2/PyKinectRuntime it already uses threads to obtain the rgb and color frames and then store them. So when you call kinect.has_new_color_frame() it checks to see the time since the last frame was obtained. So if you are running threads in threads puts a strain on the algorithm and slows it down. Also python is not the fastest language to use for threads. Also if you have a 4 core processor with 8 threads you can only run like 8 real threads and the rest will run simulteanously, meaning that if you run 9 threads, one of the 8 threads will have to run two python threads simultaenously.

In order to keep python fast make sure to clean the code and only use what you need.

[MetalStrikerXLR]

Thank you for your help and this insight, really appreciate it.

[KonstantinosAng]

You are welcome, glad to hear it. Make sure to close the issue.