How to successfully stream both RGB and Depth using CUDA?

[jiapei100]

Overview Description:

Version, Platform, and Hardware Bug Found:

1. git log -1 --oneline: Please just checkout my cloned version at Longer Vision libfreenect2
2. uname -a: ```$ uname -a Linux XXXXXXXXXX 6.8.0-38-generic #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 7 15:25:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

3. lsusb -t: ```$ lsusb | grep Microsoft Bus 009 Device 011: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor
   Bus 010 Device 004: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor
   Bus 010 Device 005: ID 045e:02c4 Microsoft Corp. Xbox NUI Sensor

4. lspci -nn: N/A

Steps to Reproduce:

1. Just git clone https://github.com/LongerVision/libfreenect2
2. and build it

Actual Results:

• ./bin/Protonect cpu

  $ ./bin/Protonect cpu
  Version: 0.2.0
  Environment variables: LOGFILE=<protonect.log>
  Usage: ./bin/Protonect [-gpu=<id>] [gl | cl | clkde | cuda | cudakde | cpu] [<device serial>]
      [-noviewer] [-norgb | -nodepth] [-help] [-version]
      [-frames <number of frames to process>]
  To pause and unpause: pkill -USR1 Protonect
  libva info: VA-API version 1.20.0
  libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
  libva info: va_openDriver() returns -1
  [Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
  [Info] [Freenect2Impl] enumerating devices...
  [Info] [Freenect2Impl] 23 usb devices connected
  [Info] [Freenect2Impl] found valid Kinect v2 @10:5 with serial 178602434347
  [Info] [Freenect2Impl] found 1 devices
  [Info] [Freenect2DeviceImpl] opening...
  [Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20*16384 ir: 60*8*33792
  [Info] [Freenect2DeviceImpl] opened
  [Info] [Freenect2DeviceImpl] starting...
  [Debug] [Freenect2DeviceImpl] status 0x090000: 9757
  [Debug] [Freenect2DeviceImpl] status 0x090000: 9731
  [Info] [Freenect2DeviceImpl] submitting rgb transfers...
  [Info] [Freenect2DeviceImpl] submitting depth transfers...
  [Debug] [DepthPacketStreamParser] not all subsequences received 0
  [Info] [Freenect2DeviceImpl] started
  device serial: 178602434347
  device firmware: 4.0.3916.0
  [Debug] [DepthPacketStreamParser] not all subsequences received 992
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Info] [DepthPacketStreamParser] 15 packets were lost
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packet
  [Debug] [DepthPacketStreamParser] skipping depth packe
  ......

• ./bin/Protonect cuda

  $ ./bin/Protonect cuda
  Version: 0.2.0
  Environment variables: LOGFILE=<protonect.log>
  Usage: ./bin/Protonect [-gpu=<id>] [gl | cl | clkde | cuda | cudakde | cpu] [<device serial>]
      [-noviewer] [-norgb | -nodepth] [-help] [-version]
      [-frames <number of frames to process>]
  To pause and unpause: pkill -USR1 Protonect
  [Info] [CudaDepthPacketProcessorImpl] device 0: NVIDIA GeForce RTX 3090 @ 1725MHz Memory 24219MB
  [Info] [CudaDepthPacketProcessorImpl] selected device 0
  [Error] [CudaDepthPacketProcessorImpl] cudaGetLastError(): the provided PTX was compiled with an unsupported toolchain.
  libva info: VA-API version 1.20.0
  libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
  libva info: va_openDriver() returns -1
  [Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
  [Info] [Freenect2Impl] enumerating devices...
  [Info] [Freenect2Impl] 23 usb devices connected
  [Info] [Freenect2Impl] found valid Kinect v2 @10:5 with serial 178602434347
  [Info] [Freenect2Impl] found 1 devices
  [Info] [Freenect2DeviceImpl] opening...
  [Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20*16384 ir: 60*8*33792
  [Info] [Freenect2DeviceImpl] opened
  [Info] [Freenect2DeviceImpl] starting...
  [Debug] [Freenect2DeviceImpl] status 0x090000: 9771
  [Debug] [Freenect2DeviceImpl] status 0x090000: 9771
  [Info] [Freenect2DeviceImpl] submitting rgb transfers...
  [Info] [Freenect2DeviceImpl] submitting depth transfers...
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Info] [Freenect2DeviceImpl] started
  device serial: 178602434347
  device firmware: 4.0.3916.0
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL
  [Error] [DepthPacketStreamParser] Packet buffer is NULL

Expected Results:

• For ./bin/Protonect cpu, I can successfully grabbed all images, including: bgr, depth, etc. However, there are tooooo many depth image skipped: [Debug] [DepthPacketStreamParser] skipping depth packet.
• For ./bin/Protonect cuda, I can NOT even successfully grab the image, there is NOTHING displayed in the GUI window. And from the console, full of [Error] [DepthPacketStreamParser] Packet buffer is NULL .

Reproducibility:

Additional Information: I didn't built libfreenect2 using OpenNI2 Driver .

Can anybody give me a hand?

[jiapei100]

Use ./bin/Protonect gpu=0 instead. It's working properly with GPU now. BUT, for CPU, there are still quite a lot of [Debug] [DepthPacketStreamParser] skipping depth packet. Anyway, close it.