I can get rgb image, but no depth and rgb image

[horxianfeng]

I am using ubutu 14.04. Ros Indigo. intel i5. Nvidia 840.

So far, i install libreenect2 succesfully. However, when i try to run rosrun kinect2_bridge kinect2_bridge, i face the error below( I had tried to update my kernel and etc, but i still cant solve the problem):

xfhor@xfhor-TP500LN:~/libfreenect2/examples/protonect$ ./bin/Protonect gl [Freenect2Impl] enumerating devices... [Freenect2Impl] 11 usb devices connected [Freenect2Impl] found valid Kinect v2 @3:7 with serial 508442542542 [Freenect2Impl] found 1 devices [Freenect2DeviceImpl] opening... [Freenect2DeviceImpl] opened [Freenect2DeviceImpl] starting... [Freenect2DeviceImpl] ReadData0x14 response 92 bytes of raw data 0x0000: 00 00 12 00 00 00 00 00 01 00 00 00 43 c1 1f 41 2e2e2e2e2e2e2e2e2e2e2e2e432e2e41 0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e 0x0020: 0a 21 33 55 c2 00 17 20 00 08 00 00 10 00 00 00 2e2133552e2e2e202e2e2e2e2e2e2e2e 0x0030: 00 01 00 00 00 10 00 00 00 00 80 00 01 00 00 00 2e2e2e2e2e2e2e2e2e2e802e2e2e2e2e 0x0040: 31 33 00 00 00 08 0d 01 47 4d 41 39 38 33 2e 31 31332e2e2e2e2e2e474d413938332e31 0x0050: 58 00 00 00 00 00 00 00 00 00 00 00 582e2e2e2e2e2e2e2e2e2e2e

[Freenect2DeviceImpl] ReadStatus0x090000 response 4 bytes of raw data 0x0000: 01 26 00 00 2e262e2e

[Freenect2DeviceImpl] ReadStatus0x090000 response 4 bytes of raw data 0x0000: 03 26 00 00 2e262e2e

[Freenect2DeviceImpl] enabling usb transfer submission... [Freenect2DeviceImpl] submitting usb transfers... [Freenect2DeviceImpl] started device serial: 508442542542 device firmware: 4.3.3912.0.7 [DepthPacketStreamParser::onDataReceived] not all subsequences received 0 [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [OpenGLDepthPacketProcessor] avg. time: 7.35344ms -> ~135.991Hz [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [TurboJpegRgbPacketProcessor] avg. time: 29.9839ms -> ~33.3513Hz [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [OpenGLDepthPacketProcessor] avg. time: 7.57655ms -> ~131.986Hz [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [TurboJpegRgbPacketProcessor] avg. time: 29.9674ms -> ~33.3696Hz [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [OpenGLDepthPacketProcessor] avg. time: 7.32483ms -> ~136.522Hz [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [TurboJpegRgbPacketProcessor] avg. time: 29.5135ms -> ~33.8828Hz [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [OpenGLDepthPacketProcessor] avg. time: 7.50409ms -> ~133.261Hz [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet! [RgbPacketStreamParser::onDataReceived] skipping rgb packet!

[xlz]

Seems another duplicate of #117 #237 of trouble with OpenGL processor.

In #117 @christiankerl said the OpenGL processor requires 3.3 (I don't know why though). Check your OpenGL version with

sudo apt-get install mesa-utils
glxinfo | grep "OpenGL version"

We really need to check the version in the OpenGL processor to quit on unsupported version instead of showing wrong output. @larshg

[horxianfeng]

string: 4.5.0 NVIDIA 352.09

is it a problem? @xlz

[horxianfeng]

I accidentally fix it..Thx

[xlz]

What do you mean by accidentally fix it?

[xlz]

I confirm this issue with OpenGL processor.

Black depth output on Intel Haswell i7-4600U/HD 4400, with Ubuntu 14.04:

OpenGL core profile version string: 3.3 (Core Profile) Mesa 10.7.0-devel (git-86a74e9 2015-06-03 trusty-oibaf-ppa)
OpenGL version string: 3.0 Mesa 10.7.0-devel (git-86a74e9 2015-06-03 trusty-oibaf-ppa)

or Debian stretch:

OpenGL core profile version string: 3.3 (Core Profile) Mesa 10.5.5
OpenGL version string: 3.0 Mesa 10.5.5

[robotsorcerer]

How do you upgrade from mesa 3.0 to 3.3? I followed the instructions on Mesa3D and installed Mesa 10.5.6 which I take to be Mesa 3.3 but it doesn't seem to work upgrade after compiling with both Scons and autoconf. When I try $glxinfo | grep "OpenGL version", I still get OpenGL version string: 3.0 Mesa 10.3.2

Am I missing something?

EDIT 1: $ glxinfo | grep "OpenGL core" gives:

~~OpenGL core profile version string: 3.3 (Core Profile) Mesa 10.3.2 OpenGL core profile shading language version string: 3.30 OpenGL core profile context flags: (none) OpenGL core profile profile mask: core profile OpenGL core profile extensions:~~

EDIT 2: $ glxinfo | grep "OpenGL"

OpenGL vendor string: Intel Open Source Technology Center OpenGL renderer string: Mesa DRI Intel(R) Haswell Mobile OpenGL core profile version string: 3.3 (Core Profile) Mesa 10.3.2 OpenGL core profile shading language version string: 3.30 OpenGL core profile context flags: (none) OpenGL core profile profile mask: core profile OpenGL core profile extensions: OpenGL version string: 3.0 Mesa 10.3.2 OpenGL shading language version string: 1.30 OpenGL context flags: (none) OpenGL extensions:

[horxianfeng]

@lakehanne i think most probably i upgraded my nvidea driver

[sketchc89]

Same issue. Only see RGB, no depth, IR.

On clean install of Ubuntu 14.04 and latest master. Libfreenect installs without issue. Kernel updated uname -a returns Linux sketchc89-XPS13-9333 4.0.0-040000-generic #201504121935 SMP Sun Apr 12 23:58:08 UTC 2015 i686 i686 i686 GNU/Linux

Ran sudo apt-get update mesa-utils

glxinfo | grep "OpenGL Core" returns nothing glxinfo | grep "OpenGL" returns

OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) Haswell Mobile x86/MMX/SSE2
OpenGL core profile version string: 3.3 (Core Profile) Mesa 10.3.2
OpenGL core profile shading language version string: 3.30
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.0 Mesa 10.3.2
OpenGL shading language version string: 1.30
OpenGL context flags: (none)
OpenGL extensions:

Haven't tried using OpenCL.

[sketchc89]

Running ./libfreenect2/examples/protonect/bin/Protonect cl returns ...

[Freenect2Impl] found valid Kinect v2 @2:5 with serial 013955545147
[Freenect2Impl] found 1 devices
OpenCL pipeline is not supported!
[Freenect2DeviceImpl] opening...
[Freenect2DeviceImpl] opened
[Freenect2DeviceImpl] starting...
[Freenect2DeviceImpl] ReadData0x14 response

... with similar issue. Note OpenCL pipeline is not supported dpkg -s beignet-dev

Package: beignet-dev
Status: install ok installed
Priority: extra
Section: libdevel
Installed-Size: 41
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: i386
Source: beignet
Version: 0.3-1
Depends: beignet (= 0.3-1)
Description: Intel OpenCL library
 OpenCL (Open Computing Language) is a multivendor open standard for
 general-purpose parallel programming of heterogeneous systems that include
 CPUs, GPUs and other processors.
 .
 This package contains the development files for directly linking against
 the Intel implementation.
Original-Maintainer: Simon Richter <sjr@debian.org>
Homepage: http://cgit.freedesktop.org/beignet/

[xlz]

beignet-dev 0.3 is not new enough. You need 1.0+, maybe from sudo apt-add-repository ppa:pmjdebruijn/beignet-testing.

[sketchc89]

Upgraded beignet-dev per instructions

Package: beignet-dev
Status: install ok installed
Priority: extra
Section: libdevel
Installed-Size: 43
Maintainer: Debian OpenCL Maintainers <pkg-opencl-devel@lists.alioth.debian.org>
Architecture: i386
Source: beignet
Version: 1.0.1-2pmjdebruijn1~trusty
Depends: beignet-opencl-icd (= 1.0.1-2pmjdebruijn1~trusty), ocl-icd-opencl-dev
Description: OpenCL library for Intel GPUs (development files)
 OpenCL (Open Computing Language) is a multivendor open standard for
 general-purpose parallel programming of heterogeneous systems that include
 CPUs, GPUs and other processors.
 .
 This package contains the development files for Intel extensions.
Homepage: http://www.freedesktop.org/wiki/Software/Beignet/

Receive same error when running ./bin/Protonect cl I then ran cmake libfreenect2/examples/protonect/CMakeLists.txt fine make now gives error

In file included from /usr/include/CL/cl.hpp:217:0,
                 from /home/sketchc89/ros/libfreenect2/examples/protonect/src/opencl_depth_packet_processor.cpp:48:
/usr/lib/gcc/i686-linux-gnu/4.8/include/emmintrin.h:31:3: error: #error "SSE2 instruction set not enabled"
 # error "SSE2 instruction set not enabled"
   ^
In file included from /usr/include/CL/cl.hpp:218:0,
                 from /home/sketchc89/ros/libfreenect2/examples/protonect/src/opencl_depth_packet_processor.cpp:48:
/usr/lib/gcc/i686-linux-gnu/4.8/include/xmmintrin.h:31:3: error: #error "SSE instruction set not enabled"
 # error "SSE instruction set not enabled"
   ^
In file included from /home/sketchc89/ros/libfreenect2/examples/protonect/src/opencl_depth_packet_processor.cpp:48:0:
/usr/include/CL/cl.hpp: In function ‘void cl::detail::fence()’:
/usr/include/CL/cl.hpp:1041:38: error: ‘_mm_mfence’ was not declared in this scope
     inline void fence() { _mm_mfence(); }

[xlz]

http://stackoverflow.com/questions/16410149/error-sse2-instruction-set-not-enabled-when-including-emmintrin-h

[sketchc89]

I tried including the -msse -msse2 -msse3 flags both in the enable C++11 block and OpenCL block per the stackoverflow and #138

SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -msse3") 
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse3")

and receive a ton of opencl_depth_packetprocessor.cpp.o errors regarding OpenCLDepthPacketProcessor does not name a type impl not declared in this scope invalid use of this in member function

[sketchc89]

I also tried with and without flags for c++11.

[staljbs]

@xlz I have beignet-dev 0.3 and when I do sudo apt-add-repository ppa:pmjdebruijn/beignet-testing it does not update my beignet to 1.0. Could you please help me?

$ dpkg -s beignet-dev

Package: beignet-dev Status: install ok installed Priority: extra Section: libdevel Installed-Size: 41 Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com Architecture: amd64 Source: beignet Version: 0.3-1 Depends: beignet (= 0.3-1) Description: Intel OpenCL library OpenCL (Open Computing Language) is a multivendor open standard for general-purpose parallel programming of heterogeneous systems that include CPUs, GPUs and other processors. . This package contains the development files for directly linking against the Intel implementation. Original-Maintainer: Simon Richter sjr@debian.org Homepage: http://cgit.freedesktop.org/beignet/

[sketchc89]

run sudo apt-get update then sudo apt-get upgrade beignet-dev running dpkg -s beignet-dev should give the output I posted above

[staljbs]

@sketchc89 Thanks for the help. I have 1.0.3 version nw

[sketchc89]

I placed SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x -msse -msse2") on line 151 below IF(OPENCL_FOUND) and I'm now able to make and install with OpenCL enabled. Output of ./bin/Protonect cl that used to say OpenCL pipeline broken now says:

[Freenect2Impl] found 1 devices
[OpenCLDepthPacketProcessor::listDevice]  devices:
  0: Intel(R) HD Graphics Haswell Ultrabook GT2 Mobile (GPU)[Intel]
  1: Intel(R) HD Graphics Haswell Ultrabook GT2 Mobile (GPU)[Intel]
[OpenCLDepthPacketProcessor::init]  selected device: Intel(R) HD Graphics Haswell Ultrabook GT2 Mobile (GPU)[Intel]
[Freenect2DeviceImpl] opening...
[Freenect2DeviceImpl] opened
[Freenect2DeviceImpl] starting...
[Freenect2DeviceImpl] ReadData0x14 response

I still have the same issue with only RGB data but now I can repeat it in both OpenGL and OpenCL :smirk:

[xlz]

@sketchc89 perhaps it has something to do with your system being 32-bit.

[sketchc89]

@xlz I'll try the same with a 64-bit setup.

[sketchc89]

I installed libfreenect2 on the same computer with Ubuntu 14.04 Trusty. Kernel is latest stable. 4.0.5-040005-generic #201506061639 SMP Sat Jun 6 16:40:45 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux I was able to compile without issue except #36 which I can workaround. I still have issues running Protonect.

1. ./bin/Protonect cpu returns all of the sensor data RGB, Depth, IR :white_check_mark:
2. ./bin/Protonect gl only returns RGB as before
3. ./bin/Protonect cl now gives a scary looking error that looks like it's a major bug in Beignet

[DepthPacketStreamParser::onDataReceived] skipping depth packet
[RgbPacketStreamParser::onDataReceived] skipping rgb packet!
drm_intel_gem_bo_context_exec() failed: Invalid argument
[OpenCLDepthPacketProcessor::run] ERROR: clEnqueueNDRangeKernel (-5)
terminate called after throwing an instance of 'cl::Error'
  what():  clEnqueueNDRangeKernel
Aborted (core dumped)

@xlz do you know what is the earliest version of beignet that might work with this?

My version

dpkg -s beignet-dev
Package: beignet-dev
Status: install ok installed
Priority: extra
Section: libdevel
Installed-Size: 50
Maintainer: Debian OpenCL Maintainers <pkg-opencl-devel@lists.alioth.debian.org>
Architecture: amd64
Multi-Arch: foreign
Source: beignet
Version: 1.0.3-0pmjdebruijn1~trusty
Depends: beignet-opencl-icd (= 1.0.3-0pmjdebruijn1~trusty), ocl-icd-opencl-dev
Description: OpenCL library for Intel GPUs (development files)

[floe]

Actually, the cl error can (probably) be fixed by disabling the cmd_parser, see the FAQ on the frontpage how to do that.

[sketchc89]

That worked, thanks @floe . I now have both OpenCL and CPU working. Only outstanding issue left is OpenGL does not return any depth or IR data. To be more specific this is the command I ran to fix OpenCL. echo 0 | sudo dd of=/sys/module/i915/parameters/enable_cmd_parser

[floe]

IIRC the OpenGL packet processor has never properly worked on Intel GPUs. To actually fix this, you would probably need to add glGetError + debug output after every single OpenGL call and see which one breaks...

[floe]

Should be fixed by #429, please re-test with latest master and re-open if necessary.