OpenNI2-Freenect2Driver

[rafapages]

Hi! Is there already an implementations or are there any plans to implement an OpenNI2-Freenect2Driver?

Thanks!

[floe]

None that I know of. What is the state of OpenNI anyway, after Apple has bought Primesense?

[rafapages]

Occipital is currently maintaining it:

http://structure.io/openni https://github.com/occipital/OpenNI2

It would be nice to be able to use our code developed with OpenNI2 using K4W2.

Thanks!

[MasWag]

I'm writting it : https://github.com/MasWag/OpenNI2/tree/libfreenect2 and may be send pull request soon(but I think this cannot be merged without switching libfreenect2's support easily).

This requires my https://github.com/MasWag/libfreenect2/tree/sequence or merge https://github.com/OpenKinect/libfreenect2/pull/148 to make oni time stamp. In my environment, I can get depth/color frame , record depth (ir may also work but not tested yet). Color stream's recording is not available because I don't know how to gain time stamp of color frame in libfreenect2.

[ricardborras]

Hi MasWag,

very interesting your work to use OpenNi2 with libfreenect2. However, I have tried your fork of OpenNi2 and says no device connected with a Kinect One (libfreenect2 detects it). I will wait for your pull request!

Thanks

[MasWag]

Hi ricardborras,

Does SimpleRead (sample of OpenNI2 placed at OpenNI2/Bin/x64-Release/SimpleRead) says no device too? I don't hide libfreenect2's log message. So some log message should appear. Please show me the message.

Best,

[ricardborras]

Hi, find SimpleRead and SimpleViewer output:

rborras@rborras-Toshiba:~/OpenNI2/Bin/x86-Release$ ./SimpleRead Couldn't open device DeviceOpen using default: no devices found

rborras@rborras-Toshiba:~/OpenNI2/Bin/x86-Release$ ./SimpleViewer After initialization:

SimpleViewer: Device open failed: DeviceOpen using default: no devices found

lsusb pertinent line: Bus 002 Device 011: ID 045e:02d9 Microsoft Corp.

I don't know if you use OpenGL for OpenNI, but only CpuDepthPacketParser from Libfreenect2 works for me to get Depth / IR. RGB works ok with any configuration

Thanks

On Wed, Feb 25, 2015 at 12:19 PM, Masaki Waga notifications@github.com wrote:

Hi ricardborras,

Does SimpleRead (sample of OpenNI2 placed at OpenNI2/Bin/x64-Release/SimpleRead) says no device too? I don't hide libfreenect2's log message. So some log message should appear. Please show me the message.

Best,

— Reply to this email directly or view it on GitHub https://github.com/OpenKinect/libfreenect2/issues/168#issuecomment-75944205 .

Ricard Borràs Navarra

Linkedin: http://www.linkedin.com/in/ricardborras Twitter: https://twitter.com/RicardBorras

[MasWag]

Hi,

In my environment, root privilege is required to open Kinect. Please try $ sudo ./SimpleViewer

I'm using OpenCLDepthPacketProcessor and got message like this

$ sudo ./SimpleRead [sudo] password for MasWag: [Freenect2Impl] enumerating devices... [Freenect2Impl] 10 usb devices connected [Freenect2Impl] found valid Kinect v2 @4:7 with serial 511898342542 [Freenect2Impl] found 1 devices [OpenCLDepthPacketProcessor::listDevice] devices: 0: NVS 5200M (GPU)[NVIDIA Corporation] ...

If libfreenect2 says Kinect v2 does not find, this is libfreenect2's problem, and vice versa. If this kind of message does not appear, please confirm libLibfreenect2.so is loaded. You can check it with gdb.(after openni2's initialize, type 'info share')

Thanks,

[ricardborras]

Hi MasWag,

looking at loaded libraries, I have realized that I have not been using the correct version in your github account. Now, with checkout on last commit id and with your fork of libfreenect2, it gives me this errors on OpenNI2 make:

LF2DepthStream.cpp: In member function ‘virtual int LF2::LF2DepthStream::BuildFrame(libfreenect2::Frame, OniFrame)’: LF2DepthStream.cpp:29:37: error: ‘struct libfreenect2::Frame’ has no member named ‘sequence’ frame_out->frameIndex = frame_in->sequence; ^ LF2DepthStream.cpp:45:36: error: ‘struct libfreenect2::Frame’ has no member named ‘sequence’ frame_out->timestamp = frame_in->sequence*33369;

Libfreenect2 compiles correctly and kinect is found

Any clues?

Thanks

[MasWag]

Hi,

OpenNI2 require my libfreenect2's branch 'sequence' (https://github.com/MasWag/libfreenect2/tree/sequence) because oni recording uses timestamp.

Thanks,

[ricardborras]

Hi,

I have downloaded proper branch of libfreenect2 and now it compiles ok. However, running make on OpenNi2 give me this error:

g++ -o ../../../Bin/x86-Release/libOpenNI2.jni.so ./../../../Bin/Intermediate/x86-Release/libOpenNI2.jni.so/OpenNI.jni.o ./../../../Bin/Intermediate/x86-Release/ libOpenNI2.jni.so/org_openni_NativeMethods.o -L../../../Bin/x86-Release -lOpenNI2 -Wl,--no-undefined -shared /usr/lib/i386-linux-gnu/libc_nonshared.a(stack_chk_fail_local.oS): In function __stack_chk_fail_local': (.text+0x10): undefined reference to__stack_chk_fail' collect2: error: ld returned 1 exit status make[1]: *\ [../../../Bin/x86-Release/libOpenNI2.jni.so] Error 1 make[1]: Leaving directory `/home/rborras/OpenNI2/Wrappers/java/OpenNI.jni

If I ignore this error (I don't need Java wrappers right now), I run make samples and compile ok.

However, If I run samples I get OpenGL context errors:

sudo ./SimpleRead [Freenect2Impl] enumerating devices... [Freenect2Impl] 15 usb devices connected [Freenect2Impl] found valid Kinect v2 @3:3 with serial 029000344547 [Freenect2Impl] found 1 devices [OpenGLContext] failed to initialize glew for the current context! rborras@rborras-Toshiba:~/OpenNI2/Bin/x86-Release$ sudo ./SimpleViewer [Freenect2Impl] enumerating devices... [Freenect2Impl] 15 usb devices connected [Freenect2Impl] found valid Kinect v2 @3:3 with serial 029000344547 [Freenect2Impl] found 1 devices [OpenGLContext] failed to initialize glew for the current context!

Any clues?

Thanks

On Thu, Feb 26, 2015 at 12:19 AM, Masaki Waga notifications@github.com wrote:

Hi,

OpenNI2 require my libfreenect2's branch 'sequence' ( https://github.com/MasWag/libfreenect2/tree/sequence) because oni recording uses timestamp.

Thanks,

— Reply to this email directly or view it on GitHub https://github.com/OpenKinect/libfreenect2/issues/168#issuecomment-76083207 .

Ricard Borràs Navarra

Linkedin: http://www.linkedin.com/in/ricardborras Twitter: https://twitter.com/RicardBorras

[gaborpapp]

I would try running it with CpuPacketPipeline only first. Does that work with OpenNI?

[ricardborras]

How can I configure OpenNI2 in order to do this?

On Fri, Feb 27, 2015 at 10:13 AM, gaborpapp notifications@github.com wrote:

I would try running it with CpuPacketPipeline only first. Does that work with OpenNI?

— Reply to this email directly or view it on GitHub https://github.com/OpenKinect/libfreenect2/issues/168#issuecomment-76360255 .

Ricard Borràs Navarra

Linkedin: http://www.linkedin.com/in/ricardborras Twitter: https://twitter.com/RicardBorras

[gaborpapp]

It's a libfreenect2 thing, use openDefaultDevice(new libfreenect2::CpuPacketPipeline()).

[ricardborras]

Hi,

after doing this, libfreenect2 works in cpu mode but OpenNi2 complains with the same messages as before:

rborras@rborras-Toshiba:~/OpenNI2/Bin/x86-Release$ sudo ./SimpleRead [Freenect2Impl] enumerating devices... [Freenect2Impl] 15 usb devices connected [Freenect2Impl] found valid Kinect v2 @3:3 with serial 029000344547 [Freenect2Impl] found 1 devices [OpenGLContext] failed to initialize glew for the current context! rborras@rborras-Toshiba:~/OpenNI2/Bin/x86-Release$ sudo ./SimpleViewer [Freenect2Impl] enumerating devices... [Freenect2Impl] 15 usb devices connected [Freenect2Impl] found valid Kinect v2 @3:3 with serial 029000344547 [Freenect2Impl] found 1 devices [OpenGLContext] failed to initialize glew for the current context!

I put here also the libfreenect2 output:

sudo ./Protonect [Freenect2Impl] enumerating devices... [Freenect2Impl] 15 usb devices connected [Freenect2Impl] found valid Kinect v2 @3:3 with serial 029000344547 [Freenect2Impl] found 1 devices [Freenect2DeviceImpl] opening... [Freenect2DeviceImpl] opened [Freenect2DeviceImpl] starting... [Freenect2DeviceImpl] ReadData0x14 response 92 bytes of raw data 0x0000: 00 00 12 00 00 00 00 00 01 00 00 00 43 c1 1f 41 ............C..A 0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x0020: 0a 21 33 55 20 00 17 ba 00 08 00 00 10 00 00 00 .!3U ........... 0x0030: 00 01 00 00 00 10 00 00 00 00 80 00 01 00 00 00 ................ 0x0040: 31 33 00 00 00 00 06 07 47 4b 54 34 31 37 2e 31 13......GKT417.1 0x0050: 58 00 00 00 00 00 00 00 07 00 00 00 X...........

[Freenect2DeviceImpl] ReadStatus0x090000 response 4 bytes of raw data 0x0000: 01 26 00 00 .&..

[Freenect2DeviceImpl] ReadStatus0x090000 response 4 bytes of raw data 0x0000: 03 26 00 00 .&..

[Freenect2DeviceImpl] enabling usb transfer submission... [Freenect2DeviceImpl] submitting usb transfers... [Freenect2DeviceImpl] started device serial: 029000344547 device firmware: 4.3.3913.0.7 [DepthPacketStreamParser::handleNewData] not all subsequences received 0 init done opengl support available [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [TurboJpegRgbPacketProcessor] avg. time: 28.3791ms -> ~35.2372Hz [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet! [RgbPacketStreamParser::handleNewData] skipping rgb packet!

(ESCAPE PRESSED)

[Freenect2DeviceImpl] stopping... [Freenect2DeviceImpl] disabling usb transfer submission... [Freenect2DeviceImpl] canceling usb transfers... [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [TransferPool::submit] transfer submission disabled! [Freenect2DeviceImpl] stopped [Freenect2DeviceImpl] closing... [Freenect2DeviceImpl] releasing usb interfaces... [Freenect2DeviceImpl] deallocating usb transfer pools... [Freenect2DeviceImpl] closing usb device... [Freenect2DeviceImpl] closed [Freenect2DeviceImpl] closing... [Freenect2DeviceImpl] already closed, doing nothing

On Fri, Feb 27, 2015 at 10:20 AM, gaborpapp notifications@github.com wrote:

It's a libfreenect2 thing, use openDefaultDevice(new libfreenect2::CpuPacketPipeline()).

— Reply to this email directly or view it on GitHub https://github.com/OpenKinect/libfreenect2/issues/168#issuecomment-76361076 .

Ricard Borràs Navarra

Linkedin: http://www.linkedin.com/in/ricardborras Twitter: https://twitter.com/RicardBorras

[MasWag]

Hi,

My fork was a little old and https://github.com/OpenKinect/libfreenect2/commit/4c027e9cfc1b19b1f67f895dcd9e5a0ce5bd3d1a was not merged. So I switched pipeline by changing

typedef OpenGLPacketPipeline DefaultPacketPipeline;

to

typedef OpenCLPacketPipeline DefaultPacketPipeline;

in packet_pipeline.h

Certainly this solution was dirty. So I have merged master and fixed OpenNI2.

Now you can disable OpenGL and OpenCL by

cmake -DENABLE_OPENGL=OFF -DENABLE_OPENCL=OFF CMakeLists.txt

when compiling libfreenect2. This solution may seem still dirty. But OpenNI2 cannot call Driver with additional parameter. So this seems best solution.

Thanks,

[svope]

Do you think that it will be possible to use combination of freenect2 and OpenNI2 with middleware libraries like NiTE2 or it is just naive imagination? I'm asking because I tried your OpenNI2 version + NiTE2 library and I wasn't successful (coudn't create UserTracker, HandTracker etc...). Sadly, NiTE2 is not an open-source so.. SimpleRead log is:

[Freenect2Impl] enumerating devices... [Freenect2Impl] 9 usb devices connected [Freenect2Impl] found valid Kinect v2 @3:3 with serial 000393642047 [Freenect2Impl] found 1 devices [Freenect2DeviceImpl] opening... [Freenect2DeviceImpl] opened [Freenect2DeviceImpl] closing... [Freenect2DeviceImpl] releasing usb interfaces... [Freenect2DeviceImpl] deallocating usb transfer pools... [Freenect2DeviceImpl] closing usb device... [Freenect2DeviceImpl] closed [Freenect2DeviceImpl] opening... [Freenect2DeviceImpl] opened [Freenect2DeviceImpl] starting... [Freenect2DeviceImpl] ReadData0x14 response 92 bytes of raw data 0x0000: 00 00 12 00 00 00 00 00 01 00 00 00 43 c1 1f 41 ............C..A 0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x0020: 0a 21 33 55 c2 00 17 20 00 08 00 00 10 00 00 00 .!3U... ........ 0x0030: 00 01 00 00 00 10 00 00 00 00 80 00 01 00 00 00 ................ 0x0040: 31 33 00 00 00 08 13 0b 47 4b 53 36 34 35 2e 31 13......GKS645.1 0x0050: 58 00 00 00 00 00 00 00 07 00 00 00 X...........

[Freenect2DeviceImpl] ReadStatus0x090000 response 4 bytes of raw data 0x0000: 01 26 00 00 .&..

[Freenect2DeviceImpl] ReadStatus0x090000 response 4 bytes of raw data 0x0000: 03 26 00 00 .&..

[Freenect2DeviceImpl] enabling usb transfer submission... [Freenect2DeviceImpl] submitting usb transfers... [Freenect2DeviceImpl] started [DepthPacketStreamParser::handleNewData] not all subsequences received 0 [DepthPacketStreamParser::handleNewData] not all subsequences received 512 [00266952] 0 [00300321] 0 [00333690] 0 [00367059] 0 [00400428] 0 [00433797] 0 [00467166] 0 [00500535] 0

... plenty of those ...

[03437007] 0 [03470376] 0 [03503745] 0 [03537114] 0 [OpenGLDepthPacketProcessor] avg. time: 7.18956ms -> ~139.091Hz [03570483] 0 [03603852] 0 etc....

Is this log as it should be or these 0s are saying something is wrong? :] Thank you for your work!

[MasWag]

Hi, I have written this driver to make an application that use OpenNI2 and is running with Xtion Pro, work with Kinect v2. And I have not used Nite. So I do not know if Nite works with Kinect v2.My patch does not supprt some features like cropping.

In my env, SimpleRead's value changes when what Kinect seeing changes. SimpleRead's value seems the value of the center pixel.Does your value change when you change the angle or shade with your hand?

[svope]

Well, I tried waving in front of the sensor and moving with the sensor etc. and nothing has changed, still 0s

[MasWag]

Hmm, in my environment it works well but I don't use OpenGL. I could not use OpenGL even with only libfreenect2 so I have tested with OpenCL and CPU only. I have pushed libfreenect2_print_depth branch in https://github.com/MasWag/OpenNI2 . This show raw depth coming from libfreenect2 and output depth. If raw depthes are not 0 , something is wrong with OpenNI2.

In my envoronment, SimpleRead shows like this.

[Freenect2Impl] enumerating devices... [Freenect2Impl] 10 usb devices connected [Freenect2Impl] found valid Kinect v2 @4:3 with serial 511898342542 [Freenect2Impl] found 1 devices [OpenCLDepthPacketProcessor::listDevice] devices: 0: NVS 5200M (GPU)[NVIDIA Corporation] [OpenCLDepthPacketProcessor::init] selected device: NVS 5200M (GPU)[NVIDIA Corporation] ...

... [Freenect2DeviceImpl] started raw openni: 6693.669922 6693 BuildFrame [27696270] 6693 raw openni: 6713.505371 6713 BuildFrame [27729639] 6713 raw openni: 6659.960449 6659 BuildFrame [27763008] 6659 raw openni: 6707.549805 6707 BuildFrame [27796377] 6707 raw openni: 6684.961426 6684 BuildFrame [27829746] 6684 raw openni: 6672.634277 6672 BuildFrame [27863115] 6672 raw openni: 6653.196777 6653 BuildFrame [27896484] 6653 raw openni: 6666.900879 6666 BuildFrame [27929853] 6666 raw openni: 6707.940430 6707 BuildFrame [27963222] 6707

[xlz]

Please continue discussion in #243 which provides implementation. This can be closed.