Raspberry Support?

[ManuelCostanzo]

Hello, i have a question... I need to know if it is possible to use Kinect ONE on Raspberry Pi. I know that kinect runs on USB 3.0 but, i think that modifying some parameters, like for example "ir_pkts_per_xfer", it can be work. Before buy one i want to know if it is possible.

Thank you!

[floe]

The Kinect v2 is a pure USB3 device, so the Raspberry Pi will not be able to establish a connection on the hardware level. Even if you managed to get it to communicate, you would still need USB3 to transfer the nearly 4 GBit/s of raw data sent by the depth camera, and you still need to process that data, too. You're better off with one of the more powerful ODroid boards that already have USB3.

[createandbuild]

The Kinect v2 is a pure USB3 device, so the Raspberry Pi will not be able to establish a connection on the hardware level. Even if you managed to get it to communicate, you would still need USB3 to transfer the nearly 4 GBit/s of raw data sent by the depth camera, and you still need to process that data, too. You're better off with one of the more powerful ODroid boards that already have USB3.

Hello, would the newer Raspberry Pi 4, which has USB3 ports, work with the Kinect? Or are there any other problems hindering connection from the Kinect (for Xbox One) to the Raspberry Pi?

Thank you!

[floe]

RasPi 4 might actually work; it also has OpenGL ES 3, so the GPU should be beefy enough to do the depth processing (like the Odroid). Worth a try!

[ManuelCostanzo]

Thanks for reply. And whats happen if i use an USB 3 HUB, with external power supply, and that hub connected on RPI3 USB2?

[floe]

Wouldn't make a difference. The RPi3 will detect the hub, but not the Kinect v2 device.

[CarlosGS]

Hi! Thank you for maintaining such a useful library! :-)

I've attempted to get it running on a Raspberry Pi 4, but the Pi is resisting. Everything compiles without an issue, but it freezes when running the Protonect example (right after initializing the stream).

It does run with only the RGB stream enabled. However it doesn't run with only the Depth and/or IR streams enabled.

To debug, I added to this line https://github.com/OpenKinect/libfreenect2/blob/master/src/depth_packet_stream_parser.cpp#L72 a logging statement that simply prints inbound packet size. On a regular computer it yielded something like this (sorry I didn't save the exact output, this is what I recall from the tests)

...
0
0
0
0
35204
35204
35204
35204
27337
0
0
0
0
35204
35204
35204
35204
27337
0
0
...

While on the pi:

...
0
27337
0
0
0
27337
0
0
35204
27337
0
0
0
...

So my guess is that unfortunately we reach a hardware limit and the Pi isn't able to grab enough packets to assemble one frame. I also tried overclocking the Pi and it didn't change anything.

I'm still hopeful that someone clever will read this and figure out something to get it working O:-)

[danruochen]

Hi you all, merry Christmas. Any progress on Raspberry pi 4?

[yomboprime]

It does run with only the RGB stream enabled. However it doesn't run with only the Depth and/or IR streams enabled.

Does it work with all 3 streams enabled, but without doing any image processing?

My idea is that I would rather use the Pi4 (plus Kinect2) as a portable scanner that stores the streams on files, and later the streams are processed in a powerful computer at home.

[CarlosGS]

That was the plan, but no, it did not even support fetching the depth stream on its own :(

[tavishm]

libfreenect2 successfully worked for me on a raspberry pi 4. It wasn't too tough, took me a day to do it.

[tavishm]

I have taken 3d images and they worked well, done this some 5 months ago I don't remember how I did it exactly but I am sure it worked.

[tavishm]

I tried doing it again on a pi-4. The install went well. I saw depth images, and now it's mostly all documented except that the images don't come from Protonect

I wasn't able to run Protonect this time, I am not sure if Protonect worked right last time, It mostly did(I forgot). Protonect showed some error, and I'm 70% sure it's due to the weak processer. but I was able to take depth pictures from a python library that is itself an interface to libfreenect2.

If anyone is interested I can give the steps to take depth images from python on raspi4.

Also, if it is a problem of the weak processer it can be tested from booting windows on a raspi 4 and use the official Microsoft software and check. I guess it is not a problem of a weak processor cause if it was, how is python taking images? by the way, I got skipping depth packet error in Protonect

[tavishm]

here's the python library, freenect2, you can follow the tutorial but there are some issues, I'll tell you somethings I've figured out the hard way -

1. enable opengl withraspi-config(you might know that)
2. specify PKG_CONFIG_PATH in your.bashrcor something(it's in freenects docs)
3. here's something I don't remember how I know, but freenect didn't work for me without this - a. extract freenect2.zip and put is somewhere on your pi b. export LD_LIBRARY_PATH to freenect2/lib (freenect2 being the extracted folder)

Hope this works for you!

[yomboprime]

Right, now I have an excuse to buy a Pi4 😂

[tavishm]

I forgot to mention, please install python cffi from apt. it's one of the things that I wasted time on

@CarlosGS did this work for you?

[CarlosGS]

OK I'm testing this now, the first thing I've done is to install the python library and CFFI. Unfortunately the two freenect2-python examples are giving the same problem I had with protonect. Basically it can't receive a full frame and never reaches the "break" condition in either example.

@tavishm now I'm setting up an installation from scratch with the latest versions of everything. Did you use Raspbian Buster Lite or the desktop version?

[CarlosGS]

It works!!! I think the cause may have been that I previously did not enable OpenGL in raspi-config :facepalm: Thank you @tavishm so very much!! :heart_eyes:

[ManuelCostanzo]

Hi @CarlosGS, can you obtain depth image with RGB ? Or you can only get depth image ?

[CarlosGS]

I think it can get everything. Though still haven't tested nor measured FPS.

[ManuelCostanzo]

You used a RPI 4, didn't you ?

[CarlosGS]

Of course.

[CarlosGS]

Out of curiosity I measured FPS and doesn't look good. It is usable for pictures but not really for video. @tavishm how fast could you get it to capture frames?

[tavishm]

Videos didn't work for me either, yet. I guess Protonec didn't work because the pictures have a live stream but because of some reason, raspi4 isn't fast enough. I think it's either a computational problem or something slowing down the stuff coming from Kinect. It may not be a computational problem because 4 GB of RAM, Broadcom BCM2711 SoC with a 1.5 GHz 64-bit quad-core ARM Cortex-A72 processor, with 1 MiB shared L2 cache and Broadcom VideoCore VI is quite good, I guess but @CarlosGS you may know better. @ManuelCostanzo, I got color images.

[CarlosGS]

Right now the Python library is opening up the three streams, maybe if we can cut one of them (i.e. IR) we could have some more margin and maybe capture faster.

Another question is bugging me though. When connecting through SSH I needed to use the -X command to forward the DISPLAY, otherwise the script doesn't run. I am not sure if this is causing overhead. I have also tested setting everything up in Raspbian Desktop and connecting with VNC, but results were similar. Which setup are you using?

At least some frames do come through, hopefully we can find the bottleneck! :)

[CarlosGS]

I may have misunderstood the libfreenect2 functionality regarding listeners for Depth/RGB/IR: https://github.com/OpenKinect/libfreenect2/blob/master/src/frame_listener_impl.cpp#L165 Maybe it is not possible to have the kinect selectively enable some of the streams. Instead it seems it always sends everything and the CPU is responsible to discard what it doesn't need. Is it true? This would make things harder.

I think we will need help from the experts to figure out the exact bottleneck & possible workarounds.

[floe]

To clarify: depth and IR stream are sent as 10 separate subframes which have to be de-interleaved in software (this is why it only works with OpenCL or OpenGL to do the heavy lifting). So you can't have one without the other. RGB, on the other hand, is a separate MJPEG stream which you could probably turn off, but this isn't saving a lot of data transfer or processing.

[CarlosGS]

Thank you! Can you suggest any "light" (windowless) Cpp example that we can use to debug this? I'm also worried Python may be incorporating some overhead.

Would it be good to test the raw reading & discarding of the subframes, to check if USB is the bottleneck?

[floe]

You can't do console-only, because OpenGL currently needs X11 to access the graphics hardware. You could try some ultra-minimal oldschool window manager such as wmx. And the Protonect example in the repo here is pure C++. You can certainly try to just not forward the frames to the deconvolution code and see how that changes FPS.

[yomboprime]

My Pi4 arrives tomorrow and I'll be trying this.

[Jyurineko]

@tavishm hi,can you teach me how to use freenect2-python? i followed document, should i run as libfreenect document? thanks in advance:)

[tavishm]

I wouldn't be the best person to answer this as I haven't developed the module. @rjw57 has developed it, you should ask him/her. This was the best documentation I could find, It's pretty good, I mostly used the example code to try to understand what was going on and how do I make stuff work. You can check out the function references in the link for more info(they are pretty well described).

[Jyurineko]

OK, thx for the explanation:) i try it again.

[Jyurineko]

@CarlosGS may i ask which pipeline you use? or you write a new pipeline youself? because i run the freenect2-python/examples the ir_camera.py, it shows

---

[Error] [OpenGLDepthPacketProcessorImpl] GLFW error 65543 GLX: Failed to create context: GLXBadFBConfig [Error] [OpenGLDepthPacketProcessor] Failed to create opengl window.

---

PS: i enabled OpenGL in raspi-config

BTW i use this OS version, maybe i should reinstall the latest version of raspiOS: PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debian

Thx in advance!:)

[CarlosGS]

It was working in two different ways:

• Connecting with SSH and the -X flag
• If using local GUI, sometimes it was needed to prepend the command with DISPLAY=0, i.e. DISPLAY=0 python3 script.py Good luck!!

[Jyurineko]

I tried SSH with -X flag and VNC Viewer connection, both same error maybe i'm doing something wrong, new day new problem :( and i realize the not root account problem, when i run as root still same problem

---

pi@raspberrypi:~/Desktop/CGPraktikum/freenect2-python/examples $ sudo DISPLAY=0 python3 ir_camera.py [Info] [Freenect2Impl] enumerating devices... [Info] [Freenect2Impl] 6 usb devices connected [Info] [Freenect2Impl] found valid Kinect v2 @2:3 with serial 014443550647 [Info] [Freenect2Impl] found 1 devices [Error] [OpenGLDepthPacketProcessorImpl] GLFW error 65544 X11: Failed to open display 0 [Error] [OpenGLDepthPacketProcessor] Failed to initialize GLFW. pi@raspberrypi:~/Desktop/CGPraktikum/freenect2-python/examples $ echo $DISPLAY :0.0 pi@raspberrypi:~/Desktop/CGPraktikum/freenect2-python/examples $ sudo DISPLAY=0.0 python3 ir_camera.py [Info] [Freenect2Impl] enumerating devices... [Info] [Freenect2Impl] 6 usb devices connected [Info] [Freenect2Impl] found valid Kinect v2 @2:3 with serial 014443550647 [Info] [Freenect2Impl] found 1 devices [Error] [OpenGLDepthPacketProcessorImpl] GLFW error 65544 X11: Failed to open display 0.0 [Error] [OpenGLDepthPacketProcessor] Failed to initialize GLFW.

---

[Jyurineko]

i tried but failed. :( are you using original OpenGL pipeline or you compile new pipeline? if you compile your own can you document something? thx in advance

[tavishm]

Try using a monitor with the raspi... This stuff is very buggy... random stuff works out

[Jyurineko]

i dirctly connect a monitor with raspi. and reenable a lot of times the OpenGL Driver(fake KMS) in advance option. and i reinstall all the staff, libfreenect2, supporting library and freenect2-python , follow the document, my pi4 seems like fake:) i'm dying..............................:(

[yomboprime]

The first error (GLFW error 65543) is the same as in this issue: #649 It may indicate a driver problem.

I can only suggest to check the opengl version (must be >= 3.1) glxinfo | grep "OpenGL version"

[Jyurineko]

OHH..it's 2.1 version, root@raspberrypi:~/Desktop/libfreenect2/depends# glxinfo | grep "OpenGL version" OpenGL version string: 2.1 Mesa 19.3.2

root@raspberrypi:~/Desktop/libfreenect2/depends# glxinfo | grep OpenGL OpenGL vendor string: Broadcom OpenGL renderer string: V3D 4.2 OpenGL version string: 2.1 Mesa 19.3.2 OpenGL shading language version string: 1.20 OpenGL extensions: OpenGL ES profile version string: OpenGL ES 3.1 Mesa 19.3.2 OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10 OpenGL ES profile extensions:

root@raspberrypi:~/Desktop/libfreenect2/depends# ./glfw_src/build/tests/glfwinfo -m 3 -n 1 GLFW header version: 3.0.4 GLFW library version: 3.0.4 GLFW library version string: "3.0.4 X11 GLX glXGetProcAddress clock_gettime /dev/js shared" Error: GLX: Failed to create context: GLXBadFBConfig

---

i follow the issue: #649 and edit install_glfw.sh and change "git checkout 3.0.4" to "git checkout master",

root@raspberrypi:~/Desktop/libfreenect2/depends# glfw_src/build/tests/glfwinfo -m 3 -n 1 GLFW header version: 3.4.0 GLFW library version: 3.4.0 GLFW library version string: "3.4.0 X11 GLX EGL OSMesa clock_gettime evdev shared" Error: GLX: Failed to create context: GLXBadFBConfig

root@raspberrypi:~/Desktop/libfreenect2/depends# glxinfo | grep "OpenGL version" OpenGL version string: 2.1 Mesa 19.3.2

---

yes as Troubleshooting says, opengl version is too low

[Jyurineko]

after update Mesa to 20.1.0-rc4, OpenGL version string is still stop at 2.1 but pi4 support OpenGL ES 3.0+, new direction for me

[tavishm]

Have you tried reinstalling your pi? You might have done something you didn't want to.

[yomboprime]

Have you tried reinstalling your pi? You might have done something you didn't want to.

I've got Raspbian newly installed in my pi4 and I have also GL 2.1.

I didn't have time to test the kinect 2 with it though.

[Jyurineko]

Edit...Using ES3.1 also works good(with 5 FPS though), Raspberry pi 4 is supported :)

[tombarratt46]

Edit...Using ES3.1 also works good(with 5 FPS though), Raspberry pi 4 is supported :)

Maybe a noob question, how did you setup ES3.1 to get freenect to work with it?

[Jyurineko]

@tombarratt46 the fragment shader change into ES3.1 format, such as add ‘precision specifer’ for float. In 'Opengl_depth_packet_processor.cpp/viewer.cpp/viewer.h' some OpenGL related codes should be changed into ES3.1 format. and flextGL.cpp & flextGL.h you can using https://github.com/mosra/flextgl to generate the ES version of flextGL, or you rewrite as original work by yourself, because some functions don't work on ES3.1. BTW:https://github.com/Jyurineko/libfreenect2.git hope it can help you :)

[Mikael17125]

@tombarratt46 the fragment shader change into ES3.1 format, such as add ‘precision specifer’ for float. In 'Opengl_depth_packet_processor.cpp/viewer.cpp/viewer.h' some OpenGL related codes should be changed into ES3.1 format. and flextGL.cpp & flextGL.h you can using https://github.com/mosra/flextgl to generate the ES version of flextGL, or you rewrite as original work by yourself, because some functions don't work on ES3.1. BTW:https://github.com/Jyurineko/libfreenect2.git hope it can help you :)

is this work for python library?

[busybeaver42]

Hello together, great job so far. I have a kinectV2 and RPI4 and I would like buildup a 3D scanner. Therefore I need background infos to setup the system. Is 64GB SD card ok ? Which OS is recommended(Ubuntu server 64bit ? Buster 32 Bit ?) And how can I install ES3.1 for openGL and enable openGL on RPI4( I don't see this option via raspi-config).

[Jyurineko]

@tombarratt46 the fragment shader change into ES3.1 format, such as add ‘precision specifer’ for float. In 'Opengl_depth_packet_processor.cpp/viewer.cpp/viewer.h' some OpenGL related codes should be changed into ES3.1 format. and flextGL.cpp & flextGL.h you can using https://github.com/mosra/flextgl to generate the ES version of flextGL, or you rewrite as original work by yourself, because some functions don't work on ES3.1. BTW:https://github.com/Jyurineko/libfreenect2.git hope it can help you :)

is this work for python library?

I didn't change the .py related file, so maybe it works

[busybeaver42]

I have try out to install Libfreenect2 on Ubuntu 64Bit. I have used the git repro from Jyurinieko https://github.com/Jyurineko/libfreenect2.git. At the end I have an issue USB bulk transfer. The kinectv2 is running on my PC via ubuntu 20.04 with libfreenect2. I have increase the usb memory and autosuspend was deactivated. At the end I found a hint, that this error can be happen because of en error inside the kernel. I guess, I need a little bit help to understand, how to investigate this. @Jyurineko, could you send me several information about your system, please. Perhaps, I use a wrong OS(e.g. older OS). I a second try I have used buster 32 bit. I get the same issue.