Skeltrack and Kinect V2

[davudadiguezel]

Hi everyone, I am interested in doing skeleton tracking with skeltrack. Is it possible to do it with a Kinect One (V2)? And has somebody advice (or some link) for me on how to do it? Greetings Davud

[criverc]

Hello, I just finished porting the skeltrack demo so it works with kinect V2. You will need the libfreenect2. I mirrored the whole skeltrack repository here: http://farside.envitel.com:8080/git/Skeltrack.git so all you need to do is:

git clone http://farside.envitel.com:8080/git/Skeltrack.git

Regards!

[joaquimrocha]

Hey @criverc ,

Pretty cool! I do not own a Kinect v2 but I am very curious to know about your impressions when porting it. It'd be great if you could also share a video of Skeltrack working with it.

I noticed you created a new demo. I haven't looked into the code but it'd be great if we could have just one demo that could handle both Kinects and call one or the other based on what it detects or on a given user parameter.

[criverc]

Yes, I somehow merged your demo with the capture part of a libfreenect2 demo. I made it a separate file to make things easier. I did have to change the dimension-reduction because the cloud of points had really very few points compared with the kinect 1 example. I will try to share a video, I do have lots of instability, meaning the joints jump around a lot, but I think I had the same problem using kinect v1, perhaps my working space is too cluttered with objects and the algorithm gets confused?

[criverc]

@joaquimrocha I tried to mix both demos. But the main difficulty I have is that of course one uses gfreenect and the other freenect2. The second is C++ and uses classes so I suppose the proper way to do it would be to extend gfreenect so it exposes the same glib interface for freenect2 as it does for freenect. That way the demo code would be pretty clean, just use one object or the other. I think for the moment I am leaving them as two separate files.

[davudadiguezel]

Hey criverc, thanks for the quick respond! I just installed Skeltrack and now, I am trying to run the example. Do I still need Gfreenect? And how can I include the libfreenect2 driver? Greetings

[criverc]

Well @davudadiguezel, you don't need gfreenect to run the kinect2 demo because it uses libfreenect2. However you will probably need it to compile the kinect 1 demo, as it is the makefile will probably stop if it cannot compile the kinect 1 demo. So it is best to have both installed. I installed libfreenect2 by cloning from: https://github.com/OpenKinect/libfreenect2.git Then did typical ./configure; make; make install sequence of commands. By the way I made a change so it is possible to see the cloud of points and the skeleton at the same time and there is some error in the kinect2 demo, the skeleton is out of sync, spatially, to the cloud. I am working on it. Sorry for the late reply this time :-( Good luck!

[criverc]

hi @joaquimrocha here it is a video: http://farside.envitel.com:8080/skeltrack/kinect2_demo.avi , as you can see I still can not get it to work very well, but I found out that I need to adjust some of the parameters. I have been trying to calibrate the measurements and it looks like kinect 2 gives much better values for all x,y,z coordinates. Perhaps the hardcoded numbers in skeltrack are set to compensate some of the error of kinect 1? By the way I have made a commit (in my repo of course) where I expose the intrinsics of the camera to the outside (this was a TODO in the code). So now you can manipulate minimum_depth and scale_factor using the gobject interface. Cheers!

[joaquimrocha]

Hi @criverc , Thanks for the video! I see that you don't have a lot of room for moving due to the two tables around you, that's why Skeltrack sometimes picks the tables as parts of the hands. This can be reduced if we establish limits on the length of the joints, but we'd have to experiment with several values. You should also point the kinect a bit up, otherwise, as you can notice, raising your arms will go out of the field of view. You can also make the detection smoother by changing the smooth factor, the movements will appear a bit slower but more natural and smooth.

I do not understand what you mean by exposing the "intrinsics" of the camera.

BTW, there is a small project we created to help debug Skeltrack which is called gfreenect-utils. There is a branch that will allow you to record a few frames from the Kinect as binary files: https://github.com/iaguis/gfreenect-utils/tree/video-recorder Then you can use the "Skeltrack video player" to load those files and show how Skeltrack finds their joints: https://github.com/iaguis/Skeltrack-video-player This is more comfortable to use than having to get up and try the changes live.

If these tools are useful to you maybe it's a good idea to adapt GFreenect to work with libfreenect2.

Cheers,

[criverc]

@joaquimrocha maybe intrinsics is the wrong term. In the formulas to calculate x,y,z from i,j,z and viceversa there are two constants: minimum_distance and scale_factor. These come from doing some perspective drawings and trigonometry and they depend on the camera optics. They were hardcoded and I have exposed them to the user so they can be changed as other properties of the skeltrack object. Thanks for your tips!! I will take a look at that debugging tool it seems very useful. I agree it would be nice to adapt gfreenect to work with libfreenect2 but I don't know what the best approach would be considering libfreenect2 is c++. What do people do in general when wrapping glib over C++ objects?

[criverc]

Dear @joaquimrocha here it is another video. It is working much better now. The tools you mentioned work for kinect1 so instead of using them I added recording and replaying capabilities to the demo tool itself. I have also added calculation of the angle of the right elbow because I need something to cause a trigger in my project. I plot a graph in the demo tool itself, it is getting quite complicated now!!

http://farside.envitel.com:8080/skeltrack/skeltrack2.avi

Regards

[mcgi5sr2]

Hi, I realise this is bumping what is probably a dead thread, but I am interested in trying to get this to work on my project. @cricerc is there any chance you could update the readme.md with install and use instructions?

[airhorns]

@criverc I'm interested as well, now that NiTE is gone any tips you can give us on how to get Skeltrack and libfreenect2 playing nice would be fantastic!

[mcgi5sr2]

@airhorns I got the demo up and running using this guide http://tayyabnaseer.blogspot.co.uk/2012/05/installing-skeltrack-on-ubuntu.html the branch you want is the criverv_WIP one

[criverc]

@mcgi5sr2 Sorry for the late reply. Where you able to make it work?

[criverc]

@airhorns the code that works with libfreenect2 is here, in a mirror I made of the main project:

git clone http://farside.envitel.com:8080/git/Skeltrack.git

@mcgi5sr2 and @airhorns Please read this whole thread I give some instructions above on how to make it work. I would gladly accept that readme file :-)

[mcgi5sr2]

Hi @criverc I did mange to get it working in the end, but the response time was slower than required, and ideally I was after fully body tracking. Initially I thought I could add in the other limbs, but I managed to get a working version using NiTE2 up and running in the mean time and so am using that. Thanks again for the work.

[bxs1260]

@mcgi5sr2 congratulations for get it working. It would be great if you could help me in achieving a full body tracking using kinect v2 and NiTE2.

[davudadiguezel]

@mcgi5sr2 Are you running on an intel processor? I just tried to run nite2 on an amd processor and got illegal instruction error. Can anyone provide a nite2 version compiled for amd processor 64 bit (linux)?

[bxs1260]

Probably the NiTE-Linux-x64-2.2 should work as you have mentioned 64 bit (linux) nite2 archive: https://bitbucket.org/kaorun55/openni-2.2/src/2f54272802bfd24ca32f03327fbabaf85ac4a5c4/NITE%202.2%20%CE%B1/?at=master

[davudadiguezel]

@bxs1260 thanks, but this version is compiled for intel processors. actually this is the version I am running on my intel pc, but I need a version thats compiled for amd because there are differences in the instruction set. thats why I get illegal instruction.

[criverc]

@bxs1260 That's the problem with NITE, it is a binary blob, no source code, decompiling and reverse engineering is explicitly forbidden in the license terms. That's why it would be nice to have people contribute to Skeltrack which is free software :-)

[bxs1260]

@criverc Does skeltrack supports full body tracking?

[criverc]

@bxs1260 No, but there are some attempts to include legs, see this thread: https://github.com/joaquimrocha/Skeltrack/issues/17 I could get legs working, but one problem was that the algorithm couldn't distinguish between feet and floor. More work is needed there.

[orgicus]

@criverc Is the libfreenect2 enabled Skeltrack project still available ? If so where can I find it ?

(can't seem to access http://farside.envitel.com:8080/git/Skeltrack.git)

Thank you

[criverc]

uh, sorry, I don't work there anymore, but I'll try to recover the code and put it here at github.

[orgicus]

That would be awesome! Thank you very much 👍

[criverc]

Here it is: https://github.com/criverc/K2Skeltrack You should look at the criverc_WIP branch. Good luck!

[orgicus]

Awesome! Thanks Carlos, I'll have a play.

[toruurakawa]

Thanks @criverc !! Your code works perfectly :)

[gunnicus]

Hello Sir @criverc , are you still working on this project? I wonder if this will work with Intel Realsense camera. I can get depth data from librealsense (https://github.com/IntelRealSense/librealsense) but there is no skeleton tracking in librealsense, I wonder if librealsense could be integrated with your skeleton tracking work. Also I don't really familiar with c/c++, I only know some c#, if it is possible a c# wrapper could be made for your work. Thanks a lot.

[joaquimrocha]

@gunnicus I am no longer working on this unfortunately. As long as you can get a depth buffer from librealsense, Skeltrack should be able to work with it.

[criverc]

@gunnicus sorry I am no longer working on skeltrack either. Good luck!