D435i hand tracking Python

[RarDay]

Hi everyone and @MartyG-RealSense )) I'm not sure what I'm writing in the right section, but how can I use hand tracking? I saw your project for C++. Maybe there is one for python? The essence of the task is to control the tablet with your hands.

Required Info
Camera Model	D435i
Operating System & Version	Windows 10
Language	Python 3.7

And another question about the camera, in the description of the D435i it is written that it has a built-in neural network for recognizing objects, including hands. What does it mean? Or it applies exclusively to the SDK. Did I understand correctly that if, for example, I work with a camera using python, then I need to search for datasets myself and write code for recognition? Or does the realsense library already have everything?

[MartyG-RealSense]

Hi @RarDay There was a C++ hand tracking project that was published by Intel in 2017 shortly before the 400 Series was released. It is based around Legacy Librealsense (librealsense1) though rather than the current librealsense2 and was not updated for the 400 Series and librealsense2.

https://github.com/IntelRealSense/hand_tracking_samples

There is not a built-in hand tracking solution in librealsense2. Instead, third-party vision computing platforms such as OpenCV and the OpenVINO Toolkit can provide these functions and offer the possibility of intergrating them in librealsense applications via the SDK's compatibility wrappers with these platforms.

An example of a non-librealsense hand tracking project using Python and OpenCV is in the link below:

https://dev.to/amarlearning/finger-detection-and-tracking-using-opencv-and-python-586m

Completely unrestricted datasets tend to be difficult to find, as there is usually some kind of administrative barrier such as the need to register to get access to them. The C++ hand example above has a small dataset included.

https://github.com/IntelRealSense/hand_tracking_samples/tree/master/datasets

Googling for free hand datasets should provide leads to further free hand dataset resources. The link below has a dataset resulting from such a search.

http://www.cs.technion.ac.il/~twerd/HandNet/

RealSense cameras can also be used to train your own joint dataset for tracking purposes, as described in an advanced-level seminar by Intel that is available on YouTube:

https://www.youtube.com/watch?v=VSHDyUXSNqY

In regard to the question about neural networks: D435i cameras, or other 400 Series models, do not have a neural network built into them and do not have built-in hand / face detection and tracking (the SR300 and SR305 do have hand joint and face landmark support though).

Additional hardware support for vision computing can be provided by the Intel Neural Compute Stick 2 (NCS2) USB stick, which is available in a bundle with 400 Series cameras or purchasable on its own.

https://www.intelrealsense.com/depth-camera-and-ncs2/

There is an example program that makes use of the NCS2:

https://github.com/PINTO0309/MobileNet-SSD-RealSense

[MartyG-RealSense]

Hi @RarDay Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case clsoed due to no further comments received.

[miguelml99]

Hi @MartyG-RealSense , first of all thanks for the links and info you provided, they are very useful. Secondly, I wanted to ask if you can use the datasets with any Intel RS camera, cause I was trying to use the source code project from this https://github.com/IntelRealSense/hand_tracking_samples page, with RS D435i but apparently they use the SR300. Everytime I try to run the project an error appears saying that no device was found, so I figured it was not compatible with my model. Thanks !

[MartyG-RealSense]

Hi @miguelml99 That samples page was released before the launch of the 400 Series cameras in late 2017. The text of the page indicates that an update for 400 Series cameras was intended but it never happened.

[miguelml99]

You are right, I didnt realize, thank you for your response. However, in case you know of any particular page with further information, tutorials or datasets on hand tracking for the 400 series please let me know. Im doing my bachelors thesis with D435i on this topic and any help is useful. Thanks again!

[MartyG-RealSense]

@miguelml99 An article about hand tracking on Intel's official RealSense blog provides links to a few research papers on the subject.

https://www.intelrealsense.com/hand-tracking-overview/

[miguelml99]

Great, I already saw that one before, the problem is that all the papers in there still use SR300 camera, so I dont believe I can use their datasets, thank you anyways!

[MartyG-RealSense]

@miguelml99 If you do not require hand gesture recognition, there is a commercial solution called the Cubemos Skeleton Tracking SDK that is compatible with the 400 Series cameras and performs approximate hand position tracking by following a point on the wrist area just above the hand (not the finger joints). It is available from the official online RealSense store and has a free trial version (see the information website at the link below).

Information https://www.intelrealsense.com/skeleton-tracking/

Store https://store.intelrealsense.com/buy-skeletal-tracking-sdk.html

Cubemos hand position detection demonstration video https://www.youtube.com/watch?v=3Zn_4I5IuwM

[666tua]

@ miguelml99 如果您不需要手势识别，则有一个称为Cubemos Skeleton Tracking SDK的商业解决方案，该解决方案与400系列摄像机兼容，并通过跟踪手上方手腕区域上的一个点来执行大致的手部位置跟踪（不是手指关节）。它可以从RealSense官方在线商店购买，并具有免费试用版（请参阅下面链接中的信息网站）。

信息 https://www.intelrealsense.com/skeleton-tracking/

商店 https://store.intelrealsense.com/buy-skeletal-tracking-sdk.html

Cubemos手部位置检测演示视频 https://www.youtube.com/watch?v=3Zn_4I5IuwM

I am using D415 for hand tracking and need to identify 22 joint points of the hand. I would like to ask, can the Cubemos Skeleton Tracking SDK be completed? Is there any code or information? Thank you very much!

[MartyG-RealSense]

Hi @L-xn The Cubemos Skeleton Tracking SDK does not have 22-point finger tracking. The closest point to the hands is a single point on the wrist just above the left and right hand, highlighted in black boxes on the image above.

[666tua]

@ miguelml99 如果您不需要手势识别，则有一个称为Cubemos Skeleton Tracking SDK的商业解决方案，该解决方案与400系列摄像机兼容，并通过跟踪手上方手腕区域上的一个点来执行大致的手部位置跟踪（不是手指关节）。它可以从RealSense官方在线商店购买，并具有免费试用版（请参阅下面链接中的信息网站）。

信息 https://www.intelrealsense.com/skeleton-tracking/

商店 https://store.intelrealsense.com/buy-skeletal-tracking-sdk.html

Cubemos手部位置检测演示视频 https://www.youtube.com/watch?v=3Zn_4I5IuwM

Then I think Cubemos Skeleton Tracking SDK is not a feasible solution.So what should I do if I want to use D415 for hand tracking. Hope to get advice from seniors! Thanks!

[miguelml99]

@MartyG-RealSense Thank you for the recommendation!! I will look over the Cubemos SDK solution and see if it fits my project needs for the D435i. I dont have many hopes on it though, since I need to focus more on Hand gesture recognition, and a single point for each hand doesnt seem to be enough, which I believe is simililar to the issue that @L-xn has.

[MartyG-RealSense]

@miguelml99 If your reason for hand joint tracking is for the purpose of gesture recognition then the commercial product Nuitrack SDK, which supports RealSense cameras (D415 / D435 / D435i / D455 / L515) has support for some hand gestures (waving, swipe left, swipe right, swipe up, swipe down and push).

Nuitrack SDK https://nuitrack.com/ https://nuitrack.com/blog#d455_l515

Nuitrack Gesture Recognizer https://download.3divi.com/Nuitrack/doc/group__GestureRecognizer__group.html

[miguelml99]

@MartyG-RealSense Thank you ! I sent both SDKs options to my thesis professor, and i will wait to see what he says. The inconvenience these two have for me, is that you have to pay a subscription, but hopefully my professor provides me one. I also have to combine whatever my D435i sensor will detect (hand position prob), later on in Unity, so that I can interact with the sensor in this platform. I dont know if you have any experience with this, but any assitance is welcome.

[MartyG-RealSense]

The Unity Asset Store also has an asset called Kinect V2 Examples for Windows that is stated to be compatible with 400 Series cameras despite its name and has a large range of sample scenes that include hand-driven applications.

https://assetstore.unity.com/packages/3d/characters/kinect-v2-examples-with-ms-sdk-18708

Details of the examples of this asset - a one-time purchase instead of a subscription - can be found in the link below. I recommend performing a Find search on the browser page for the word hand to highlight that examples that make use of hand control.

https://ratemt.com/k2docs/DemoScenes.html

Alternatively, there is a free OpenCV integration for Unity in the Asset Store that could potentially provide in-Unity access to OpenCV-based hand / finger tracking solutions.

https://assetstore.unity.com/packages/tools/integration/opencv-plus-unity-85928

[miguelml99]

Great, I am looking over the OpenCv plus Unity Assets but every time I try to import them from the Unity asset Store the error on the image pops up.

I've tried to solve it the way its recommended in the following link, but it isnt working, do you have any idea on how to solve the issue?

https://proceduralworlds.freshdesk.com/support/solutions/articles/33000216286-failed-to-import-package-with-error-couldn-t-decompress-package

Thanks again for you help and the fast replies!

[MartyG-RealSense]

As the link that you provided suggests, this error can be caused by having insufficient disk space on the drive where Unity is installed. Or perhaps it is unable to write to that drive for some reason because it is read-only. The package is only 128 mb though.

I obtained the OpenCV Plus Unity package from the Asset Store myself in Unity 2019.4 and imported it into an empty new Unity project. The import went fine, though the program does demand that 'unsafe' code is allowed to run. This can be done with the following steps:

1. Go to the Edit > Project Settings menu option to bring up the Project Settings interface and select the Player category.

2. Scroll down to the Allow 'unsafe' code option and place a tick in the box beside it, then exit the Project Settings window.

3. Import the OpenCV Plus Unity package.

[miguelml99]

I have tried what you suggest but the same error keeps appearing. Do you know if I could download the assets as a .zip file ? If thats possible I could try to import the assets folder drirectly form my computer.

[MartyG-RealSense]

Downloading an Asset Store package before important stores it in a folder on your computer. If you open the folder of the downloaded OpenCV Plus Unity package though then it just contains the Unity package file OpenCV plus Unity.unitypackage

So conceivably you could import the file's contents as a custom package through the Assets menu instead of using the Import button on the Asset Store interface. I would think that it would likely have the same results though.

[MartyG-RealSense]

I recalled that there was a GitHub site for OpenCV plus Unity.

https://github.com/Gobra/OpenCV-Unity

[miguelml99]

I solve it by deleting the package and reinstalling it. However, I have been trying to work with it and the Realsense camera, and It seems like the projects from this OpenCV package are only using the RGB frames of the camera, while not taking into account the depth information from the sensor. I am unsure whether this libraries are only trained for recognizing 2D images or there is something critical that i am not considering. Thank again @MartyG-RealSense !

[MartyG-RealSense]

I would expect an OpenCV project to be aimed at ordinary RGB webcams unless the instructions specifically state that they require or support a depth camera.

[miguelml99]

@MartyG-RealSense Ah okey, my bad!

On another subject, I am trying to use the Cubemos SDK from Intel that you recommended me with D435i and Unity. However, I am having some issues, which I believe are due to the fact the the SDK guide uses Unity 2018.4 and Visual Studio 2017 and I have Unity 2019.4 and VS 2019. The issues are the following:

1 When running the Unity programs once all the assets have been imported and the camera is connected, this pops up:

the first error appears on the guide to be solved as follows: However I cannot find the highlighted forlder in my computer nor in the Cubemos folder

Regarding the second Unity error (NullReferenceException) I have not been able to solve it, but Unity points to this line in the code.

2 Lastly, while trying to build the SDK Samples with CMake just as the instructions in the guide suggest, I get this error:

I apologize in advance for the extend inquiry, but I am kinda stuck with this and I really need to try this SDK before moving to other options.

Thanks for all the help!

[MartyG-RealSense]

Intel does not provide technical support for Cubemos Skeleton Tracking SDK, as it is not an Intel product and so I cannot assist with the set of questions above as they all relate to Cubemos. Please contact the help channel at the website of the Cubemos company at the link below.

https://www.cubemos.com/skeleton-tracking-sdk

[miguelml99]

I will do that, thanks!