d435 When will body tracking, bone information, gestures, and face information functions be provided?

[asalis]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	d435
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	{Win (8.1/10) / Linux (Ubuntu 14/16/17) / MacOS
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	all
SDK Version	librealsense2.0 }

Issue Description

<Describe your issue / question / feature request / etc..>

[unagiHuman]

Hi @asalis

If it is BodyTracking, it is not an Intel product, but there is software like below. https://nuitrack.com/#rec38627247

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hello @asalis,

The RealSense SDK 2.0 is focused on providing depth across multiple operating systems and wrapper. All open source. We provide a few code samples which we hope the community will add to. We also provide tools like the viewer. What we are hoping to do is expand into middleware like person tracking or scanning via third-party partners. Please watch our site as we bring these partners on board.

Regards, Jesus G. Intel Customer Support

[djnugent]

How long can we expect to watch the site? A month? a year?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hello @djnugent,

Unfortunately, there is not a specific timeframe. Projects are ongoing continuously and we will announce partnerships and products as we are legally allowed after they are completed.

Regards, Jesus G. Intel Customer Support

[moyayo]

has anyone tested the Nuitrack for skeleton tracking? how does it compare to the Kinect v2

[threat0day]

@moyayo , I'm currently working nuitrack. well recognizes in front, on other sides there are small problems.

[moyayo]

@threat0day, I was finally able to test it using the d415 and the results were good, but the kinect is better. The problem is that the skeleton turns into the garbage and then switches back to good again. I hope they continue improving it since it is the best current solution

[threat0day]

@moyayo i tried using OpenPose, but there output 2d in default. Exist 3d point cloud module, but bind to D415 i could not. OpenPose very good tracking person.

[BerryOne]

D435 and RealSense SDK 2.0,can use these get the infomation of hand position and posture now?Thanks

[threat0day]

@BerryOne not yet

[BerryOne]

@threat0day,i just need the information of hand position and orientation,what can i use,nuitrack orOpenPose?Thanks

[threat0day]

@BerryOne, for 3d recognition hand, gestures you can using NuiTrack, but without fingers. In your case better using leapMotion

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Is this still an issue? If not, ticket will be closed in 5 days.

[wtatters]

This is an ongoing question of interest to many - Nuitrack has some good possibilities - though still not as accurate as we would like in some user cases .... but worth a look.

Westa

[wtatters]

Fwiw - openpose licencing for commercial uses is just plain dumb .... just saying

[paulhoux]

And FWIW, Nuitrack only supports a single camera per machine. If you want to use more than one camera to cover a larger area (which is one of the reasons to choose RealSense over the competition because of its dual-IR depth tracking), you're out of luck. One way to work around that, is to use a little NUC per camera, do the detection on the NUC and transmit the results across the network.

Personally, I really don't get why Intel chooses to just provide raw data, instead of processed data like skeletons or human tracking. Your product is a glorified webcam, no more no less. You're putting all the hard work on the shoulders of your target audience. RealSense is not worth the investment.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] hi @paulhoux,

We are working with several software vendors to provide solutions such as skeletal tracking and object tracking. We do plan to have a page that points people to software vendors, the page is not ready yet, but we have the first vendor LIPS released their SDK supporting Realsense cameras, please visit http://www.lips-hci.com/downloads/category, and select the model as intel Realsense camera.

[vpenades]

@RealSense-Customer-Engineering

I believe what users are asking for is an off the shelf solution for skeletal/hand tracking, integrated with the RealSense SDK.

From what I've seen so far, third party vendors don't "extend" RealSense SDK, they completely wrap it and offer developers an entirely different SDK. This makes extremely difficult to switch from one vendor to another, or even to use combinable features of two different vendors, let's say, skeleton tracking of vendor A and face tracking of vendor B.

Also, RealSense SDK is still in beta state, and third party vendor's SDKs are also in beta state. In my book, beta x beta = alpha state, extremely unstable and difficult to debug.

In my current case, I'm using one of these third party vendor SDKs that use RealSense D435 under the hood, with libreal SDK completely hidden. The whole thing is failing to detect the sensor once every two times. Who's at fault? the third party's SDK, or libreal SDK? who knows... because I can't debug under the hood.

Also, my language of choice is C# ... what's the point of having a libreal c# wrapper if third party vendors completely wrap the native code with its own wrapper code? it forces everyone to reinvent the wheel down the chain of developent.

[wtatters]

Hi There,

Would be intersted in knowing which SDK's you are testing with. We have found connection issues with nuitrack on second connection attempts - the work around for us was to create a realsense context as well - and force a hardware reset on the d435 using the realsense context - BEFORE each attempt to start the nuitrack system.

Provided you are careful - the realsense context can be started and released - without affecting the nuitrack sdk.

Westa

[vpenades]

@wtatters, yes, on that particular case is nuitrack. We can try the hardware restart to see if it solve our issues.

But we're leveraging other SDKs, and all of them suffer from a lack of stability and robustnes. Most of this stuff is fine to use for experiments, but so far, nothing can be reliably used in real consumer scenarios.

We already have a Kinect based commercial product being sold to consumers, we want to replace the Kinect with newer sensors, but seeing the state of the art, it's obvious that the new stuff is far from ready for consumers.

[wtatters]

@vpenades we are in a similar position - having spent the last 12 months investigating new options. The thing most people seem to forget is that the Kinect 2 system was using a true TimeOfFlight sensor being sold at a price well below what even current true ToF sensors are charging. Until someone gets a price compatible ToF sensor to market we are dubious about commercial applications. Westa

[rofoto]

@wtatters

exactly the same position, I didn't want to go with the kinect v2 ( as it is now EOL and discontinued ) and opted for the D435. I am quite disappointed with the lack of support from @RealSense-Customer-Engineering on not only poor docs but a lack of caring in implementing things they had running on older versions of the hardware

[wtatters]

D435 is not a good product for our needs - not exactly sure who it would fit - but if you only need 50cm depth and wide angle scan it may be an option - for everyone else - D415 is a better fit IMHO.

Westa

[vpenades]

@etcétera the very only reason most people originally bid for the d435 is precisely the wide angle. With Kinect you have full body tracking within 2 metres from the camera. With all other cameras you have to move far away. In many scenarios you often don't have that much room, so sometimes wide angle cameras are the only solution

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] hi @rofoto, If you are talking about MW that we will have 3rd party to work with that, this still the direction, else, please file issue ticket so we can help. For poor docs? please help to point out what kind of doc we are missing, we can improve, and welcome your contribution too.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Our 3rd Party Software Referral Page is now live. We will continue to add ISVs as more become available. https://realsense.intel.com/software-for-intel-realsense/

[lucasjinreal]

@RealSenseCustomerSupport serveral sites provided in that link were down