search

Reality-Dev / BodyTracking

A swift package making it easy to implement body tracking in ARKit and RealityKit.
Other
77 stars 15 forks source link

Body tracking 2D version - spine joints? #3

Closed csabau closed 1 year ago

csabau commented 2 years ago

Hi there,

I'm extremely new to coding in Swift, but I'm trying to create this project for my MSc that uses body tracking with an iPhone. Being such a new technology the resources are extremely scarce so THANK YOU for making this public!

I struggle to figure out how to add more joints to the spine of the 2D body. Is that even possible? I see the 3D version has 7 different joints for the spine. Would that be because it's using ARBodyTrackingConfiguration instead of ARWorldTrackingConfiguration (used in 2D)?

I'm trying to figure out a way to have more joints in the spine so that spine flexion is detected / angles between joints can vary. Although 3D has more joints, it is less accurate for some reason, the joints are often displayed out of the body, whereas the 2D seems much more consistent.

Can more joints be added to the 2D tracking?

Many thanks!

Reality-Dev commented 2 years ago

Hi there, Unfortunately with the built-in ARKit and Vision algorithms, we do not have any access to any more joints. We did get two more joints (left and right ear) announced at this year's WWDC in the "Discover ARKit 6" video. We also got improved performance and stability with the 3D tracking (and my package has a smoothing option built in that uses simple linear interpolation) to quote the video "We also improved the overall pose detection. On iPhone 12 and up, as well as on the latest iPad Pro and iPad Air models with the M1 chip, the 3D skeleton, as shown in red, has been improved. You will experience less jitter and overall more temporal consistency. Tracking is also more stable if parts of the person are occluded or when walking up close to the camera. To make use of the improved MotionCapture, set your deployment target to iOS 16 in your Xcode settings." To get more 2D joints you have a couple of options. One is to use a different Machine-Learning algorithms on your ARFrame's Captured image - either a model you create and train yourself or one you find from somewhere else. Here are two examples of using ML models with AR (for other purposes). This one is with CoreML (which is probably more helpful for you) and this one is with Vision. Another option is to use the project method of ARView to project the 3D joint coordinates onto the 2D screen, effectively giving you 2D joints. I hope this was helpful! Good luck!