search

Unity-Technologies / arfoundation-samples

Example content for Unity projects based on AR Foundation
Other
2.98k stars 1.11k forks source link

[Bug] ARBodyTracker is reporting wrong values, and it does not seem to be an Apple issue. #1040

Closed dganzella closed 1 year ago

dganzella commented 1 year ago

Unity bug report case number IN-25303

Describe the bug

When using Body Tracking and using the code from the AR Foundation Samples project, the head, hands and feet joints are not reporting values (local pose)

When I log values, and compare them between very different poses I made, I can see that only a few joints actually have values in them in unity, these are:

Root (1) LeftUpLeg, LeftLeg, RightUpLeg, RightLeg (4) Spine1,Spine2,Spine3,Spine4,Spine5,Spine6,Spine7 (7) LeftShoulder1,LeftArm,LeftForearm, RightShoulder1,RightArm,RightForearm (6) Neck1,Neck2,Neck3,Neck4 (4)

Screen Shot 2022-12-08 at 12 02 00 Screen Shot 2022-12-08 at 12 02 03 Screen Shot 2022-12-08 at 12 02 13

Here are the sample files if you wanna check for yourself: human_pose_1670505250.txt human_pose_1670505256.txt human_pose_1670505260.txt human_pose_1670505273.txt human_pose_1670505230.txt

What calls my attention is that 5 very important bones are missing having values:

Head, Left Hand, Right Hand, Left Foot and Right Foot.

Apple's ARKIt have those available joints on the ARBodyAnchor.skeleton:

ARSkeleton.JointName.root
**ARSkeleton.JointName.head**
ARSkeleton.JointName.leftShoulder
**ARSkeleton.JointName.leftHand**
ARSkeleton.JointName.rightShoulder
**ARSkeleton.JointName.rightHand**
**ARSkeleton.JointName.leftFoot**
**ARSkeleton.JointName.rightFoot**
Screen Shot 2022-12-08 at 11 32 16

so, my guess is, The guy who implement the unity version tried to "break down" the poses provided by apple to fill the in-between joints that were missing (like spines and necks), and in doing so, they probably missed an index, thats why head, hand and feet themselves are missing.

To Reproduce Steps to reproduce the behavior: Load the sample project. Open body tracking. Try moving wrists, moving feet or moving head, none of them are actually moving the robot at all.

Expected behavior Hand, feet and head should move.

Actual behavior Hand, feet and head do not move.

Smartphone (please complete the following information):

Thanks a lot!!

Related: https://github.com/Unity-Technologies/arfoundation-samples/issues/920 https://issuetracker.unity3d.com/issues/arkit-body-tracking-neck-joint-s-incorrectly-reports-rotations-when-the-scanned-person-rotates-their-head https://forum.unity.com/threads/arkit-3d-body-tracking.1370118/

The response is always:

"We were able to reproduce visibly inaccurate neck tracking in a variety of body poses using ARKit; however, these values come from ARKit itself, and ARFoundation makes no modifications to the joint transforms. Please direct further questions to Apple for this issue."

"AR Foundation simply reports the platform provider's tracking data into your Unity scene. Feedback about the quality of the tracking is better directed to the platform provider who provides the tracking implementation (in this case, Apple)."

But Im pretty sure theres the issue is in the conversion of data (Native -> Unity)

dganzella commented 1 year ago

I just tested the native build of apple and overall it seems terrible as well. But it seems slightly better because, at least, the positions and rotations are on the proper bone. But its the same issue w/ bad quality

andyb-unity commented 1 year ago

Hi @dganzella,

I just wanted to update you that we have received your bug report and hope to investigate this issue soon. As this has come up a few times, I'd like us to write an automated test that compares all the C# joint values against everything Apple provides in C++, but due to resource constraints we haven't been able to assign this work to the team yet.

jackt-unity commented 1 year ago

Hi @dganzella ,

Thanks for your report and patience. From digging into native ARKit's body tracking and running their official sample (https://developer.apple.com/documentation/arkit/content_anchors/capturing_body_motion_in_3d), the issue has been existing in ARKit already.

Please check the WWDC video (start from 27:40), they pointed out the green joints are trackable, and the yellow joints are not.

In both ARFoundation and ARkit, I logged all joints' transformation to check which joints's transformation got updated, and found both of them have the same certain joints got updated, and they are the green joints mentioned in the video.

ARFbodytrack_timestamp_1.txt ARFbodytrack_timestamp_2.txt ARkit-BodyTracking-timestamp_1.txt ARkit-BodyTracking-timestamp_2.txt

Please find the above logs and check the difference of ARFbodytrack_timestamp_1 and ARFbodytrack_timestamp_2, and the difference of ARkit-BodyTracking-timestamp_1 and ARkit-BodyTracking-timestamp_2. They both have the same certain joints got updated.

ARFoundation is just receiving the joints' transformation value from ARKit. So this issue should be resolved once ARKit's body tracking get improved.