oculus-samples / Unity-Movement

Body, Eye and Face Tracking code sample.
Other
274 stars 47 forks source link

How to make it work with Meta XR OVR Interaction for grabbing using hand and ray ? #84

Open ramkeshcse084 opened 2 months ago

ramkeshcse084 commented 2 months ago

I want to add feature of Grabbing . For this i used OVR camera rig and from example scene. image When i test in Meta device my meta movement character hands is not sync with my Interaction hands and controller. There is little difference in it. We will use this in multiplayer .What we have to do for it ? I am sharing video for reference.

https://github.com/user-attachments/assets/2e513fa5-4d42-48df-847c-4da1c607ac64

Please guide me what will be best way to do it ? Thank You

sohailshafiiWk commented 2 months ago

Hello, can you take a look at the ISDK integration sample scene? You may find documentation here https://developer.oculus.com/documentation/unity/move-samples/#isdk-integration about that sample.

ramkeshcse084 commented 2 months ago

Hello, Thank you for quick reply. I just tested Sample MovementISDKInteraction scene .

https://github.com/user-attachments/assets/b0e332fe-0c9b-41d9-aa93-11f7b05886ee

Grabbing is not working there. When I just opened scene Hands rotation was not ok, when I start recording rotation of hands come in good position. Here is reference video :

ramkeshcse084 commented 2 months ago

I followed docs https://developer.oculus.com/documentation/unity/move-samples/#isdk-integration to integrate ISDK on my character. Grabbing is working but same issue with additional one more issue. 1. Gap in Character hands and synthetic hands 2. Character wrist is getting disconnected to character. I am sharing Screenshot and video for better understanding. image

https://github.com/user-attachments/assets/612bcb30-d6c9-465f-8bb9-a9d0d42980f7

sohailshafiiWk commented 2 months ago

Can you tell us the versions of the Meta XR packages that you've included? You can find this in Unity Package Manager or your project's manifest.json. Please list the versions of packages with the prefix com.meta.xr.sdk, such as com.meta.xr.sdk.core and so on. I assume that you are using Unity 2022.3.39f1 as well?

ramkeshcse084 commented 2 months ago

Yes, Unity version is same as mentioned. Here is Meta SDKs with version from package manager: image

Thank You

sohailshafiiWk commented 1 month ago

Hello, please download the attached project. Go to the Meta Movement package in the Unity Package Manager, and import the advanced samples. Please try the ISDK scene as it is imported into the Assets folder. Testv67Int.zip

Please note that you might see an error related to Unity input system. We are currently looking into that issue.

ramkeshcse084 commented 1 month ago

Hello, Thank You. I just tested ISDK scene in given project, having same issue not able to grab blue cup. Here is video for ref.

https://github.com/user-attachments/assets/daf477b7-176d-44e1-9e00-907091a5c132

sohailshafiiWk commented 1 month ago

Could you tell us what OS version your device is running on? This can be found in the "Software update" section of system settings. Which device is this running on (Quest 2? Quest 3?)?

ramkeshcse084 commented 1 month ago

Here is version details :
image Thanki You

sohailshafiiWk commented 1 month ago

Thank you for the information. We have been looking into this problem and will report back when we have a fix available.

ramkeshcse084 commented 1 month ago

ok. Thank You. We are waiting for this. When fixed done, please us know.

sohailshafiiWk commented 1 month ago

Hello. We have provided this fix (https://github.com/oculus-samples/Unity-Movement/commit/d837b0ccd72a9b9e31115d656548172bb08b1ad0) to #dev branch and will do a release to main and do a document update once that process is complete.

In the meantime, please make the following changes that the fix applies, which use Capsense (https://developers.meta.com/horizon/documentation/unity/unity-capsense/):

  1. Delete OVRControllersHands and also the references to its SkeletalHandsAdjustment component under ArmatureSkinningUpdateRetargetSkeletonProcessor; this means deleting the entries LeftControllerHandSynthentic and RightControllerHandSynthentic. The LeftHand/RightHandSynthetic references will remain.
  2. In OVRCameraRig->OVRManager, navigate to the "Controller Driven Hand Poses Type" and set it to Natural.
  3. In each OVRHandPrefab under the related hand anchors (such as OVRCameraRig->TrackingSpace->LeftHandAnchor->OVRHandPrefab), navigate to the OVRHand component and set "Show State" to Always (not something like "Contoroller Not In Hand").

Because OVRControllerHands will be deleted and you can rely on OVRHands to be driven by controllers inputs, the scene will be simpler. The ISDK scene that is fixed in that revision on the #dev branch can be used as a reference.

ramkeshcse084 commented 1 month ago

ok. Please let me know when new version of Meta movement SDK will arrive. I have another question related to Meta movement SDK update; I will ask in other thread.

ramkeshcse084 commented 1 month ago

I

https://github.com/user-attachments/assets/b4c3ceeb-5e22-41cd-96cf-957e31f90968

followed given steps, now ISDK Samples scene started working. When I implemented this step with my locomotion character, grabbing is working with hand tracking but there is gap in Character hands and tracking hands. Any idea how to fix it? We are at initial stage of issue . Here is video for ref.

sohailshafiiWk commented 2 weeks ago

For this character, navigate to the RetargetingLayer's RetargetingProcessors section, go to the "Correct Hand" processor and disable "Use Secondary Bone Id" checkbox for the "Left Hand Processor" and "Right Hand Processor" sections.

image

With this setting off, the hand should look more accurate, however the upper arm won't be rotated toward the tracked lower arm position. This means that the arm bones above the wrist might look incorrect.

Furthermore, the mesh appears to disconnect from itself when the bone positions are stretched beyond what the character is designed for. You can consult https://developers.meta.com/horizon/staging/moveUnity/documentation/unity/move-body-tracking#modifying-character-height-to-match-varying-user-heights for more information regarding how to adjust the model.