search

open-ecosystem-development / OpenXR-SDK

Apache License 2.0
2 stars 3 forks source link

Feedback on the use of temporary SDK #26

Open Lixming opened 2 years ago

Lixming commented 2 years ago

We are planning to release a new version of the SDK with 6Dof OpenXR plugin. Since we have shared a temporary version of SDK (named SDK_OpenXR_36DoFProfile_Supported_BaseTemporaryVersion73)for you last time, do you have any problems in the use process? If not, the next released SDK will merge any changes to the 3-6dof profile and officially release it. I hope this merge can ensure the normal operation of your application. @fwei-io @MortimerGoro

fwei-io commented 2 years ago

@Lixming @dingsing2021 From the developer: We have thoroughly reviewed the controller issues with the new runtime. We have also carefully inspected all the code involved and performed lots of testing.

There are two different issues when testing the current code with the new runtime v3.5.0.76, the first one is that most buttons don't work and the second one is the incorrect poses (position&orientation) of the controllers. Let's address them one by one.

With regard to the buttons, we have a patch that modifies the profile for the 6dof controllers based on the info provided by @fwei-io days ago. With that patch all the buttons work as expected except the thumbstick. Click and touch in the thumbstick are properly detected but axis seem not to work.

With regard to the poses, it's a bit more complicated. Good news is that with the new runtime we can remove some of the workarounds we have for the previous one (like inverted controllers). However it looks like the runtime does not provide correct poses (position&orientation) when the 6dof profile is selected. Again we have extensively reviewed our code, and we have tested it both in Huawei glasses and Oculus Quest2 using the OpenXR backend. The code works fine in the Oculus and in the Huawei only if the old profile is used. As soon as we replace it by the new one then some things start to go wrong:

All in all, after reviewing the code and testing it in many different configurations we came to the conclusion that there seem to be a bug in the new 6dof profile which reports wrong poses to OpenXR and thus to Wolvic.

Please help to investigate and respond, looks like OpenXR SDK profile is incorrect or there are bugs in VR SDK runtime. @Lixming @dingsing2021 thank you.

fwei-io commented 2 years ago

The developer has created two demo APK with the latest runtime VR SDK 3.5.0.76.

Wolvic-hvr-arm64-debug-1-2022.03.10.apk In the first APK, the developer just performed the following changes:

As you can see:

Wolvic-hvr-arm64-debug-2-2022.03.10.apk In this second APK, developer performed the following changes:

As you can see:

Hope this could be useful to illustrate the prior comments.

lzhangcs commented 2 years ago

https://user-images.githubusercontent.com/63130594/158440900-ea7afe02-ba23-4388-bdb6-e9757daa64a2.mp4

The developer has tested the provided APK. Developer built the source code and tested it just in case. Both provided the same results. As the developer mentioned before, the runtime is not providing good correct poses. The attached video showcases the issue. For reference, the developer was holding the controller just in front of his face. • the First problem is that the controller in VR is moved in the -z axis as the developer reported. • the Second problem appears when you rotate the controller in place. This is very important, the developer was just rotating it, not translating it into space. As you can see in the video the rotation in place, is shown in VR as a rotation plus a translation in space to the point that the 3D object in the video moves out of the viewport.

Lixming commented 2 years ago

Hi. The coordinate system of the glass is that right x+, up y+, front -z. Thus, the controller in VR is moved in the -z axis is normal. More detail about the pose of both controllers and head can be found in the log by serching the key word "spaceLocation". As about the second problem, this may be caused by some special settings of the controller in our demo. But it may not affect the usage for you.

lzhangcs commented 2 years ago

openxr-log.txt

So the developer has run the example code the HW VR sent previously and it clearly shows that there is indeed some issue with regard to the controller poses. Here attached is a log of a session for the right controller. See below, the developer going through the traces explaining what he did.

He starts from an initial position:

03-21 15:58:26.480 25978 28465 I XRDemo  : spaceLocation_r_position:0.052627,1.744467,-0.805432
03-21 15:58:26.480 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.653508,-0.016915,0.030438,0.756119
03-21 15:58:26.480 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15
03-21 15:58:28.065 25978 28465 I XRDemo  : spaceLocation_r_position:0.056032,1.744530,-0.802776
03-21 15:58:28.066 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.653262,-0.018554,0.029464,0.756330
03-21 15:58:28.066 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15
03-21 15:58:29.723 25978 28465 I XRDemo  : spaceLocation_r_position:0.054375,1.749591,-0.797817
03-21 15:58:29.723 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.656224,-0.016496,0.027528,0.753883
03-21 15:58:29.723 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15
03-21 15:58:31.423 25978 28465 I XRDemo  : spaceLocation_r_position:0.053330,1.749420,-0.798716
03-21 15:58:31.423 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.655834,-0.015734,0.022840,0.754396
03-21 15:58:31.423 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15 ```

and then he proceeds to move the controller to the right (trying not to rotate it not to move it in the other axis)

03-21 15:58:32.994 25978 28465 I XRDemo : spaceLocation_r_position:0.115884,1.741109,-0.812386 03-21 15:58:32.994 25978 28465 I XRDemo : spaceLocation_r_orientation:0.650274,-0.032506,0.043641,0.757749 03-21 15:58:32.994 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 03-21 15:58:34.651 25978 28465 I XRDemo : spaceLocation_r_position:0.276600,1.737609,-0.797731 03-21 15:58:34.651 25978 28465 I XRDemo : spaceLocation_r_orientation:0.643707,-0.066877,0.107888,0.754672 03-21 15:58:34.651 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 03-21 15:58:36.365 25978 28465 I XRDemo : spaceLocation_r_position:0.595336,1.767161,-0.670053 03-21 15:58:36.365 25978 28465 I XRDemo : spaceLocation_r_orientation:0.652741,-0.156735,0.135869,0.728630 03-21 15:58:36.365 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 ```

Note that the x coordinate in the position moves from 0.05 up to 0.65 while the other values are mostly unchanged, that's fine. Then after that, he does the opposite and moves the controller in the x-axis to the left.

03-21 15:58:38.009 25978 28465 I XRDemo  : spaceLocation_r_position:0.478229,1.773441,-0.699614
03-21 15:58:38.009 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.665118,-0.116318,0.124921,0.726968
03-21 15:58:38.009 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15
03-21 15:58:39.722 25978 28465 I XRDemo  : spaceLocation_r_position:-0.060650,1.755763,-0.750182
03-21 15:58:39.722 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.665862,0.056412,-0.065336,0.741065
03-21 15:58:39.722 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15
03-21 15:58:41.408 25978 28465 I XRDemo  : spaceLocation_r_position:-0.425639,1.781836,-0.622047
03-21 15:58:41.408 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.648906,0.185012,-0.181924,0.715259
03-21 15:58:41.408 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15
03-21 15:58:43.065 25978 28465 I XRDemo  : spaceLocation_r_position:-0.458477,1.777952,-0.602227
03-21 15:58:43.065 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.642209,0.190463,-0.194097,0.716671
03-21 15:58:43.065 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15 ```

It's easy to see how the x coordinate goes from 0.47 (far right) to -0.45 (far left). The other values remain more or less constant indeed. Everything is OK so far. Then developer moves the controller back to the initial position.

03-21 15:58:44.765 25978 28465 I XRDemo : spaceLocation_r_position:-0.202256,1.751532,-0.715268 03-21 15:58:44.765 25978 28465 I XRDemo : spaceLocation_r_orientation:0.661512,0.107851,-0.112978,0.733489 03-21 15:58:44.766 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 03-21 15:58:46.423 25978 28465 I XRDemo : spaceLocation_r_position:0.084631,1.729904,-0.774212 03-21 15:58:46.423 25978 28465 I XRDemo : spaceLocation_r_orientation:0.666148,-0.006602,-0.000548,0.745790 03-21 15:58:46.423 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 03-21 15:58:48.123 25978 28465 I XRDemo : spaceLocation_r_position:0.105706,1.727637,-0.786073 03-21 15:58:48.123 25978 28465 I XRDemo : spaceLocation_r_orientation:0.661671,-0.010508,0.016141,0.749547 03-21 15:58:48.123 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 03-21 15:58:49.751 25978 28465 I XRDemo : spaceLocation_r_position:0.085051,1.732779,-0.783295 03-21 15:58:49.751 25978 28465 I XRDemo : spaceLocation_r_orientation:0.663910,-0.001345,0.011777,0.747718 03-21 15:58:49.751 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 ```

And once the developer is there he proceeds to rotate the controller 90º in a clockwise direction without translating it, just rotating in place.

03-21 15:58:51.880 25978 28465 I XRDemo  : spaceLocation_r_position:0.362211,1.630215,-0.791919
03-21 15:58:51.881 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.624795,-0.188769,-0.252073,0.714463
03-21 15:58:51.881 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15
03-21 15:58:54.723 25978 28465 I XRDemo  : spaceLocation_r_position:0.567861,1.267276,-0.801137
03-21 15:58:54.723 25978 28465 I XRDemo  : spaceLocation_r_orientation:0.477171,-0.427617,-0.539205,0.546543
03-21 15:58:54.723 25978 28465 I XRDemo  : spaceLocation_r_locationFlags: 15 ```

Now the orientation values do change which is correct, **but** the problem is that the reported position in the x-axis goes up to 0.56 which is more or less equivalent to the translation to the right he did in the beginning with the difference that in this case, the developer hasn't moved the controller at all.

Then the developer rotates back the controller to the initial position and performs a 90º counter-clockwise rotation.

03-21 15:58:57.312 25978 28465 I XRDemo : spaceLocation_r_position:0.093565,1.778994,-0.737444 03-21 15:58:57.312 25978 28465 I XRDemo : spaceLocation_r_orientation:0.691768,0.016662,-0.100299,0.714926 03-21 15:58:57.312 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 03-21 15:58:59.710 25978 28465 I XRDemo : spaceLocation_r_position:-0.485855,1.242033,-0.714562 03-21 15:58:59.710 25978 28465 I XRDemo : spaceLocation_r_orientation:0.501161,0.511914,0.467713,0.517712 03-21 15:58:59.710 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 03-21 15:59:01.967 25978 28465 I XRDemo : spaceLocation_r_position:-0.429089,1.515252,-0.713725 03-21 15:59:01.967 25978 28465 I XRDemo : spaceLocation_r_orientation:0.598531,0.387662,0.300998,0.633150 03-21 15:59:01.967 25978 28465 I XRDemo : spaceLocation_r_locationFlags: 15 ```

Again same weird result. Rotating the controller in place is incorrectly reported as a translation (the x coordinate position moves from 0.09 to -0.42 when the developer hasn't translated it at all).

lzhangcs commented 2 years ago

Some further information:

  1. The developer has also checked the logs directly from the system (by filtering by sixDofPostureVector in adb logcat without launching any application). In that case, the poses look good, there is no translation in the position when the controller is rotated. It looks like the OpenXR implementation does something weird in-between.

  2. The developer has managed to circumvent the issues he was seeing by doing a special workaround for the HW hardware. The way he managed to do that is: by exchanging the poses returned for the grip and the aim. The developer belives that the OpenXR current implementation is exchanging them by mistake.

Lixming commented 2 years ago

Hi. We did make some changes to aimpose to convert the pose we actually obtained to a correct aiming pose，when the controller needs to draw rays. Instead, the grip pose will do nothing, and it will be equal to the value obtained from the runtime. Thus, when you rotate the controller, the aim pose will change.

lzhangcs commented 2 years ago

Thanks, Lixming. Some more info. from the developer, see if this will help you to understand his case more. Please let me know if you have more info. to rely to the developer: "Basically what I've done in Wolvic to workaround the issue is to exchange grip and aim poses. So I'm using the aim pose as it was the grip pose and vice-versa. Note that this is just a workaround for HW 6DoF while the issue in the runtime is not fixed, I don't have to do that for other devices like the Oculus Quest for example."

Lixming commented 2 years ago

Thanks for your feedback. We have reviewed our implement for the aim pose again and found that it did have some incorrect conversion. It will be fixed in the next version. By the way, do you have any other problems about 3.5.0.76 SDK?

lzhangcs commented 2 years ago

Thanks, Lixming. Do you have an ETA for the fix? This is the major open issue right now. I'll pass on you the info. if any further findings.

Lixming commented 2 years ago

If there are no other questions, we plan to release a new version next week. In addition, we are willing to provide a revised version for you to determine whether you can solve the current problem. Finally, we are deeply sorry for the trouble you have encountered.

svillar commented 2 years ago

So there is another issue we have with the thumbsticks. We can properly detect click and touch events in the thumbsticks but we get no information from the axis. For your information we're creating bindings for /interaction_profiles/huawei/6dof_controller with pathnames /user/hand/right/input/thumbstick and /user/hand/left/input/thumbstick.

When checking the logs I see this just after trying to create the binding

03-22 14:01:21.397  4318  5814 W HxrBinding: [@SDK][HxrBinding.cpp:addKeyToMatchingBindings](80): not find binding path
03-22 14:01:21.397  4318  5814 W HxrBinding: [@SDK][HxrBinding.cpp:addKeyToMatchingBindings](80): not find binding path
03-22 14:01:21.397  4318  5814 W HxrSession: [@SDK][HxrSession.cpp:getBinding](1273): No bindings
03-22 14:01:21.397  4318  5814 W HxrSession: [@SDK][HxrSession.cpp:getBinding](1273): No bindings
Lixming commented 2 years ago

Please use the following path to get position from the axis: • …/input/thumbstick/x • …/input/thumbstick/y

svillar commented 2 years ago

Please use the following path to get position from the axis: • …/input/thumbstick/x • …/input/thumbstick/y

Yeah, I can read the thumbstick positions by retrieving the value of input/thumbstick/x (or y) if and only if I create the actions with XR_ACTION_TYPE_FLOAT_INPUT and read them with xrGetActionStateFloat.

However that's not the way it should work. As mentioned by Imanol in this issue from last year, this is not how thumbsticks values should be read. Thumbstick axis should be read using XR_TYPE_ACTION_STATE_VECTOR2F. That's the way it works with Oculus runtime or with Monado runtime.

Also the specs seems to point in the same direction. The WebXR gamepad module says "Related axes (such as both axes of a thumbstick) MUST be grouped". And the OpenXR specs mentions that "For actions created with XR_ACTION_TYPE_VECTOR2F_INPUT when the runtime is obeying suggested bindings: The suggested binding path must refer to the parent of input values instead of to the input values themselves".

This means that for things like thumbsticks the applications should use ../input/thumbstick as path and XR_ACTION_TYPE_VECTOR2F_INPUT as action type. Then they should use xrGetActionStateVector2f to read the x and y coordinates in the very same call.

lzhangcs commented 2 years ago

Hi @Lixming, Is the VR new release ready per the original plan? As I emailed Weiding yesterday, please also include the fix for above @svillar's issue in this new release. Thanks a lot.

Lixming commented 2 years ago

Hi. svillar's issue is releated to the revision of the profile which we have been contributing to the khronos. Thanks for svillar's suggestion and we will discuss how to modify it appropriately.

Lixming commented 2 years ago

https://github.com/hms-ecosystem/OpenXR-SDK/issues/26#issuecomment-1076923395 We decided to fix this bug as svillar's suggestion. ../input/thumbstick as path and XR_ACTION_TYPE_VECTOR2F_INPUT as action type. Thanks a lot for the advice. @svillar By the way, we will send a tempory version of SDK for you to test the function next week, before the official version released. Thus, if there is any other question, please tell us soon.

svillar commented 2 years ago

#26 (comment) We decided to fix this bug as svillar's suggestion. ../input/thumbstick as path and XR_ACTION_TYPE_VECTOR2F_INPUT as action type. Thanks a lot for the advice. @svillar By the way, we will send a tempory version of SDK for you to test the function next week, before the official version released. Thus, if there is any other question, please tell us soon.

That's really great to hear! Many thanks for considering our suggestions

lzhangcs commented 2 years ago

@Lixming Is the new release ready to use? Please let us know. Thanks

fwei-io commented 2 years ago

@Lixming svillar has tested the new (debug) runtime 3.5.0.77. These are the comments:

  1. Controllers are no longer inverted, that's fixed. Workaround not needed anymore
  2. Controllers poses are correct, that's fixed. Workaround not needed anymore
  3. thumbstick axis positions are properly read using the "input/thumbstick" path and reading the values using a vector, that's fixed. Workaround not needed anymore
  4. Security zone not shown for the stationary experience. If I move out of the security zone nothing is shown. In the past, a blue line was shown. I found that this was not an issue in Wolvic (see this SDK issue). But now it looks like not even the blue line is shown (I tested this with Huawei's OpenXR sample as well).

So the only remaining issue is # 4., i.e., the security zone is not shown when you move the controllers out of it, that happens in the VR environment even without having any application running.

Thanks

Lixming commented 2 years ago

"the security zone is not shown when you move the controllers out of it"--This bug will be fixed in the next SDK version.