OpenXR - Basic integration for Meta Quest

[lvonasek]

Introduction

This Pr adds OpenXR support for Meta Quest (2,3,Pro). Using the smartphone Android version in the headset is very hard due to missing controllers support and relative mouse pointer. Intent of this PR is to add full controller support and render the screen using OpenXR (no stereo/6DoF).

How does it work

In the code is detection if it is running on a Meta/Oculus device. The OpenXR is initialized only if the detection indicates it runs on a XR headset. By that said, it means the same APK will be possible to use on mobile and XR. This is possible due to hybrid apps support (Hybrid app == part of the app could be 2D and another XR).

Instead of drawing on a screen, it is rendered into OpenGL framebuffer which is then transformed into a flat screen in XR space. Mouse cursor is still relative but it is mapped on controller translation which works perfectly even in games. Controller buttons are mapped to most common game keys.

Notes

• Anyone who spams this discussion with unrelated stuff will be without warning blocked
• The main intent is to support Linux programs inside of the headset
• This addresses https://github.com/termux/termux-x11/issues/564
• The same approach was done in Winlator project (video)

[lvonasek]

Thank you for the screenshot. It seems like a GitHub issue not showing it on my side.

1) EditText It is the only way I managed to get the software keyboard output. The system keyboard on Quest doesn't fulfill the Android API.

2) Brackets Right, I will fix the formatting

3) JVM and activity pointers If I remember correctly the call returned XR_ERROR_HANDLE_INVALID. The docs are not specifying it: https://registry.khronos.org/OpenXR/specs/1.0/man/html/XrLoaderInitInfoAndroidKHR.html

[twaik]

• It is the only way I managed to get the software keyboard output. The system keyboard on Quest doesn't fulfill the Android API.

And I told you what exactly you can do to investigate why IME does not show up.

If I remember correctly the call returned XR_ERROR_HANDLE_INVALID. The docs are not specifying it:

Ok, so both must be valid pointers. JVM pointer is not a problem for termux-x11 X server process, but Activity can be. Can you please investigate what functions it invokes? You can use ContextWrapper for this, just do not attach real context.

[twaik]

In the case if we can fake or proxy the functions needed by OpenXR we can move rendering code to X server process completely. And maybe input related code too. And this will improve performance.

[lvonasek]

• It is the only way I managed to get the software keyboard output. The system keyboard on Quest doesn't fulfill the Android API.

And I told you what exactly you can do to investigate why IME does not show up.

The problem is not with showing the IME, the problem is reading the characters as the software keyboard writes it directly into UI and not calling key events.

If I remember correctly the call returned XR_ERROR_HANDLE_INVALID. The docs are not specifying it:

Ok, so both must be valid pointers. JVM pointer is not a problem for termux-x11 X server process, but Activity can be. Can you please investigate what functions it invokes? You can use ContextWrapper for this, just do not attach real context.

I will give it a try.

[twaik]

The problem is not with showing the IME, the problem is reading the characters as the software keyboard writes it directly into UI and not calling key events.

And again, we will not be able to reproduce normal behaviour until you make a minimal viable TextView-based view which can receive input normally.

My guess is that Quest's IME simply sends characters through InputConnection but we will not figure it out without testing on real device...

[lvonasek]

I looked into the code of EditText and TextView but it contains a lot of internal classes. I didnt find enough time to put it everything into the project and to figure out what parts are needed.

Another option would be to use this extension https://developer.oculus.com/documentation/native/android/mobile-openxr-virtual-keyboard-sample/ . It would give more control over the keyboard. Unfortunatelly, it doesn`t support pasting from clipboard which is at least from my point of view a no-go. I will invest more into the EditText use but it might take longer.

---

I tried mocking the Context using ContextWrapper. In the beginning it required getAssets, getClassLoader, createPackageContext, getApplicationInfo and about ten others but then it throwed some exception from com.oculus.vrapi.PackageValidator which I wasnt able to satisfy (I dont have the relevant stacktrace because as it was called from JNI).

[twaik]

My guess is that Quest's IME simply sends characters through InputConnection

...

[lvonasek]

I tested InputConnection but didn't have any success with it. It seems Quest is using another way.

[twaik]

You always can put a breakpoint in your input handling code and explore the stacktrace to find the code that triggers it. Probably it will be better to publish the whole stacktrace here.

[twaik]

Or you can put Log.e("stacktrace", "text", new Throwable()); to input handling code to put the stacktrace to the log directly.

[lvonasek]

ok, it is using InputConnection. I must be doing something wrong:

java.lang.Throwable
    at com.termux.x11.XrActivity.afterTextChanged(XrActivity.java:205)
    at android.widget.TextView.sendAfterTextChanged(TextView.java:10805)
    at android.widget.TextView$ChangeWatcher.afterTextChanged(TextView.java:13820)
    at android.text.SpannableStringBuilder.sendAfterTextChanged(SpannableStringBuilder.java:1278)
    at android.text.SpannableStringBuilder.replace(SpannableStringBuilder.java:578)
    at android.text.SpannableStringBuilder.replace(SpannableStringBuilder.java:508)
    at android.text.SpannableStringBuilder.replace(SpannableStringBuilder.java:38)
    at android.view.inputmethod.BaseInputConnection.replaceText(BaseInputConnection.java:941)
    at android.view.inputmethod.BaseInputConnection.setComposingText(BaseInputConnection.java:712)
    at com.android.internal.view.IInputConnectionWrapper.executeMessage(IInputConnectionWrapper.java:633)
    at com.android.internal.view.IInputConnectionWrapper$MyHandler.handleMessage(IInputConnectionWrapper.java:111)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loopOnce(Looper.java:214)
    at android.os.Looper.loop(Looper.java:304)
    at android.app.ActivityThread.main(ActivityThread.java:7918)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1010)

[twaik]

Probably onCreateInputConnection can work without EditText so you can move it to LorieView and get rid of EditText completely.

[twaik]

I think you should try to replace extending EditText with extending with View (and making it focusable and focusable in touch mode like LorieView) and if it works you can make LorieView return your InputConnection implementation in the case if listener is set.

[lvonasek]

I would like to keep it out of LorieView to keep XR and Android behaviour separated.

There is GLSurfaceView used, I prefer to create XrSurfaceView and put there the custom InputConnection and other XR specific stuff.

[ewt45]

termux-app also use oncreateinputconnection with custon view. matbe you can take a reference https://github.com/termux/termux-app/blob/2f40df91e54662190befe3b981595209944348e8/terminal-view/src/main/java/com/termux/view/TerminalView.java#L269

[twaik]

termux-app also use oncreateinputconnection

termux-app works with keyboard pretty much different way...

[lvonasek]

Just to update: I am struggling with the removal of EditText dependency from XrKeyboard. I have to concentrate the whole May on other projects, I will revisit this afterwards.

[lvonasek]

Working on the keyboard support somehow killed my motivation to continue on this project. Not sure if I find energy to come back to this project.

[lvonasek]

The behavior on the VR headset changed. Now it is simple to integrate software keyboard.

@twaik, I would appreciate a code review.

[twaik]

I am not skilled enough to check your C/Java OpenXR code, but I think I can review the Java part.

1. Is there a reason to have XrActivity::isEnabled if there is XrActivity::isSupported? I mean do we really need a wrapper?
2. In dispatchKeyEvent you check only KeyEvent.ACTION_UP action, and not checking KeyEvent.ACTION_DOWN at all. Also you compare system time of event. Probably you can check the KeyEvent::getDownTime and check if it is the same as one before this. I am pretty sure 4 consequent Enter KeyEvents will have the same down time. Also you can check the KeyEvent::getRepeatCount and ignore repeats.
3. About GLUtility::drawTexture. You generate ByteBuffer of constant data on every frame. I am pretty much sure it generates a lot of objects you abandon right after function returns. Consider creating pre-allocated ByteBuffer instance with static modifier.

The rest of code seems to be fine.

[lvonasek]

Thank you for the review.

1. That's for a follow-up PR to add into the settings an option to disable OpenXR even when it is supported. Someone here asked for it.

2. The headset's software keyboard sends only UP event. This way it will keep working correctly in case OS update adds a proper implementation. I will try to avoid using system time there.

3. Good catch, I will fix that.

[lvonasek]

Any another merge blocker @twaik?

[twaik]

I do not think so. I'll check it later.

[twaik]

Don't you mind if I squash all the commits into one? 47 commits is too much.

[lvonasek]

I am fine with squashing it.

[5A52]

Sorry for noob question, I managed to run XFCE4 on Q2 as 2d window, but when I switch to "oculus mode" just wait spinner (oculus 3 dot) continuously runs. What can I run in XR mode?

[lvonasek]

Sorry for noob question, I managed to run XFCE4 on Q2 as 2d window, but when I switch to "oculus mode" just wait spinner (oculus 3 dot) continuously runs. What can I run in XR mode?

Without log from logcat I cannot say much. Maybe an older OS version? (AFAIK it should work on V62 and higher)

The XR mode is mostly usable for gaming (using "mobox").

[5A52]

Sorry for noob question, I managed to run XFCE4 on Q2 as 2d window, but when I switch to "oculus mode" just wait spinner (oculus 3 dot) continuously runs. What can I run in XR mode?

Without log from logcat I cannot say much. Maybe an older OS version? (AFAIK it should work on V62 and higher)

The XR mode is mostly usable for gaming (using "mobox").

Do you mean > Mobox is a project designed to run windows x86 applications in Termux? Than XFCE4 (any X window) must be also visible as 2d window in 3d? If yes will try to catch tx11 logs, Im on V66. P.S. Im new to oculus, and it is the only one my android device without root, this is sad.

[lvonasek]

Do you mean > Mobox is a project designed to run windows x86 applications in Termux?

Yes

Than XFCE4 (any X window) must be also visible as 2d window in 3d?

Yes but without any advantages.

I did a similar integration like this for Winlator project: https://youtu.be/c4faL1G1St4?feature=shared

[5A52]

Cool! XFCE4 works in both modes, howeever. No way to return from preferencies activity, pressing/clicking/touching BACK buttons does nothing (in both modes). Have to close tx11 and open again to apply changes. In Winlator I did not find how to switch to 2D at all

[lvonasek]

If you open system notifications, there is tx11 and from there you can open preferences.

I stopped working on Winlator because it isn't fully opensource.

[5A52]

If you open system notifications, there is tx11 and from there you can open preferences.

Ofcourse, but question is - how to close preferences

[lvonasek]

With the X button. However TX11 might need a restart if you change XR mode when running.

[twaik]

@lvonasek Can you please tell me why you are doing this?

    private static int getMainDisplay(Context context) {
        final DisplayManager displayManager =
                (DisplayManager) context.getSystemService(Context.DISPLAY_SERVICE);
        for (Display display : displayManager.getDisplays()) {
            if (display.getDisplayId() == Display.DEFAULT_DISPLAY) {
                return display.getDisplayId();
            }
        }
        return -1;
    }

Display.DEFAULT_DISPLAY is always a display id you are looking for...

[twaik]

I mean you can use it directly.

[lvonasek]

I just followed this: https://github.com/amwatson/2DVrHybrid

I will create a PR cleaning it up.

[twaik]

I am pretty much sure the

        // 1. Locate the main display ID and add that to the intent
        final int mainDisplayId = getMainDisplay(context);
        ActivityOptions options = ActivityOptions.makeBasic().setLaunchDisplayId(mainDisplayId);

is not necessary, but only you can check if that is true.

[twaik]

Can you please find out how to get out of 3d mode programmatically? Starting MainActivity and finishing XrActivity does not seem to work.

[twaik]

I mean

getBaseContext().startActivity(new Intent(this, MainActivity.class)
        .addFlags(Intent.FLAG_ACTIVITY_NEW_TASK | Intent.FLAG_ACTIVITY_CLEAR_TASK));
finish();

[lvonasek]

I will take a look

[Xaarrt]

Hi I get this screen tearing or whatever in quest 3 v67 in both xr mode and 2d though in xr mode it happens only when its not in fullscreen Could you have a look please

[lvonasek]

No, this isn't connected to my implementation. It seems something to do with a GPU driver. Report it to Mobox or whatever you use.

[Xaarrt]

Hi If its something to do with a GPU driver it shouldn't work in xr mode .it does I think it related to scaling (resize) a window it start to behave like that, in fullscreen its good In 2d mode its broken Also is the feature moving screen closer or farther possible in xr mode cause its fixed Thanx

[Xaarrt]

If its possible to move screen in xr mode that solves my problem

[lvonasek]

It is a GPU problem. XR renders the data into an own framebuffer and then on a screen. I guess that makes the difference.

I stop doing support in this PR. All further messages will be ignored.

[Xaarrt]

Sorry for wasting your time with🙏 this All I want to know if it plausible to move and resize screen in xr mode Thanx